I simply deleted the files by connecting the phone by USB to my computer and selected to connect it as a disk device. Then I went down in the Downloads/.Mail folder and deleted the files. This was faster than using the file manager app in the phone, which took forever to open the .Mail folder since it had >21000 files in it using up about 700 MB of space.
The next time I opened the mail app it said "Preparing SD device" for a short moment. Everything has worked fine after that and my mail is working faster than before. In my belief it is ok just to delete the files.
(Apologises for adding to an old thread but I thought this could be helpful information for others with the same issue.)