Thunderbird very slow to open certain .eml files from disk

[Carl Colijn]

I have lots of emails archived by saving them to disk. For quite some time now double-clicking these loose .eml files (either with Thunderbird already open or not) will cause Thunderbird to open an email window, load up the header info (slow), then takes its time to display the body (up to several minutes), and all the while Thunderbird is stalled (unresponsive). This problem is worse on HTML emails (like PayPal receipts), and not so bad (but still tens of seconds) for plain text mails.

I do have a somewhat unusual set-up, but I have factored that out of the equation, and got some insight in what is causing the slow down. My setup is as follows:

• I'm using the latest Thunderbird (78.9.0, 32 bit) on Windows 10 (latest updates installed).
• Thunderbird is installed on my main SSD drive.
• My profile is located on my secondary platter drive.
• All email is stored on IMAP servers.
• I have a few extensions installed.
• I have split off the Mail folder from my main profile folder, and placed it next to it. So I have a "Thunderbird data" folder, with in it a "Mail" subfolder and a "Profile" subfolder. My profile points at the "Thunderbird data" folder.
• The entire "Thunderbird data" folder is stored in a Cryptomator vault to encrypt it when at rest and in the backups.
• Individual archived mail files live all over the place in loose .eml files.
• The whole thing is running in a Sandboxie sandbox.
• Windows Defender is active too.

As an aside: Windows Defender causes Thunderbird to become quite sluggish in every regard when I do not explicitly whitelist the "Mail" subfolder. For almost all actions imaginable Thunderbird is slowed down considerably, leading to emails loading slow (seconds to display with a stall of a second between displaying the header and the body), moving mails becoming slow, etc. When looking in Task Manager, Thunderbird itself is mostly idling (only a few % cpu usage), but Antimalware Service Executable is hogging an entire core's worth of processor time. After whitelisting it, Thunderbird is fast again.
I mention this because when researching this I saw others having trouble with this too while others did not, and all sorts of fixes got suggested like turning off sample submission etc. All of these didn't work for me, only whitelisting the mail folder did. Whitelisting the profile folder didn't have any effect. So it must be something related to mail files getting scanned over and over; maybe this suggests Thunderbird is not efficiently open-read-closing them in one go, but re-opens or re-queries these files very frequently, each time triggering another rescan? Which leads me to the following:

The real issue (opening eml files) seems to boil down to the following:

• Disabling Windows Defender doesn't make a change.
• Running Thunderbird outside Sandboxie doesn't make a change.
• Disabling all add-ons by starting Thunderbird in safe mode doesn't make a change.
• Moving the profile folder outside of it's Cryptomator vault does make a HUGE change; opening eml files is almost instantaneous again.

Looking into the latter point, I found out that when opening eml files, Cryptomator is peaking at a full 100% usage of a cpu core while it is processing all encrypt/decrypt requests from Thunderbird. Thunderbird itself is barely using any CPU at all; I assume it just waits for the file I/O requests to complete. Cryptomator did let me output a log of what file actions were performed in that period, and there I found this:

When an external eml file is opened, Thunderbird starts scanning my mail folder, and iterates over each mailbox file (about 600) and .sbd folder (about 40), opening it, asking for file properties and date/time stamps, and then immediately closing it, over and over and over again. The log uses quite a few lines per file action (about 20, then again for each file it asks for the information twice in a row), but the log is actually 1.9 million lines long when the .eml file finally is open... That amounts to about 150 queries for each .sdb folder and mailbox file. And that's why Cryptomator crawls to a halt; it is asked to decrypt all these files over and over again -- according to it's own stats, after this ordeal it had decoded 12GB worth of data.

What could be going on??? Why is Thunderbird even scanning my mail folder when opening an external .eml file? And, no, "stop using Cryptomator" is not a solution

I have this log stored locally if any developer is interested in taking a peek. It is about 10MB zipped. I'm hesitant to attach the log file here since it contains quite a lot of folder names I do not want to expose, and anonymizing the log file is undoable.

The following is an excerpt from the log dealing with one mailbox file. There are thus a gazillion of these blocks in the log, and the below file is re-queries a few seconds later again. And again.

Code: Select all

20:14:12.445 zwCreateFile() is called for /Mail/folder1 with the following parameters:
	CreateDisposition -- OPEN_EXISTING
	createOptions -- []
	accessMasks -- []
	fileAccessMasks -- []
	fileAttributes -- [].
20:14:12.445 Try to open /Mail/folder1 as File.
20:14:12.445 Reading file header from D:\App data\Cryptomator self\Thunderbird\d\NH\JO3NEOEV47J5SBG6X26KZG5HWKO6CT\QOicxipX66ar1kiKViw38jhLYPQA5tPs8Z-A3yU=.c9r
20:14:12.445 First channel for this openFile. Initializing file size...
20:14:12.445 (81601) /Mail/folder1 opened successful with handle 81601.
20:14:12.446 (81601) getFileInformation() is called for /Mail/folder1.
20:14:12.446 (81601) Filesize of /Mail/folder1 is 111181.
20:14:12.446 Timestamps for /Mail/folder1:	 mTime 1594500792487 	 cTime 1613665339718 	 aTime 1617300852446
20:14:12.446 (81601) File Information successful read from /Mail/folder1.
20:14:12.446 (81601) getFileInformation() is called for /Mail/folder1.
20:14:12.446 (81601) Filesize of /Mail/folder1 is 111181.
20:14:12.446 Timestamps for /Mail/folder1:	 mTime 1594500792487 	 cTime 1613665339718 	 aTime 1617300852446
20:14:12.446 (81601) File Information successful read from /Mail/folder1.
20:14:12.446 (81601) cleanup() is called for /Mail/folder1.
20:14:12.447 (81601) closeFile() is called for /Mail/folder1.

[tanstaafl]

"Moving the profile folder outside of it's Cryptomator vault does make a HUGE change; opening eml files is almost instantaneous again."

I don't have similar problems when using a Veracrypt partition stored locally on my hard disk. It sounds like if you didn't archive messages as .eml files on your cloud service using Cryptomator everything would be fine. You could still use Cryptomator with your profile. Its sounds like either you are accessing your archived messages way more often that what I would expect for archived mail or you're archiving them too early.

Are you using POP or IMAP accounts?
Are you running into limits on how much space your IMAP account can use?

The simplest solution if you use IMAP might be to pay for more space in your mailbox and archive mail in other remote folders rather than as .eml files in your Cryptomator vault. I assume you're using a commercial email provider rather than a free one. Do you trust them to do a good job backing up/restoring mail?

[Carl Colijn]

I don't have similar problems when using a Veracrypt partition stored locally on my hard disk.

Yeah, before Cryptomator I also used Veracrypt for encryption and had my mail folder in a Veracrypt volume, and then there was a lot less slowness for me too. But there was a noticeable delay nonetheless.

It sounds like if you didn't archive messages as .eml files on your cloud service using Cryptomator everything would be fine. You could still use Cryptomator with your profile. Its sounds like either you are accessing your archived messages way more often that what I would expect for archived mail or you're archiving them too early.

Oh, the archived .eml files live as truly individual files on my HDD, so not in the cloud and also not under Cryptomator; I create them via a "File / Save As .eml" (well, via ImportExportTools tbh). The profile itself (incl. my Thunderbird mail folder) is what I have stored in a Cryptomator vault, but that vault is also not located in the cloud; it just resides on my HDD.

So when working regularly with Thunderbird I do not access these .eml files at all, and all is fine. But once I try to open an .eml file in Thunderbird by double-clicking on it in Explorer, Thunderbird goes on a stampede scanning through my profile's mail folder files. This is what is causing the extreme slowdown; each mail folder file Thunderbird wants to scan triggers an (understandable) decrypt of sorts in Cryptomator, and since it scans all these files 150-fold, this adds up enormously.

I expect this re-re-re-scanning was also happening under Veracrypt, but in that case the file info is stored in a true NTFS file structure (small) which Veracrypt only needs to decode once and is then probably cached for repeat queries. With Cryptomator each file is represented one-on-one by an encrypted file, but the names of these files are also encrypted, so when Thunderbird asks for a file's properties it seems there is no single read-once-and-cache block of data to consult; I guess Cryptomator computes the encrypted file name + accesses that file's properties over and over again, having much higher processing costs (5+ minutes to be exact).

But I fail to see the connection between opening an .eml file and Thunderbird scanning my entire mail folder, especially the need to scan the same files over and over again 150-fold, so I assume this could be a bug in Thunderbird?

What did cross my mind is: could this be a compact run gone wrong? Scanning each mail folder file and only asking for file size and creation/modification dates? Then again I compact regularly, so there should be no need to auto-compact again for each .eml file I open. Or rather: opening eml files shouldn't trigger a compact at all I guess.

Are you using POP or IMAP accounts?
Are you running into limits on how much space your IMAP account can use?

The simplest solution if you use IMAP might be to pay for more space in your mailbox and archive mail in other remote folders rather than as .eml files in your Cryptomator vault. I assume you're using a commercial email provider rather than a free one. Do you trust them to do a good job backing up/restoring mail?

I'm using IMAP, but that is irrelevant here I assume, since this issue concerns only files on my local HDD. No cloud, no server sync, no remote folders. Just a local .eml file I want to open after which Thunderbird goes on a massive rampage in my mail folder.

[tanstaafl]

Mbox files get compacted. .eml files don't (because there is one file per message and its not a supported storage scheme).

My first thought would be to temporarily configure a different program to be called by Windows to view .eml files. See if similar problems occur using it.

You could also configure Thunderbird to default to using maildir files (rather than mbox files) for any new accounts you add, sign up for a free account somewhere and add it to Thunderbird. maildir uses renamed .eml files to store messages (plus has a special directory layout). See if you have problems using that new account. If you do you'd have much better chances getting a developer to look into that bug (which might fix your original problem as a side effect) than if you submit a bug report about one about opening a .eml file using explorer.

[Carl Colijn]

Mbox files get compacted. .eml files don't (because there is one file per message and its not a supported storage scheme).

My first thought would be to temporarily configure a different program to be called by Windows to view .eml files. See if similar problems occur using it.

I know; it's not that I thought opening the .eml files caused Thunderbird to compact those. Opening individual .eml files in Thunderbird causes Thunderbird to stampede through my mailbox files, scanning them over and over again for five minutes before it settles down and becomes responsive again. As indicated in the fragment of the log I posted along. The real log is 10MB big and contains thousands of those fragments in quick succession (all caused by opening 1 eml file). Once Thunderbird has settled, the .eml shows fine. After closing the .eml and opening another one, the same process starts over again. For now I've resorted to opening the .eml files in Notepad++ and just reading the mail source; this doesn't get any slowdowns (but is a pain with html mails). Importing an .eml again (with the ImportExportTools plugin) goes fine and is instantaneous too.

So it definitely seems to be Thunderbird causing the mailbox file interrogations.

You could also configure Thunderbird to default to using maildir files (rather than mbox files) for any new accounts you add, sign up for a free account somewhere and add it to Thunderbird. maildir uses renamed .eml files to store messages (plus has a special directory layout). See if you have problems using that new account. If you do you'd have much better chances getting a developer to look into that bug (which might fix your original problem as a side effect) than if you submit a bug report about one about opening a .eml file using explorer.

I was already on the fence for quite a few years about maildir files. I really want to start using that (for speedier backups and lack of compacting etc.). Have they been deemed stable enough for serious production use yet? I really hope so!

Anyway, I've tested it with a maildir files setup in a new profile;

• Started Thunderbird with -p switch to create a new profile
• Created the new profile next to my existing one, but also under Cryptomator supervision (since that is the perfect canary here to detect file accessing; it adds some file access slowdown and can generate file access logs)
• Added one of my IMAP mail accounts using maildir format
• Let Thunderbird sync the account; all maildir\cur folders got populated with .eml files
• Restarted Thunderbird (just to be sure)
• Double-clicked a pre-existing loose .eml file on disk (so not one in the new maildir folders)

And the same thing happened again; Thunderbird hung for about 2 minutes while Cryptomator got loads and loads of file queries to process. After that Thunderbird got responsive again and showed the .eml file. The reason it took less time this time is that I only added my main mail account and not the 7 others. The Cryptomator log shows similar behavior: each .sbd file is queried twice in a row for it's file size and time stamps, and that for each .sbd it seems, and that repeated lots of times. I now saw lots of the .msf files are queried and fully read too. The file access log from this one .eml file open is half a million lines long...

So I think this is a genuine Thunderbird bug or misbehavior. One to probably not get noticed easily when working directly from an NTFS volume, since that's fast enough to take this kind of abuse?

[tanstaafl]

You could file a bug report at https://bugzilla.mozilla.org/home but I think you'd have better luck creating an issue with the project at https://github.com/cryptomator/cryptomator/issues . If they can duplicate the problem and decide to submit a bug report a Thunderbird developer would probably take it much more seriously.

I've been using maildir storage for several years without problems since I don't do anything that is listed in "Operations still unsuccessful with Maildir" in https://wiki.mozilla.org/Thunderbird/Maildir . The version 91+ roadmap at https://developer.thunderbird.net/planning/roadmap states:

Our maildir support is pretty good by now. Finish up the last things needed to have maildir (our one file per message storage) on by default for new accounts.
Tracked in bug bug 845952