Thunderbird mbox conversion to maildir

Discussion of general topics about Mozilla Thunderbird
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Thunderbird mbox conversion to maildir

Post by tanstaafl »

Why is 500K files in the profile an issue? They're scattered across various child directories. I'd assume any performance hit due to having many more files in the accounts local directory is easily offset by not having a large mbox file for the inbox.
porcupine5
Posts: 9
Joined: December 10th, 2017, 9:45 pm

Re: Thunderbird mbox conversion to maildir

Post by porcupine5 »

tanstaafl wrote:Why is 500K files in the profile an issue? They're scattered across various child directories. I'd assume any performance hit due to having many more files in the accounts local directory is easily offset by not having a large mbox file for the inbox.
I'm actually not sure that it is.

The original thinking moving from mbox > maildir was my experience in a heavily multi-user environment, failing to consider my local PC is specifically not multi-user, and thus a lot of the performance improvements I saw on server side, just wouldn't apply. I failed to consider that when doing the original migration.

Now I've had very spotty performance for a long time, over a number of different versions, and I'm grasping at straws really. My speculation here is that a lot of the time, FS's/OS's don't handle having a million tiny files very well, so perhaps consolidating everything back to mbox will actually yield a performance gain.

I could be totally and completely wrong of course. I just updated to 60.3.1 (off the older version, 52.x I think?), so maybe I'll have some magic performance boost in the process there, I haven't started flipping anything over yet, plenty of hope and all :D.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Thunderbird mbox conversion to maildir

Post by tanstaafl »

How many messages do you typically have in a inbox folder?
porcupine5
Posts: 9
Joined: December 10th, 2017, 9:45 pm

Re: Thunderbird mbox conversion to maildir

Post by porcupine5 »

tanstaafl wrote:How many messages do you typically have in a inbox folder?
Usually < 100 in my actual/direct "inbox" folder.

A number of the folders (IE: annual broken down archives of different types) have 5-15k messages.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Thunderbird mbox conversion to maildir

Post by tanstaafl »

That should work well. You have a small amount of files in the directory used for the inbox, and Thunderbird will ignore the many files in the directories used for archives unless you opened one of those folders.

"Thunderbird loads a folder's *.msf file into memory whenever it opens the folder. By default it keeps up to 30 *.msf files open for 5 minutes (and their contents in memory), unless they are "special" folders, which it never closes. It loads the inbox.msf file for every account on startup."

Have you checked for bad sectors recently?

https://www.howtogeek.com/howto/windows ... ows-vista/
porcupine5
Posts: 9
Joined: December 10th, 2017, 9:45 pm

Re: Thunderbird mbox conversion to maildir

Post by porcupine5 »

tanstaafl wrote: Have you checked for bad sectors recently?
Yeah I've had bad sectors for awhile, but they're remapped. I ran a check after you suggested (as it's not a bad idea), no new ones found though. Too few to RMA the drives (disappointing, but ... yeah, Intel hasn't been very friendly about their SSD RMA's recently).

I'm idling on the new version to see if that makes a difference while I tackle some other stuff. We shall see if the annoying halting continue/return I suppose :).
porcupine5
Posts: 9
Joined: December 10th, 2017, 9:45 pm

Re: Thunderbird mbox conversion to maildir

Post by porcupine5 »

I wanted to update this post, for anybody curious.

I finally made the leap back to mbox. The method I used (since I haven't seen it posted specifically), and this would work in either direction being the lowest effort to migrate a number of accounts/messages/etc. relative to other posted solutions:

[in active profile, after backing up]:
#1 - Reset default message store to mbox
#2 - Create new fake email account for storage, its mailstore gets setup as mbox.
#3 - Copy folders in their entirety drag/drop over to the new email account, renaming folders as necessary, copying some stuff [inbox/sent/etc.] by mass selecting messages, as folders won't drag/drop. Let thunderbird convert the messages *back* to mbox from maildir. Note drag/drop for folders is copy, drag/drop for files is move (lack of consistency!)
#4 - Verify all messages copied over to new account, randomly check messages, folder message counts, etc. If everything looks good?
#5 - Open up the config editor, locate the syntax for mbox vs maildir (searching "maildir" in the config editor for me, since I was already maildir converting backwards), and force local folders [where I store all my mail] over to mbox.
#6 - Close TB, navigate to the mail storage directory on the PC, move local folders out of the /mail/ directory, and create a new directory called "Local Folders" (case sensitive counts for some stuff apparently).
#7 - Copy over message rules/whatnot from the old local folders.
#8 - Go to the mailstore of the account you moved everything over to, and copy back to local folders by hand, renaming files as necessary.
#9 - Startup TB, mail should all appear, check to make sure everything is there. Congrats, you're now back in mbox mode. Delete the extra storage mail account once you're certain that all data is back properly.

The above method prevents you from having to recreate your profile, re-create email accounts, re-set signatures, etc. etc. So much less work. Obviously if you don't actively use Local Folders your steps will vary (but the basic concept is the same, use TB to convert the message store type for folders in a temp account, then mangle the message store settings on the proper/individual email accounts).

That said, the moment I swapped back to mbox, performance looked enormously better. Some tasks, like ctrl + a in a folder with 10-20k messages would take ~15 seconds on maildir with TB fully locked, now that same operation is < 1/2 second. I haven't been running this for more then 24 hours at this point, but stuttering and pausing appear to have subsided (before it was within the first 15 minutes of opening TB), it *appears* as though this may have finally resolved my issues.

TL;DR: Stick with mbox, unless you *NEED* something maildir (lite) is offering.
Post Reply