Thunderbird mbox conversion to maildir

Discussion of general topics about Mozilla Thunderbird
shiku
Posts: 4
Joined: March 20th, 2017, 9:14 pm

Thunderbird mbox conversion to maildir

Post by shiku »

Please kindly advise/assist with the conversion of mbox email accounts to maildir. I have some users on mbox(POP3) and we intend moving to business gmail of which all accounts have to be in IMAP so i have to convert the POP3 ones to IMAP returning the emails and folder structure.
I could have recreated the email accounts and dragged the emails to the new folders but some of the users have too many folders and sub folders to take the manual route will be cumbersome.
I have ready on mb2md but i'm not too sure how to go about it on my setup, the conversion is done on the server ? then how does it work with the already created folders and downloaded emails by pop3. Is there any other go around this or how to with mb2md ?
The file path for my emails is /var/vmail/domain/user/ and its a postfix, dovecot setup.

Thanks
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Thunderbird mbox conversion to maildir

Post by tanstaafl »

There is no support for converting an existing account from one storage format to another. Thunderbird's maildir support is sometimes called maildir-lite because "You cannot point a Thunderbird account to a mail server directory. You do not get message flags stored with emails. You only get a single file per message, and nothing more."

You can specify in tools -> options -> advanced -> general the message storage format that should be used when creating a new account.

Google imposes a bandwidth limit, but you could upload messages from the POP accounts to the Gmail IMAP account simply by copying or moving the folders (or their contents if the folder already exists). That works with folder hierarchies, you don't have to copy each folder separately. However, if its too cumbersome to do it manually try the "Google webMail Filer for Thunderbird" utility at http://www.softpedia.com/get/Internet/E ... bird.shtml . That should let you automate it. I tried it about 5 years ago and added a paragraph about using it in http://kb.mozillazine.org/Importing_and ... _your_mail . Since the mbox file format hasn't changed and its a standalone utility, hopefully it will still work. See http://www.ghacks.net/2011/01/07/google ... underbird/

However, I'm not sure why you want to convert to maildir. Despite its inherent flaws mbox is more reliable than the current implementation of maildir. Thunderbird's maildir support is usable, but it is a work in progress. You have to be aware of what type of actions are risky. See https://wiki.mozilla.org/Thunderbird/Maildir . If you are replacing some accounts with IMAP Gmail business accounts that shouldn't care what mail storage format Thunderbird uses.
shiku
Posts: 4
Joined: March 20th, 2017, 9:14 pm

Re: Thunderbird mbox conversion to maildir

Post by shiku »

Thank you for the response, i think Google webMail Filer utility we do the job for more, i was under the impression, to move to Google, Thunderbird emails have to be in IMAP(maildir) format.
Using Google webMail Filer utility, i will still have my folder structures as there were ?
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Thunderbird mbox conversion to maildir

Post by tanstaafl »

Yes, the Google webMail Filer utility should preserve the folder structure when it uploads the messages.

When you create the Gmail IMAP account I suggest you keep using mbox. It doesn't matter if the Gmail server uses mbox, maildir, UW, MH files or a database to store the messages, that has no effect on Thunderbird.
shiku
Posts: 4
Joined: March 20th, 2017, 9:14 pm

Re: Thunderbird mbox conversion to maildir

Post by shiku »

Thanks. The issue is these guys want to continue using Thunderbird, but i'm told to point to Business Gmail, everyone has to be on either MS Outlook(IMAP) and Thunderbird(IMAP). Therefore the need to convert all Pop3 accounts to IMAP on these mail clients.Some of them wont use the Gmail interface, there will stick to Thunderbird.

My question therefore is, those staying with Thunderbird(POP3), will they be able to use Business Gmail.

Thank you for all your help.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Thunderbird mbox conversion to maildir

Post by tanstaafl »

G Suite (which I think is what you meant by Business Gmail, based on https://gsuite.google.co.in/intl/en_in/products/gmail/) supports both IMAP and POP. But its up to the admin to enable it. Another potential issue is if your organization wants to use one or more "public folders" (a way to share a mail folder among selected users or a team, without it having to be in their mailbox) that requires a IMAP account. "G Suite Business" used to be called "Google Apps for Work Unlimited".

https://support.google.com/a/answer/105694?hl=en
https://support.google.com/a/answer/7126147?hl=en
shiku
Posts: 4
Joined: March 20th, 2017, 9:14 pm

Re: Thunderbird mbox conversion to maildir

Post by shiku »

Thank you for the info.

Just one more thing, after checking on my server, the pop3 accounts are in maildir format on that server. But on the user PC they are in mbox format. I wanted to find out , if i have any options in converting the files on the user PC from mbox to mail. Thereafter i wish to just create the same account as imap and pick up the new maildir created files ?
Or is there any other option to get around this ? It seems the client wants it done this way.

Regards
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Thunderbird mbox conversion to maildir

Post by tanstaafl »

AFAIK there is nothing to be gained by having the email client and the server use the same message storage format. With TB 52.0.1 you can change the format used for new accounts. So if you wanted you should be able to copy the messages from the POP account to Local Folders, delete the account, change the new format to maildir, create the account again, and move the messages back to it. However, there is no support for you doing something like copying the maildir files from the server using a ftp utility to the profile and then try to have Thunderbird use those files. I'm not familiar enough with maildir-lite to know what issues you'll be running into if you try to do that but all that you are guaranteed by the maildir-lite implementation is that there is a single file per message. Its not real maildir. Tell the client its too risky.
porcupine5
Posts: 9
Joined: December 10th, 2017, 9:45 pm

Re: Thunderbird mbox conversion to maildir

Post by porcupine5 »

I figured I'd necro this thread (since I can't find a lot on Maildir). I swapped over from Mbox to Maildir in the summer, under the misguided (mailserver admin) mindset that I'd see a performance boost (as I admin a number of servers, that saw substantial performance gains when making the swap, failing to consider that's primarily because it's a huge multi-user environment, vs a single user environment).

Anyway, I'm seriously regretting the swap. Performance was poor before the swap, but since then it's been even substantially worse. I have multi-second input lag, frequently in TB, without an obvious cause. Searching messages has certainly not become any faster (which shouldn't be surprising really, I don't know what I was thinking honestly).

My local TB profile/mail install has 23GB of mail, across 385 folders, and ~470k messages (just checked the mail dir in the profile). I've not yet been able to find a way to convert *from* maildir, back to mbox, probably something I should have checked waaayyy back when I made the original move (I kept backups of the mbox profile, but it obviously is out of date by now).

Anybody run across a method of doing the reverse conversion?

Part of me is tempted to uninstall/re-install TB, but I'm not sure if that'll do much, since I still need to retain all of my email/archives/settings/filters/etc.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Thunderbird mbox conversion to maildir

Post by tanstaafl »

Re-installing Thunderbird is unlikely to solve a performance problem. That is usually due to either problems in your profile, its location or external factors such as what security/AV software you have. See http://kb.mozillazine.org/Performance_-_Thunderbird . It's long, but that is due to it being so thorough. If you have a multi-second lag when you are not reading a message I'd suspect it has nothing to do with your storage type, especially since it was bad beforehand. My first thought would be its either due to real time scanning of your email via a security suite, using Direct2D hardware acceleration or due to there being places.sqlite-shm and places.sqlite-wal directories (NOT files) in the profile. See how wide spread the problem is, for example does it also occur when composing a new message? Knowing how wide spread the problem is helps narrow down the causes to worry about.

Some of my 8 IMAP accounts use mbox while others use maildir. I don't notice a difference.

One way to convert back to mbox would be to backup the profile, change the default back to mbox and then

1. Move all of the mail for a POP account to "Local Folders"
2. Delete the account (checking the checkbox to delete its data)
3. Add the account again (it should be mbox this time)
4. Move the mail back from Local Folders.
5. Repeat for the next account.

If you have a IMAP account its simpler as your mail is stored in remote folders. Its only the optional offline folders that would use maildir. Syncing the remote and offline folders will write to the maildir files, other than that they're normally ignored unless you are configured to work offline. They are not used as a cache. So if you have a IMAP account it might be simplest to leave it alone.

The support page (https://freeshell.de/~kaosmos/mboximport-en.html) for the ImportExportTools add-on at https://addons.mozilla.org/en-US/thunde ... porttools/ claims it supports importing/exporting maildir files. However I only see a menu item for importing maildir when I try it against a account using maildir. That might be an artifact of all of my maildir accounts being IMAP.
porcupine5
Posts: 9
Joined: December 10th, 2017, 9:45 pm

Re: Thunderbird mbox conversion to maildir

Post by porcupine5 »

tanstaafl wrote:Re-installing Thunderbird is unlikely to solve a performance problem. That is usually due to either problems in your profile, its location or external factors such as what security/AV software you have. See http://kb.mozillazine.org/Performance_-_Thunderbird . It's long, but that is due to it being so thorough. If you have a multi-second lag when you are not reading a message I'd suspect it has nothing to do with your storage type, especially since it was bad beforehand. My first thought would be its either due to real time scanning of your email via a security suite, using Direct2D hardware acceleration or due to there being places.sqlite-shm and places.sqlite-wal directories (NOT files) in the profile. See how wide spread the problem is, for example does it also occur when composing a new message? Knowing how wide spread the problem is helps narrow down the causes to worry about.

Some of my 8 IMAP accounts use mbox while others use maildir. I don't notice a difference.

One way to convert back to mbox would be to backup the profile, change the default back to mbox and then

1. Move all of the mail for a POP account to "Local Folders"
2. Delete the account (checking the checkbox to delete its data)
3. Add the account again (it should be mbox this time)
4. Move the mail back from Local Folders.
5. Repeat for the next account.

If you have a IMAP account its simpler as your mail is stored in remote folders. Its only the optional offline folders that would use maildir. Syncing the remote and offline folders will write to the maildir files, other than that they're normally ignored unless you are configured to work offline. They are not used as a cache. So if you have a IMAP account it might be simplest to leave it alone.

The support page (https://freeshell.de/~kaosmos/mboximport-en.html) for the ImportExportTools add-on at https://addons.mozilla.org/en-US/thunde ... porttools/ claims it supports importing/exporting maildir files. However I only see a menu item for importing maildir when I try it against a account using maildir. That might be an artifact of all of my maildir accounts being IMAP.



To help paint the picture a bit more, I don't have any active (3rd party) addons/plugins, only the lightning extension is installed. I do have a number of custom message filters, but nothing too nutty (they only run on retrieval too right, so not suspecting that). Email is pop3s/smtps. The input delay is often when selecting/moving between messages, typing replies, etc. (I've also noticed the dictionary finds spelling errors by highlighting half a word as invalid, tries to move the keyboard cursor in unexpected manners sometimes if I erase too far [IE: seems to leave the message pane entirely, or dumps into the signature]). Most of the input delay is < 1 second, pauses while typing here and there, that just make you think you've screwed up/typo'd/etc., enough to feel slow/messy. Message highlighting/selection are often longer delays, and occur pretty frequently (IE: when gathering up spam to delete, etc.).

Only AV/scanning I have is MS security essentials, AFAIK it's pretty much the most benign option one could have. I can always turn off real-time scanning to see if that makes a difference granted.

I've tried flicking the HW acceleration on/off (hit that on another thread in this forum a few months ago), without any noticeable impact.

"places.sqlite-shm and places.sqlite-wal " are both files, not directories (neither being particularly large, the former 1.2M, the later 33kb). There's a hefty "global-messages-db.sqlite", which sits at 1.1GB, and at a glance I'd assume is what TB indexes/searches.

It sounds like the way to move back to mbox is pretty similar to how I moved out of it. I remember having various issues (because I already use local folders, and IIRC, I had issues with the volume of messages, TB locking up, etc.), but I definitely did lots of that "move back and forth between profiles/folders/etc." to make TB do the original conversion of the mbox files.

Sounds like I've got some reading to do, much appreciated on the link!
porcupine5
Posts: 9
Joined: December 10th, 2017, 9:45 pm

Re: Thunderbird mbox conversion to maildir

Post by porcupine5 »

tanstaafl wrote:Re-installing Thunderbird is unlikely to solve a performance problem. That is usually due to either problems in your profile, its location or external factors such as what security/AV software you have. See http://kb.mozillazine.org/Performance_-_Thunderbird . It's long, but that is due to it being so thorough.
Reading through this, wow I realize how much info I left out heh...

PC is running windows7-64, TB is 52.5.0-32 (last I checked there was no x86_64, they put that in there to tease!).

CPU is an under I7-2600k, 32G of DDR3, with a geforce 960 GPU, and native res is 4k (I mention, as it definitely messes up some stuff, IE: Google Maps doesn't love 4k in FF). The mailstore is on a 3ware 9750-8i, 2 drives used for a RAID1 of 2xIntel 540-480GB SSD's (the other 6 are unrelated/mostly inactive for other archiving). RAID card has caching/etc. enabled.

Archiving is done manually (IE: outside of the actual FF archive folders), split to ~3 folders/year, inbox generally has 40-100 messages, usually < 20MB in size. Global folders are used for almost everything (hence the message filters/rules, to organize stuff).

During the input lag, mouse cursor/etc. still moves fine (as you'd expect), CPU/GPU do not spool up by any measurable amount (CPU fan is irritating, and you can distinctly hear it whine-up when a process is using a lot of CPU, doesn't even need to be 100% of a core).
porcupine5
Posts: 9
Joined: December 10th, 2017, 9:45 pm

Re: Thunderbird mbox conversion to maildir

Post by porcupine5 »

Just figured I'd post an update, hopefully this nails it for somebody else.

After reading through the performance article linked by tanstaafl, I think I've solved my problem. Only change I made was:
One user reported that disabling chat by setting mail.chat.enabled and mail.chat.show_desktop_notifications to false eliminated a 5 second delay every time he selected a different message in the folder listing [11]
Guess my problem was an outlier.
wsmwk
Posts: 2831
Joined: December 7th, 2004, 6:52 am
Contact:

Re: Thunderbird mbox conversion to maildir

Post by wsmwk »

porcupine5 wrote:Just figured I'd post an update, hopefully this nails it for somebody else.

After reading through the performance article linked by tanstaafl, I think I've solved my problem. Only change I made was:
One user reported that disabling chat by setting mail.chat.enabled and mail.chat.show_desktop_notifications to false eliminated a 5 second delay every time he selected a different message in the folder listing [11]
Guess my problem was an outlier.
porcupine5, do you actually have a chat account defined? If so, what chat servers and channels are enabled?
porcupine5
Posts: 9
Joined: December 10th, 2017, 9:45 pm

Re: Thunderbird mbox conversion to maildir

Post by porcupine5 »

wsmwk wrote:
porcupine5 wrote:Just figured I'd post an update, hopefully this nails it for somebody else.

After reading through the performance article linked by tanstaafl, I think I've solved my problem. Only change I made was:
One user reported that disabling chat by setting mail.chat.enabled and mail.chat.show_desktop_notifications to false eliminated a 5 second delay every time he selected a different message in the folder listing [11]
Guess my problem was an outlier.
porcupine5, do you actually have a chat account defined? If so, what chat servers and channels are enabled?
My apologies on the 2nd necro now, heh. I apparently never setup notifications on this post (and once my issue was resolved, I didn't end up manually searching anymore).

Ironically the reason I'm back in this thread/on the forum, is because the repair wasn't long lived. TB performance took a dump (with random extended pauses while writing messages, scanning messages, mysterious cursor bumps only existing in TB when hitting stuff like [home] to go back to the start of a line, ending somewhere random, etc.).

I see now TB is addressing Maildir more directly in the latest releases, this being in the notes of course: https://support.mozilla.org/en-US/kb/ma ... hunderbird

Thinking perhaps ~30gb of emails, > 500k files in the profile directory, etc. is just a bad idea, and maybe I'll have to do the painful mangle back to mbox (potentially complicated by the fact I'm dumping most to local folders from multiple accounts).

In answer to the original questions, I never had any chat account setup, had no idea it was a feature. Right now I'm just running TB and Lightning, with most stuff over POP3 (as I like to archive endlessly, as you might have guessed).
Post Reply