POP3-to-IMAP conversion for large email sets, folders: help?

hydrostarr · Post by **hydrostarr** » August 29th, 2010, 9:05 am

I'm converting multiple email accounts (on 2 different servers, one at tuffmail.net, the other a Zimbra server) from POP3-based access--with thousands of email folders, some of them quite large--to IMAP-based. On TB3.

This is quite helpful:
http://kb.mozillazine.org/Convert_a_POP ... AP_account
(Many thanks to Tanstaafl for writing this wiki page.)

However, I list some challenges below. Seeking thoughts/suggestions/help.

How to get all mailboxes compacted?

Been a problem for quite some time. I'm not convinced that the manual designation to compact folders actually "hits" all the folders. Maybe someday I'll try all the extra/add-on stuff listed at http://kb.mozillazine.org/Compacting_folders . For now, I'm hoping this is not contributing to any of my current problems (listed below).

Drag-and-drop of multiple folder sets from POP to IMAP account gets interrupted

This is the real killer. I'm copying over gigabytes of email folders, and when anything hiccups, the whole copy just STOPS, and I have to start over completely (unless someone knows a handy "resync" mechanisms that doesn't require an entire copy...? imapsync utility only seems to apply from imap account to imap account). One of these problems: an "mail header is not properly formatted"--not sure if that's the exact error message, last time I saw it I didn't dictate/copy the exact syntax, it just appears in a popup/scroll window that disappears after you click on it. I fear I could be repeating this excercise over and over again and never really get finished. ugh. (Note: these email folders were converted from a Eudora folder set long ago, could be contributing to the problem. Also, this was on the tuffmail.net server, btw.)

Folders with large number of emails (30K+) don't completely copy

They seem to copy over ok, given the "real-time" status in the bottom status bar of TB3, but when I go to view the email folder with 32,000+ emails in it, it only lists about 6700+ emails. Hmmmm. (This was the Zimbra server, btw.)

Thanks for any help....

Post by **tanstaafl** » August 29th, 2010, 10:52 am

http://kb.mozillazine.org/Copying_or_mo ... s_messages

I believe "file -> compact folders" hits all folders.

I'd be tempted to export the folders as a hierarchy of .eml files using the ImportExportTools add-on and then upload them using IMAPSize. See this blog post for screen shots of how to backup and then restore an IMAP account using IMAPSize. I suspect it just backs them up as .eml files in a hierarchy of directories (no extra files storing config information that you have to reverse engineer) - so you could upload the .eml files by pretending they are an account backup. I suggest you backup a trivial size remote folder to verify this. The reason I'm concerned about this is I know IMAPsize is smart enough to do incremental backups (won't backup the same message twice).

It claims it can "Download/upload messages to/from mbox and eml files". The reason I suggest you convert the mbox files to .eml files using the ImportExportTools add-on is that it might make it easier to detect and deal with any upload problems. For example, you might have a corrupted message in one of your folders. Hopefully ImportExportTools will isolate that by either generating a bad .eml file for it or skipping over it. I have more faith in ImportExportTools than IMAPSize handling any problems in converting a mbox to .eml files due to the good reputation of the ImportExportTools author.

I thought about suggesting you get a free Gmail account (it supports IMAP) and use the google gmail loader utility to upload the messages (and then use another utility to move between IMAP servers) but Gmail imposes a bandwidth quota that can cause uploads/downloads to silently fail. You can find Perl scripts such as this one to upload from local folders to a IMAP server but you'd have to install Perl and there is always the question of how good some script you find on the network is. Another one (this one is in Python and at least its a sourceforge project) is IMAP Upload. There is a blog post about it here with some screen shots. You can get Python for Windows here.

If any of this works for you please (briefly) describe what you did. I'd like to write a KB article about this topic some day.

hydrostarr · Post by **hydrostarr** » August 29th, 2010, 11:42 am

Extremely helpful tanstaafl. I'll followup with my findings.

hydrostarr · Post by **hydrostarr** » August 29th, 2010, 2:49 pm

Are the email files supposed to end in ".eml"? If so, this didn't work on first pass. Any idea what I need to do to tell the Export stuff to export .eml files?

Also, I'm not sure that IMAPsize is going to magically restore files created by ImportExportTools; IMAPsize creates "database" data of the files it creates. Might there be some other means to upload .eml files?

hydrostarr · Post by **hydrostarr** » August 29th, 2010, 3:17 pm

Ok, so I figured out how to export .eml files...and it only seems to work on a per-folder basis. ImportExportTools doesn't seem to "recurse" into subfolders to export .eml files. Maybe I'm missing some option?

Otherwise: this may be a "back to the drawing" board maneuver.

Post by **tanstaafl** » August 29th, 2010, 3:20 pm

I'm not sure what setting "tools -> importexporttools -> options -> export directories" to "exports folders as single files" does. It might let you export the entire folder tree as .eml files using tools -> importexporttools ->export folder with subfolder (with structure)

Post by **tanstaafl** » August 29th, 2010, 4:13 pm

"Also, I'm not sure that IMAPsize is going to magically restore files created by ImportExportTools; IMAPsize creates "database" data of the files it creates. Might there be some other means to upload .eml files?"

I just backed up a folder using IMAPSize. Besides the *.eml files it created a backup.db file that says it uses the SQLite format. http://www.broobles.com/imapsize/imap-backup.php describes it. Browsing that page and its forum its clear you can restore to a different server or folder, but I can't find any way to upload without knowing how to hack that file. You might try one of the SQLite Manager tools with it if you want to pursue it.

offlineimap is another potential tool but it expects the messages to be stored using the MailDir format. There are scripts that will convert Mbox files to MailDir files but if you're going to monkey around with scripts I suggest trying IMAP Upload.

You could run a local IMAP server that supports mbox files. Then you could use either IMAPSize or imapsync to move the folders between servers. The problem is the only easy to use/install ones I know of don't support mbox files. http://wiki.zimbra.com/wiki/Mail_Migrat ... al_Folders says "The easiest way I found for importing a lot of Thunderbird local folders is to set up a temporary dovecot server, convert the thunderbird mail stores to linux mbox mailstores and then imapsync it ". In this case you'd imapsync it to your real imap server, rather than zimbra.

The GNU mailutils package has a lot of mail utilities that support IMAP but none of them seem appropriate.

http://wiki.zimbra.com/wiki/Mail_Migration#pop2imap mentions pop2imap. You'd need to create the destination folders and modify a script so that it doesn't try to download the messages from a POP server to get the headers.

http://www.athensfbc.com/imap_tools/ has a lot of IMAP-centric Perl scripts. MboxtoIMAP.pl copies mbox files to a IMAP server.

I suggest you either try the IMAP Upload Python script or install Dovecot locally and use IMAPsize to transfer between the two mail servers. Installing Python and trying the script is the easier of the two

hydrostarr · Post by **hydrostarr** » August 29th, 2010, 4:25 pm

Once again, excellent details tanstaafl, very much appreciated. I'll followup with my next round of results when I get them...

hydrostarr · Post by **hydrostarr** » September 4th, 2010, 4:43 pm

Update:

imap-upload is problematic in that: 1) it doesn't recurse through folders, 2) it doesn't create folders, and 3) it doesn't check to see if the email/msg has already been uploaded. (#3 may be a problem for every tool...but for the per-folder-only tools like imap-upload, it's particularly problematic, especially given I've already synced/uploaded thousands/millions of emails already, would prefer not to have to resync all of them...but if I must I must.) fwiw, imap-upload do seem quite trustworthy, just limited.

Some of the other tool(s) have save limitation(s), namely per-folder-only operation. (Yeah, I could scriptware on top of the tool to recurse through files/dirs...and I do have dual-boot, linux and windows, system with ext3 data directory that could easily do this...but I'd prefer not to go this far if I can avoid it.)

This is looking like the most-attractive next step/attempt:

tanstaafl wrote:You could run a local IMAP server that supports mbox files. Then you could use either IMAPSize or imapsync to move the folders between servers. The problem is the only easy to use/install ones I know of don't support mbox files. http://wiki.zimbra.com/wiki/Mail_Migrat ... al_Folders says "The easiest way I found for importing a lot of Thunderbird local folders is to set up a temporary dovecot server, convert the thunderbird mail stores to linux mbox mailstores and then imapsync it ". In this case you'd imapsync it to your real imap server, rather than zimbra.

Post by **tanstaafl** » September 4th, 2010, 5:45 pm

Unfortunately, Dovecot doesn't run under Windows . You could try running it using Cygwin (to emulate the UNIX environment) , http://wiki.dovecot.org/OSCompatibility says that works with one limitation that you might be able to workaround by starting the mail server with inetd. However, that requires you to learn three separate tools just to try it out.

http://kb.mozillazine.org/IMAP_servers lists some IMAP servers that work with Thunderbird. The problem is finding one that supports both windows, mboxs and is easy to install. Smartermail for example uses mbox files and runs under Windows, but installs a ton of stuff because its really a suite (not a simple IMAP server). Mercury/32 might be the easiest, but I'm concerned about its history of interoperability problems

Post by **tanstaafl** » September 7th, 2010, 4:58 am

I asked for advice in this thread in the official support forums but so far nobody has replied. A couple of people flagged it as a problem they're running into too.

Any luck finding an easy to setup IMAP server?

hydrostarr · Post by **hydrostarr** » September 21st, 2010, 7:28 am

Thanks tanstaafl, big time. Haven't yet tried to install an IMAP server. I may try a Linux dual-boot to solve this problem. Think that will work? eg, possibly by running Thunderbird (temporarily of course) in Linux to do the conversion, then switch back to Windows? Will the Linux (assuming there is a TB Linux port) and Windows versions of Thunderbird speak exactly the same dataset/files?

(While I'm not currently setup to do this, all of my Thunderbird data and profile is in an ext3 partition which I mount in Windows with http://www.fs-driver.org/ in large part so that I can manage my data in any OS without fear of ntfs-conversion problems. I've dual-booted before with this configuration, I just don't have it setup in my current laptop, but relatively easy to do.)

Post by **tanstaafl** » September 21st, 2010, 3:00 pm

You can move the profile to Linux and use it as is. The only issue is whether you have any add-ons that are operating system specific such as Lightning. Few are other than it and some themes.

http://kb.mozillazine.org/Moving_from_Windows_to_Linux explains how to install Thunderbird under Linux. I recommend you install the Dovecot IMAP server, making certain you specify mbox files (since it also supports maildir).

https://help.ubuntu.com/community/Dovecot

hydrostarr · Post by **hydrostarr** » May 7th, 2011, 6:06 pm

fyi, hope to attempt the Dovecat-based method pretty soon as per:
http://getsatisfaction.com/mozilla_mess ... ly_5499099

hydrostarr · Post by **hydrostarr** » May 8th, 2011, 2:00 pm

Nothing all this stuff here for easy reference:
http://wiki.zimbra.com/wiki/Mail_Migrat ... al_Folders
http://wiki.zimbra.com/wiki/Guide_to_imapsync
http://wiki.zimbra.com/wiki/Mail_Migrat ... _Method.29

POP3-to-IMAP conversion for large email sets, folders: help?

POP3-to-IMAP conversion for large email sets, folders: help?

Re: POP3-to-IMAP conversion for large email sets, folders: h

Re: POP3-to-IMAP conversion for large email sets, folders: h

Re: POP3-to-IMAP conversion for large email sets, folders: h

Re: POP3-to-IMAP conversion for large email sets, folders: h

Re: POP3-to-IMAP conversion for large email sets, folders: h

Re: POP3-to-IMAP conversion for large email sets, folders: h

Re: POP3-to-IMAP conversion for large email sets, folders: h

Re: POP3-to-IMAP conversion for large email sets, folders: h

Re: POP3-to-IMAP conversion for large email sets, folders: h

Re: POP3-to-IMAP conversion for large email sets, folders: h

Re: POP3-to-IMAP conversion for large email sets, folders: h

Re: POP3-to-IMAP conversion for large email sets, folders: h

Re: POP3-to-IMAP conversion for large email sets, folders: h

Re: POP3-to-IMAP conversion for large email sets, folders: h