Trouble moving a large number of emails...

Discussion of general topics about Mozilla Thunderbird
Post Reply
Tantamount
Posts: 2
Joined: March 26th, 2016, 6:58 pm

Trouble moving a large number of emails...

Post by Tantamount »

This may be outside of the scope of what Thunderbird can do, but I thought I'd ask here in case people know some tricks...

I'm attempting to migrate our Exchange server email archive over to Linux so that I can do some regex searching for a legal discovery request. There are over 744k of emails that I need to transfer. What I tried doing was to create two accounts in Thunderbird, a source and destination, selecting all in the source account, clicking "Move", and choosing a folder in the destination account.

During the transfer, I found that the 32-bit version fairly quickly ate up all the memory it could allocate and subsequently the move slowed until it eventually crashed. I tried a 64-bit build (Daily) and that was able to get much farther, but after 4 days, I noticed it was using 20gigs of ram and the transfer had ceased midway. The program is still responding to user input (albeit slowly). Meanwhile, it looks like none of the emails that have transferred have been deleted from the source, so I'm concerned that if I have to abort this and re-start, I'll have to deal with duplicates at the destination or completely restart the process.

I assume that my problems are the result of either message caching or perhaps code that tries to follow threads of communication (Doing this for all emails from all users for a year's worth of emails). Is there any way to "dumb down" Thunderbird so that it just blindly downloads emails from one account to the other?

Perhaps I should just write a perl script instead?
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Trouble moving a large number of emails...

Post by tanstaafl »

I'd never move a tremendous amount of messages in one step if only because it makes it too hard to verify that you didn't lose anything. I suggest you move each folder separately. That should also reduce the resources used.

If you're using IMAP accounts consider using a Windows utility such as IMAPSize to move the messages instead. See http://www.broobles.com/imapsize/ . There are also GitHub projects like offlineimap that requires cygwin and Python 2.7 be installed. See http://www.offlineimap.org/
Post Reply