Thunderbird Sometimes Re-Downloading All Emails

User Help for Mozilla Thunderbird
Post Reply
BigbirdPhila
Posts: 9
Joined: July 28th, 2017, 9:14 am

Thunderbird Sometimes Re-Downloading All Emails

Post by BigbirdPhila »

Windows 10/Thunderbird 52.6.0

I'm using Thunderbird to access two email accounts via POP3. Occasionally Thunderbird re-downloads all the emails on the email server (pop.verizon.net, now being managed by AOL, a subsidiary of Verizon). It does not happen to both accounts at the same time - it happens randomly to each account. They all come in as unread, so I can filter by "unread" and then delete them. But, it's becoming a "PITA". Did not seem to happen when I first installed Thunderbird a year or so ago, but now it seems to happen once a week to at least one of the accounts.

I have both accounts set to leave the messages on the server. I want to keep it that way to have an archive of all my emails (AOL gives us unlimited server space). I also let Thunderbird periodically compress its files (I have it set to query me beforehand). Compression seems to occur w/o errors. I am using Windows Defender as my anti-virus program and, as far as I can tell, it does not get involved with scanning emails.

Articles I have found on the support forums recommend deleting "popstate.dat" and "inbox.msf", but I'm not sure they are describing my problem. I don't get occasional multiple copies of one or two emails; I get occasional re-downloading of my whole inbox. And, as long as I don't get inundated with hundreds/thousands of repeat emails, Thunderbird seems to be running fine. I'm not losing any emails or attachments as far as I can tell, and not getting random duplicates of one or two emails.

I have an Android phone and an iPhone both configured to access the same accounts via POP3 and they do *not* exhibit this problem, so I'm sure it's a Thunderbird issue.

Any help most appreciated.
User avatar
DanRaisch
Moderator
Posts: 127188
Joined: September 23rd, 2004, 8:57 pm
Location: Somewhere on the right coast

Re: Thunderbird Sometimes Re-Downloading All Emails

Post by DanRaisch »

I have an Android phone and an iPhone both configured to access the same accounts via POP3 and they do *not* exhibit this problem, so I'm sure it's a Thunderbird issue.
No, that's not a valid evaluation. Both phones are likely accessing the accounts as IMAP type accounts due to the lower storage space required on the phone side with that type of accounts. IMAP accounts don't use a popstate.dat file and the messages are (in general) stored on the server, not on the local computer/tablet/phone.

The redownloading issue would usually be associated with a corrupt popstate.dat file as that is the file used by Thunderbird to determine which messages have been downloaded from the server. If you rename or delete that file, TB will believe that there have been no messages downloaded from the server for the account and download all messages from the server inbox folder. If deleting the popstate.dat file in the future, do so with TB closed. Move all messages from the Inbox folder of the POP account to sub-folders of the Thunderbird Local Folders pseudo-account before deleting the .dat file. That will make it easy to delete all previously downloaded messages in the Inbox after launching with the redownloaded messages.

To avoid this issue, you could periodically access the account webmail pages through a browser and move all but the most current messages to server folders other than the inbox. That would make them invisible/inaccessible to Thunderbird so they could not be downloaded again but would still be available to you on the server.

Are you using Ccleaner or any similar clean up utility on that system? If so, that could be a factor in any corruption of the popstate.dat file. It would be best to switch off any activities relating to Thunderbird in those utilities and let TB handle any required clean up of files on its own.
BigbirdPhila
Posts: 9
Joined: July 28th, 2017, 9:14 am

Re: Thunderbird Sometimes Re-Downloading All Emails

Post by BigbirdPhila »

Thanks for your quick response.

Both phones are using POP3 to access both email accounts. The email apps I'm using take care of memory usage on their own (basically by just providing "windows" into the complete inbox on the server). Problem has never happened with either phone.

I already do periodically move older emails to a another folder on the server via webmail. Good thinking (on both of our parts!). By doing this, I've managed to limit the "pain" to a few hundred emails with each occurrence. But, I'm looking for the "right" solution.

You might have hit on something regarding CCleaner. I use it once a week, and Thunderbird is on its list of applications it "cleans up" after. It doesn't happen immediately after I run CCleaner, and it never happens simultaneously on both accounts. But, we're dealing with complicated software here. I'm going to disable CCleaner's Thunderbird cleanup & watch what happens. It will probably take a couple weeks of observations. Will post my results.

Am saving deleting "popstate.dat" as a last resort. I don't think it's corrupted because Thunderbird runs fine until the moment it decides that it never had downloaded anything. Then all the emails start coming down in an avalanche.

Will try the CCleaner idea first. Thanks.
User avatar
DanRaisch
Moderator
Posts: 127188
Joined: September 23rd, 2004, 8:57 pm
Location: Somewhere on the right coast

Re: Thunderbird Sometimes Re-Downloading All Emails

Post by DanRaisch »

Both phones are using POP3 to access both email accounts. The email apps I'm using take care of memory usage on their own
That doesn't sound like POP access. With POP, the client downloads all of the message information from the server and stores it locally. That's not usually something that mobile devices usually want to do. Wikipedia notes that "Although most POP clients have an option to leave mail on server after download, e-mail clients using POP generally connect, retrieve all messages, store them on the client system, and delete them from the server."
Am saving deleting "popstate.dat" as a last resort. I don't think it's corrupted because Thunderbird runs fine until the moment it decides that it never had downloaded anything. Then all the emails start coming down in an avalanche.
And that's exactly a symptom of a popstate.dat file that can not be read by Thunderbird at all or that contains garbled information. The result is Thunderbird's "assumption" that none of the messages currently in the server inbox have ever been downloaded to the client (TB) so it triggers the download of all of those message. But I agree that you should defer renaming or deleting the popstate.dat file for now as it would force you to deal with the download of everything currently in that server inbox folder.
User avatar
makaiguy
Posts: 16878
Joined: November 18th, 2002, 6:44 pm
Location: Somewhere in SE USA
Contact:

Re: Thunderbird Sometimes Re-Downloading All Emails

Post by makaiguy »

If you are having this happen frequently, it indicates that something is causing the popstate.dat files for one or more of your accounts to be deleted or corrupted repeatedly.

Two possibilities come to mind:

1) Repeatedly shutting off your computer by the power switch instead of an orderly software shutdown via Windows. This can cause corruption of files programs may have open at the time.

2) Allowing CCleaner or other cleanup program to touch Thunderbird's files. Ccleaner especially has been found to mess with Thunderbird and delete files necessary for proper TBird operation. Ditto for Firefox.
Doug Wilson
Win10 64bit: FF 115.0.02 64bit, TB 102.12.0 32-bit ║ Android 13/10: FF 115.2.0/115.0.1 ║ No TB for Android available, dammit!
What a fool believes he sees, no wise man has the power to reason away - Doobie Brothers
BigbirdPhila
Posts: 9
Joined: July 28th, 2017, 9:14 am

Re: Thunderbird Sometimes Re-Downloading All Emails

Post by BigbirdPhila »

Ok. Disabled all Thunderbird cleanup activities in CCleaner. It took a couple days, but the problem occurred again. Thunderbird was running fine all day since about 8 AM, fetching about 50 emails. Then, around 3 PM, it spontaneously started to re-download my whole inbox (1500+ emails). It happened to only one account; the other account is running normally -- only downloading new emails.
BigbirdPhila
Posts: 9
Joined: July 28th, 2017, 9:14 am

Re: Thunderbird Sometimes Re-Downloading All Emails

Post by BigbirdPhila »

The other account just spontaneously downloaded its whole inbox. 5 days after running CCleaner (w/o Thunderbird cleanup), and 3 days after the other account spontaneously downloaded. I'm not going to run CCleaner again and watch what happens over the next few days. Maybe Thunderbird might have "fixed" itself after these incorrect downloads. I want to remove CCleaner entirely from the picture.

Saving messing around with popstate.dat and inbox.msf until later. Change one thing at a time. I want to methodically document this problem in case others hit it in the future.
BigbirdPhila
Posts: 9
Joined: July 28th, 2017, 9:14 am

Re: Thunderbird Sometimes Re-Downloading All Emails

Post by BigbirdPhila »

I tried deleting popstate.dat and inbox.msf. It takes a couple days, but the problem re-occurs.

I also tried uninstalling Thunderbird, running CCleaner file & registry cleanup, and re-installing Thunderbird. Again, it takes a couple days but the problem re-occurs.

I have submitted a bug report on Bugzilla. I will post the results here if the developers come up with anything.
User avatar
DanRaisch
Moderator
Posts: 127188
Joined: September 23rd, 2004, 8:57 pm
Location: Somewhere on the right coast

Re: Thunderbird Sometimes Re-Downloading All Emails

Post by DanRaisch »

Are you still periodically running Ccleaner? Is it set to NOT clean up after Thunderbird?

I ask because all of the other reports of TB downloading POP messages multiple times seem to be resolved for the long term by deleting the original(and probably corrupted) popstate.dat file. I can't recall any other reports of the issue happening over and over again despite replacement of that file.

Given the lack of other reports of the same problem, I'll be surprised if much comes of a bug report on the issue. It would be extremely hard to "correct" something that takes time to reoccur and which hasn't been replicated by other users.
BigbirdPhila
Posts: 9
Joined: July 28th, 2017, 9:14 am

Re: Thunderbird Sometimes Re-Downloading All Emails

Post by BigbirdPhila »

See my post of April 10th. All CCleaner cleanup involving Thunderbird has been disabled. I have been methodical in trying to track down the problem by sequentially trying one thing at a time.

I think the problem may be related to the fact that I'm using POP3 to access the emails w/o deleting them from the server. It might not happen with IMAP or other POP3 scenarios. I have submitted an exhaustive description of the problem on Bugzilla. I'm an ex-software developer myself, and if someone gave me the details that I gave on Bugzilla I would have a lot to go on if I knew the internals of the product -- which the Bugzilla people do.
User avatar
DanRaisch
Moderator
Posts: 127188
Joined: September 23rd, 2004, 8:57 pm
Location: Somewhere on the right coast

Re: Thunderbird Sometimes Re-Downloading All Emails

Post by DanRaisch »

I use POP in the same way and have not experienced the problem.
wsmwk
Posts: 2833
Joined: December 7th, 2004, 6:52 am
Contact:

Re: Thunderbird Sometimes Re-Downloading All Emails

Post by wsmwk »

BigbirdPhila wrote:I have submitted a bug report on Bugzilla. I will post the results here if the developers come up with anything.
https://bugzilla.mozilla.org/show_bug.cgi?id=1461194
Post Reply