Downloaded folders much larger than on server

User Help for Mozilla Thunderbird
lowraxe
Posts: 17
Joined: March 27th, 2017, 8:38 am

Downloaded folders much larger than on server

Post by lowraxe »

I am using a pretty standard installation, with five accounts in my setup. Three of the accounts are basically archive or storage accounts that are shared between my computers. I have noticed that these folders on my computer continue to grow after being initially downloaded though. Right now, I have a folder for the larger account on my local drive that clocks in at 58.4GB. In Thunderbird, the same account indicates that it should be about 18.9GB. In another case the account folder is 2.03GB on the hard drive, but 793MB in Thunderbird. Incidentally, the values that Thunderbird reports are quite similar to the values on the server where the IMAP store is actually kept.

I compact folders regularly, so that's not the issue. Anyone out there have any ideas?
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Downloaded folders much larger than on server

Post by tanstaafl »

More details please. Without a screen shot of a directory listing its tough to tell if are *.tmp files leftover due to interrupted attempts at compacting or maybe duplicates of mbox files in your accounts "local directory". It sounds like something is bloating it but without examples it's tough to guess what might be causing it.

http://kb.mozillazine.org/Posting_a_scr ... _the_forum
lowraxe
Posts: 17
Joined: March 27th, 2017, 8:38 am

Re: Downloaded folders much larger than on server

Post by lowraxe »

Sorry it took me a while to get back on this. I was away from my computer for a few days.

Just before I made this post, I deleted the Windows folder from my computer and allowed Thunderbird to re-download the emails from the IMAP store. At the moment, all the Windows folder sizes are approximately correct. It's not clear to me what is causing things to blow up one way or another, or when it happens. However, in response to your post, I can say that there were no *.tmp files in the bloated folder.

When the problem occurs, the individual mbox files (no extension) within the Windows folder are considerably larger than the size of the IMAP folder as indicated in Thunderbird.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Downloaded folders much larger than on server

Post by tanstaafl »

I suggest you think about checking Tools -> Account Settings -> Server Settings -> Clean up ("Expunge") Inbox on Exit and using the config editor to set mail.imap.expunge_option to 2. That will expunge (physically remove) any deleted messages when there are at least 20 messages waiting to be expunged. It's not clear why your mbox files aren't syncing correctly, but it might help as expunging works differently than compacting.

http://kb.mozillazine.org/Deleting_mess ... P_accounts
lowraxe
Posts: 17
Joined: March 27th, 2017, 8:38 am

Re: Downloaded folders much larger than on server

Post by lowraxe »

I use the MultiXpunge add-in at least daily (usually a few times a day) to clean everything up. I think it basically compacts every folder. Can you explain what the difference is between expunging and compacting? I tried to look that up and couldn't find any clear explanation of the difference.
lowraxe
Posts: 17
Joined: March 27th, 2017, 8:38 am

Re: Downloaded folders much larger than on server

Post by lowraxe »

OK, the folders blew up again, so I have images of the following:
1) The folder pane on the left side of Thunderbird with the size column showing.
2) A screenshot of the properties window for the folder where my emails are stored on the local hard drive.

ImageImage

You can see here that Thunderbird indicates the entire DSI shared account should be 19082MB. However, on the hard drive, it is 32.5 GB.

To show a more specific example, the following two pictures show that the highlighted folder in Thunderbird should be 16MB, but the corresponding MSF file on my local hard drive is 33.1MB.

ImageImage

So, somehow, the individual MSF files are duplicating information (I think). Any ideas why this may be happening?
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Downloaded folders much larger than on server

Post by tanstaafl »

Even zooming to 300% I can only read the text in the first image (which I could read without zooming).
User avatar
DanRaisch
Moderator
Posts: 127187
Joined: September 23rd, 2004, 8:57 pm
Location: Somewhere on the right coast

Re: Downloaded folders much larger than on server

Post by DanRaisch »

By the way, MSF files are not actually message files. They are, for lack of a better term, index files.
lowraxe
Posts: 17
Joined: March 27th, 2017, 8:38 am

Re: Downloaded folders much larger than on server

Post by lowraxe »

Sorry the images didn't link well. They're actually posted in another forum where I've posed this same question, so you can visit this link https://support.mozilla.org/en-US/quest ... er-1166007 to see them a bit more clearly.

The MSF files appear to be the containers for all the emails in any given folder. Is that an inaccurate assessment? In the case shown above, that container is, on my local drive, twice the size of the folder reported in Thunderbird.
User avatar
DanRaisch
Moderator
Posts: 127187
Joined: September 23rd, 2004, 8:57 pm
Location: Somewhere on the right coast

Re: Downloaded folders much larger than on server

Post by DanRaisch »

The MSF files appear to be the containers for all the emails in any given folder. Is that an inaccurate assessment?
Yes, that is inaccurate. The files that actually contain Thunderbird's messages are the ones with the same name as the .MSF files but with no file name extension.
lowraxe
Posts: 17
Joined: March 27th, 2017, 8:38 am

Re: Downloaded folders much larger than on server

Post by lowraxe »

Sorry, it is the extensionless files that correspond to each MSF that I meant to say are the containers. It appears these files are becoming larger than what is contained in the Thunderbird folder.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Downloaded folders much larger than on server

Post by tanstaafl »

Your screen shots seem to state that e:\email is 32.5GB according to windows explorer. I don't see why Thunderbird claiming the DSI account is 19GB is a problem. That is far less than what appears to be your profile size (32.5GB). None of the screen shots appear to show a 58.4GB folder (either from Thunderbird's perspective or windows explorer's perspective).

What would help is a simple text table (not a screen shot) with three columns. It should have the size according to TB, the size according to windows explorer and the location of the profile, each account, and any extremely large folders.

In windows explorer the size of a mail folder would be the size of the file named after the folder, with no file extension. i.e. for the inbox folder, the mbox file "inbox.".
lowraxe
Posts: 17
Joined: March 27th, 2017, 8:38 am

Re: Downloaded folders much larger than on server

Post by lowraxe »

The 58.4GB number was what the folder had blown up to when I originally made this post. Right after posting, I deleted the entire email folder and re-downloaded it from the IMAP server. For a week or so, the size of the folder on my local hard drive matched what was shown in Thunderbird, but when it blew up again, I posted the screen shots.

The 32.5GB is not the profile size. It is just the size of the email store on the local hard drive. I have five accounts in Thunderbird. Two of them are kept in my profile folder within %appdata%, but the other three are kept in different locations on a local drive. The local hard drive folder I show in the second screenshot corresponds to the DSI shared account. The sizes should be the same, and after deleting the folder and re-downloading everything in Thunderbird, they typically are. The problem is that after a few weeks, the local hard drive store inflates inexplicably. As I showed in the second set of screenshots, a particular example within the account shows that the S10012-0001 Shipping folder is 16MB but the corresponding file in the E drive is over 33MB. This indicates to me that the individual files within the email store folder are inflating, but I just don't understand why, which is why I'm reaching out to all you smart folks here :)
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Downloaded folders much larger than on server

Post by tanstaafl »

Why do you have offline folders enabled? Is it to read them offline or to have something that you can backup?

What version are you using? One possible workaround would be to convert the account from mbox files to maildir (single file per messages). The maildir code is not as mature (biggest risk seems to be when moving a message between accounts using maildir and accounts using mbox or vice versa) but eliminates many potential problems because you no longer have multiple messages in a single file.

https://wiki.mozilla.org/Thunderbird/Maildir (out of date but useful background)

Have you verified the global search/indexing database (global-messages-db.sqlite) isn't growing out of control too? It should be no larger (usually smaller) than the combined size of the remote folders and whatever POP accounts/Local Folders you have (though I think it has a minimum size of about 10MB).
lowraxe
Posts: 17
Joined: March 27th, 2017, 8:38 am

Re: Downloaded folders much larger than on server

Post by lowraxe »

Do you mean the "Local Folders" at the bottom of my list? I don't actually use this folder. If you are instead asking why do I keep my email stored locally, the answer is that we have set up our system so that all emails are shared across our organization. We, of course, receive emails at our own individual email addresses, through the corresponding account, but an email that is related to a given project is then moved to a folder in the DSI shared account IMAP so that everyone in the company has access to it. The DSI partnership and Personal IMAPs that you see in my screenshot are similar. Every computer in our organization has all of the emails from the DSI shared account saved to their local system as a redundant backup.

I'm running two different versions (52.9.1 on my laptop and 60.2.1 on my desktop). The problem occurs on both installations, and has been occurring through multiple upgrades.

I used to have all our systems on maildir, but as you mentioned, moving messages around was causing too many problems. And as I recall, I had similar issues with the local stores inflating over time. I also found that offline access didn't work well with maildir.

My global-messages-db.sqlite file is 214.6MB on both computers. Far smaller than the combined size of all my IMAPs.
Post Reply