I have a google mail account with bunch of folders, I enabled IMAP and maximum 5000 message accessible via IMAP; configured TB, works fine except it takes way way too much time to open the folder and show a message which has not been downloaded yet - TB either says "Loading message..." or "Opening folder", in any case it takes up to a minute to show the message.
The "Activity Manage" is not showing anything about that folder. I got a feeling that every time TB reconnects to the gmail server, it silently sychronizes all the folders and only then downloads a message but I cannot find any way to confirm this - is there an addon or a secret debug hotkey? Are things like keep-connection-alive/imap-timeout/etc configurable? Anything else to tune? I went through all steps to "accelerate TB" such as clearing disk caches, etc, it did not change things even a bit.
On the exact same machine with the exact same account Evolution opens mails faster.
Thunderbird 45.4.0, fedora 23, Thinkpad X1 16GB RAM, SSD, core i7 - pretty fast hardware.
Thanks.
Opening a folder takes forever
-
tanstaafl
- Moderator
- Posts: 49647
- Joined: July 30th, 2003, 5:06 pm
Re: Opening a folder takes forever
Try moving most of the messages that you want to keep from your inbox to other folders, and then empty the trash and compact the folders (file -> compact folders). My rule of thumb is that if the inbox has more than about 300 messages I need to clean it up.
By default Thunderbird will create offline folders (they're stored as mbox files just like POP accounts) that mirror the contents of the remote folders on the IMAP mail server, and global search/indexing will build a search index to make global search fast. Usually you won't notice this going on the background after you've first used Thunderbird for about an hour because then its caught up and only has to download/index new messages. If you'd like to minimize the profile size and increase performance by only downloading the headers for messages to the hard disk see http://kb.mozillazine.org/Minimize_the_ ... _a_profile .
In either case Thunderbird always fetches the contents of the message from the remote folder when you open it. The main tradeoff in minimizing the size of the profile is that requires disabling global search and using one of the two other search methods. The offline folders aren't a reliable backup because they're kept in sync with the remote folder, they're mainly there to help global search and in case you want to read old messages while working offline. I never understood why the developers didn't use the offline folders as a cache, like many other email clients do.
Having "All Mail" as a offline folder doubles the amount of messages that are downloaded because it contains a copy of all of the messages. If you don't want to give up global search think about unsubscribing it, exiting Thunderbird, and then deleting both the "All Mail." and "All Mail.msf" files.
If none of this helps see http://kb.mozillazine.org/Performance_-_Thunderbird
By default Thunderbird will create offline folders (they're stored as mbox files just like POP accounts) that mirror the contents of the remote folders on the IMAP mail server, and global search/indexing will build a search index to make global search fast. Usually you won't notice this going on the background after you've first used Thunderbird for about an hour because then its caught up and only has to download/index new messages. If you'd like to minimize the profile size and increase performance by only downloading the headers for messages to the hard disk see http://kb.mozillazine.org/Minimize_the_ ... _a_profile .
In either case Thunderbird always fetches the contents of the message from the remote folder when you open it. The main tradeoff in minimizing the size of the profile is that requires disabling global search and using one of the two other search methods. The offline folders aren't a reliable backup because they're kept in sync with the remote folder, they're mainly there to help global search and in case you want to read old messages while working offline. I never understood why the developers didn't use the offline folders as a cache, like many other email clients do.
Having "All Mail" as a offline folder doubles the amount of messages that are downloaded because it contains a copy of all of the messages. If you don't want to give up global search think about unsubscribing it, exiting Thunderbird, and then deleting both the "All Mail." and "All Mail.msf" files.
If none of this helps see http://kb.mozillazine.org/Performance_-_Thunderbird
-
aik_au
- Posts: 63
- Joined: November 13th, 2016, 10:13 pm
Re: Opening a folder takes forever
300 is a very, very little number, TB should be able to do better :-/ 10 years ago, the windows-only mailer "The Bat!" could easily handle more than 300 on slower hardware...tanstaafl wrote:Try moving most of the messages that you want to keep from your inbox to other folders, and then empty the trash and compact the folders (file -> compact folders). My rule of thumb is that if the inbox has more than about 300 messages I need to clean it up.
tanstaafl wrote: By default Thunderbird will create offline folders (they're stored as mbox files just like POP accounts) that mirror the contents of the remote folders on the IMAP mail server, and global search/indexing will build a search index to make global search fast. Usually you won't notice this going on the background after you've first used Thunderbird for about an hour because then its caught up and only has to download/index new messages. If you'd like to minimize the profile size and increase performance by only downloading the headers for messages to the hard disk see http://kb.mozillazine.org/Minimize_the_ ... _a_profile .
[aik@aik lists-1.sbd]$ ls -la
[...]
-rw-------. 1 aik aik 1808258 Nov 14 18:03 watch
-rw-rw-r--. 1 aik aik 2737446 Nov 14 18:32 watch.msf
5MB, however TB says (in folder properties) - 52.5MB. Where are all these MBs stored? Everything in .thunderbird is about 140MB but if I sum up individual folder sizes reported by TB, I get lot lot more....
This is all true, I am complaining about detailstanstaafl wrote: In either case Thunderbird always fetches the contents of the message from the remote folder when you open it. The main tradeoff in minimizing the size of the profile is that requires disabling global search and using one of the two other search methods. The offline folders aren't a reliable backup because they're kept in sync with the remote folder, they're mainly there to help global search and in case you want to read old messages while working offline. I never understood why the developers didn't use the offline folders as a cache, like many other email clients do.
Basically, I pick a folder, click on a message - and I am watching - 10+ seconds for "opening folder", 2-3 seconds "logging on", 10+ seconds "loading message", then it appears.Then I just wait for a minute and simply click on the next message in the very same folder - aaaaand it starts all over again - 25+ seconds to see a new message - I cannot explain this particular effect by not compacting folders, honestly. It looks like after a minute or so the mail database gets closed.
Interesting reading, thanks.tanstaafl wrote: If none of this helps see http://kb.mozillazine.org/Performance_-_Thunderbird
- "By default it keeps up to 30 *.msf files open for 5 minutes" - can this be changed? I have lots of RAM and the laptop is not doing anything else at all so I could use all RAM for mail. I could not spot the setting for this in about:config.
- "Thunderbird appears to parse the contents of the entire Inbox folder when it downloads new mail for POP accounts".
It appears to me that it does the same for IMAP as well - but I cannot prove and tune this either...
-
tanstaafl
- Moderator
- Posts: 49647
- Joined: July 30th, 2003, 5:06 pm
Re: Opening a folder takes forever
>300 is a very, very little number, TB should be able to do better :-/ 10 years ago, the windows-only mailer "The Bat!" could easily handle more than 300 on slower hardware...
There is a big difference between what works and what's recommended. Plenty of people use Thunderbird with inboxes that have tens of thousands of messages. I'm recommending a "very, very little number" for the inbox to improve performance and reduce the risk of corruption. The inbox folder is a special case because its always opened on startup and has so many deleted messages in it. Most people don't compact it often enough.
>5MB, however TB says (in folder properties) - 52.5MB. Where are all these MBs stored? Everything in .thunderbird is about 140MB but if I sum up individual folder sizes reported by TB, I get lot lot more....
The folder properties for my Fastmail IMAP accounts inbox says its 11.4MB. The file manager says "inbox." (the mbox file that stores its messages) is 11,751KB and "inbox.msf" (the index or cache of the folder listing, which is basically just headers) is 681KB. The numbers don't match, but they're in the right ballpark. Ditto for other accounts that I look at.
Try pressing the repair folder button in the folders properties.
>This is all true, I am complaining about details Basically, I pick a folder, click on a message - and I am watching - 10+ seconds for "opening folder", 2-3 seconds "logging on", 10+ seconds "loading message", then it appears. Then I just wait for a minute and simply click on the next message in the very same folder - aaaaand it starts all over again - 25+ seconds to see a new message - I cannot explain this particular effect by not compacting folders, honestly. It looks like after a minute or so the mail database gets closed.
That's terrible. I'd assume you're running into some sort of bug. I suggest you take another look at the "unusual problems" section in http://kb.mozillazine.org/Performance_-_Thunderbird . especially: "A few users have run into a problem where every operation takes about a minute. This is due to a bug creating places.sqlite-shm and places.sqlite-wal directories in the profile. There should be files with those names, not directories. The workaround is to exit Thunderbird and delete those two directories."
I have 9 accounts (8 IMAP, 1 POP) and it typically takes 2 seconds to start Thunderbird, check for updated add-ons, check for new mail, and update all of the inbox folder listings. That's if Thunderbird.exe had been run before (after booting) so that the executable is cached in memory. I normally don't see any delay when switching folders or opening a message.
>"By default it keeps up to 30 *.msf files open for 5 minutes" - can this be changed? I have lots of RAM and the laptop is not doing anything else at all so I could use all RAM for mail. I could not spot the setting for this in about:config.
I'm not aware of a setting to control that.
>"Thunderbird appears to parse the contents of the entire Inbox folder when it downloads new mail for POP accounts".
It appears to me that it does the same for IMAP as well - but I cannot prove and tune this either...\
You keep saying your IMAP account downloads new mail. That's in a background thread, for the offline folders and global search. That normally doesn't effect your opening a new message and reading it. That fetches the message from the IMAP server, it doesn't use the downloaded messages on the hard disk.
There is a big difference between what works and what's recommended. Plenty of people use Thunderbird with inboxes that have tens of thousands of messages. I'm recommending a "very, very little number" for the inbox to improve performance and reduce the risk of corruption. The inbox folder is a special case because its always opened on startup and has so many deleted messages in it. Most people don't compact it often enough.
>5MB, however TB says (in folder properties) - 52.5MB. Where are all these MBs stored? Everything in .thunderbird is about 140MB but if I sum up individual folder sizes reported by TB, I get lot lot more....
The folder properties for my Fastmail IMAP accounts inbox says its 11.4MB. The file manager says "inbox." (the mbox file that stores its messages) is 11,751KB and "inbox.msf" (the index or cache of the folder listing, which is basically just headers) is 681KB. The numbers don't match, but they're in the right ballpark. Ditto for other accounts that I look at.
Try pressing the repair folder button in the folders properties.
>This is all true, I am complaining about details
Basically, I pick a folder, click on a message - and I am watching - 10+ seconds for "opening folder", 2-3 seconds "logging on", 10+ seconds "loading message", then it appears. Then I just wait for a minute and simply click on the next message in the very same folder - aaaaand it starts all over again - 25+ seconds to see a new message - I cannot explain this particular effect by not compacting folders, honestly. It looks like after a minute or so the mail database gets closed.That's terrible. I'd assume you're running into some sort of bug. I suggest you take another look at the "unusual problems" section in http://kb.mozillazine.org/Performance_-_Thunderbird . especially: "A few users have run into a problem where every operation takes about a minute. This is due to a bug creating places.sqlite-shm and places.sqlite-wal directories in the profile. There should be files with those names, not directories. The workaround is to exit Thunderbird and delete those two directories."
I have 9 accounts (8 IMAP, 1 POP) and it typically takes 2 seconds to start Thunderbird, check for updated add-ons, check for new mail, and update all of the inbox folder listings. That's if Thunderbird.exe had been run before (after booting) so that the executable is cached in memory. I normally don't see any delay when switching folders or opening a message.
>"By default it keeps up to 30 *.msf files open for 5 minutes" - can this be changed? I have lots of RAM and the laptop is not doing anything else at all so I could use all RAM for mail. I could not spot the setting for this in about:config.
I'm not aware of a setting to control that.
>"Thunderbird appears to parse the contents of the entire Inbox folder when it downloads new mail for POP accounts".
It appears to me that it does the same for IMAP as well - but I cannot prove and tune this either...\
You keep saying your IMAP account downloads new mail. That's in a background thread, for the offline folders and global search. That normally doesn't effect your opening a new message and reading it. That fetches the message from the IMAP server, it doesn't use the downloaded messages on the hard disk.
-
wsmwk
- Posts: 2833
- Joined: December 7th, 2004, 6:52 am
- Contact:
Re: Opening a folder takes forever
> Having "All Mail" as a offline folder doubles the amount of messages that are downloaded because it contains a copy of all of the messages. If you don't want to give up global search think about unsubscribing it, exiting Thunderbird, and then deleting both the "All Mail." and "All Mail.msf" files.
The existence of All Mail should not double disk storage. Nor should it impact the load time of a different mail folder. However, having All Mail subscribed can impact overall performance
The existence of All Mail should not double disk storage. Nor should it impact the load time of a different mail folder. However, having All Mail subscribed can impact overall performance
-
tanstaafl
- Moderator
- Posts: 49647
- Joined: July 30th, 2003, 5:06 pm
Re: Opening a folder takes forever
The last time we discussed this you said "I'm not sure what happens with messages that originate on the PC, such as Sent." That matches the data I've seen, where copies of every sent message also shows up in the All Mail mbox folder. However, its Gmail's SMTP server, not Thunderbird, which is saving that copy of the message. When an architect says in the bug report that "This is doable, but tricky" that's not a good sign.wsmwk wrote:The existence of All Mail should not double disk storage.
In bug report Use Gmail IMAP Extension of X-GM-MSGID, X-GM-THRID, X-GM-LABELS in addition to XLIST they mentioned that they tried to avoid duplicating messages that were already downloaded from Gmail. However, see http://forums.mozillazine.org/viewtopic ... #p14437739 for a example showing that does not work in practice.
-
aik_au
- Posts: 63
- Joined: November 13th, 2016, 10:13 pm
Re: Opening a folder takes forever
> That's terrible. I'd assume you're running into some sort of bug. I suggest you take another look at the "unusual problems" section in http://kb.mozillazine.org/Performance_-_Thunderbird .
Not this time:
[aik@aik gutukncm.default]$ ls -la places.sqlite
-rw-r--r--. 1 aik aik 2260992 Nov 15 16:05 places.sqlite
> I have 9 accounts (8 IMAP, 1 POP) and it typically takes 2 seconds to start Thunderbird, check for updated add-ons, check for new mail, and update all of the inbox folder listings. That's if Thunderbird.exe had been run before (after booting) so that the executable is cached in memory. I normally don't see any delay when switching folders or opening a message.
Well, I compacted folders, and it was quite fast for half a day and then it became slow again for 10 minutes or so, then TB seems to have finished whatever it was doing - and this is my problem, if I knew what it is doing at any given time, I could debug it....
> You keep saying your IMAP account downloads new mail. That's in a background thread, for the offline folders and global search. That normally doesn't effect your opening a new message and reading it. That fetches the message from the IMAP server, it doesn't use the downloaded messages on the hard disk.
It feels like until TB updates the message list, it won't download a message, simply because after message list update, the selected message may not be on the server any more (deleted via webinterface or the server decided to hide it).
Not this time:
[aik@aik gutukncm.default]$ ls -la places.sqlite
-rw-r--r--. 1 aik aik 2260992 Nov 15 16:05 places.sqlite
> I have 9 accounts (8 IMAP, 1 POP) and it typically takes 2 seconds to start Thunderbird, check for updated add-ons, check for new mail, and update all of the inbox folder listings. That's if Thunderbird.exe had been run before (after booting) so that the executable is cached in memory. I normally don't see any delay when switching folders or opening a message.
Well, I compacted folders, and it was quite fast for half a day and then it became slow again for 10 minutes or so, then TB seems to have finished whatever it was doing - and this is my problem, if I knew what it is doing at any given time, I could debug it....
> You keep saying your IMAP account downloads new mail. That's in a background thread, for the offline folders and global search. That normally doesn't effect your opening a new message and reading it. That fetches the message from the IMAP server, it doesn't use the downloaded messages on the hard disk.
It feels like until TB updates the message list, it won't download a message, simply because after message list update, the selected message may not be on the server any more (deleted via webinterface or the server decided to hide it).
-
tanstaafl
- Moderator
- Posts: 49647
- Joined: July 30th, 2003, 5:06 pm
Re: Opening a folder takes forever
Try temporarily disabling offline folders and global search/indexing. See if it makes a difference. That doesn't delete the offline folders and you don't need new messages to be downloaded to the offline folders in order to read them.
If you had deleted a message via webmail it would be deleted in the offline folder when Thunderbird synched the remote and offline folders, as part of checking for new mail.
If you had deleted a message via webmail it would be deleted in the offline folder when Thunderbird synched the remote and offline folders, as part of checking for new mail.
-
wsmwk
- Posts: 2833
- Joined: December 7th, 2004, 6:52 am
- Contact:
Re: Opening a folder takes forever
> I have a google mail account with bunch of folders, I enabled IMAP and maximum 5000 message accessible via IMAP; configured TB, works fine except it takes way way too much time to open the folder and show a message which has not been downloaded yet - TB either says "Loading message..." or "Opening folder", in any case it takes up to a minute to show the message.
Is the "watch" folder you previously listed the size for one of the folders where you see this bad performance?
Does this happen to every folder, regardless of size?
What happens if you right+click a problem folder and have it open in tab (don't go to the tab), then in the current tab click a different folder, and to back to the problem folder?
Is the "watch" folder you previously listed the size for one of the folders where you see this bad performance?
Does this happen to every folder, regardless of size?
What happens if you right+click a problem folder and have it open in tab (don't go to the tab), then in the current tab click a different folder, and to back to the problem folder?
-
aik_au
- Posts: 63
- Joined: November 13th, 2016, 10:13 pm
Re: Opening a folder takes forever
Offline folders and global search has always been disabled.
This slow behaviour happens to every folder. Right now I am reading a folder - opened a mail, watched TB doing "sending login information", "opening folder", "loading message" - first two took 30 seconds or so. Then if I go to next message within a minute - it is quick. If I read a message for more than a minute - TB starts the dance (logging - opening - downloading) all over again :-E
> What happens if you right+click a problem folder and have it open in tab (don't go to the tab), then in the current tab click a different folder, and to back to the problem folder?
I get to the original folder and see the message which was open when I right+clicked. Then I click on another mail in the same folder, observed "Loading message" for 30 seconds, then status became clear, then 30 more seconds - and only then I got my new mail opened. Weird. Feels like updating message list has higher priority than loading a message.
This slow behaviour happens to every folder. Right now I am reading a folder - opened a mail, watched TB doing "sending login information", "opening folder", "loading message" - first two took 30 seconds or so. Then if I go to next message within a minute - it is quick. If I read a message for more than a minute - TB starts the dance (logging - opening - downloading) all over again :-E
> What happens if you right+click a problem folder and have it open in tab (don't go to the tab), then in the current tab click a different folder, and to back to the problem folder?
I get to the original folder and see the message which was open when I right+clicked. Then I click on another mail in the same folder, observed "Loading message" for 30 seconds, then status became clear, then 30 more seconds - and only then I got my new mail opened. Weird. Feels like updating message list has higher priority than loading a message.
Last edited by aik_au on November 15th, 2016, 8:55 pm, edited 1 time in total.
-
tanstaafl
- Moderator
- Posts: 49647
- Joined: July 30th, 2003, 5:06 pm
Re: Opening a folder takes forever
Does "help -> restart with add-ons disabled" make a difference? That will temporarily disable your extensions, theme and the optional userchrome.css file if you have one.
-
aik_au
- Posts: 63
- Joined: November 13th, 2016, 10:13 pm
Re: Opening a folder takes forever
No, not really. While connection is alive, it is all fast, once it is closed (after a minute or so), reopening is slow. Quite often after downloading a message I see (in the status bar) that some messages were deleted - this is happening as the actual folder in gmail contains >20000 messages (linux kernel dev maillist), IMAP on gmail exposes only 5000 so I believe resynching these 5000 message headers is happening every reconnect and this takes time (even though I think that 5000 is not that much really)...tanstaafl wrote:Does "help -> restart with add-ons disabled" make a difference? That will temporarily disable your extensions, theme and the optional userchrome.css file if you have one.
-
tanstaafl
- Moderator
- Posts: 49647
- Joined: July 30th, 2003, 5:06 pm
Re: Opening a folder takes forever
I don't know anybody else who has set "Limit IMAP folders to no more than this many messages: X" in Gmail's Forwarding and POP/IMAP settings. I would expect that to increase performance, but that's just an assumption. It's also possible it works well with most email clients but not with Thunderbird due to some incompatibility in Thunderbird. I suggest you experiment with disabling that feature and see if it makes it better or worse.
-
wsmwk
- Posts: 2833
- Joined: December 7th, 2004, 6:52 am
- Contact:
Re: Opening a folder takes forever
Perhaps you are seeing https://bugzilla.mozilla.org/show_bug.cgi?id=816327
-
tanstaafl
- Moderator
- Posts: 49647
- Joined: July 30th, 2003, 5:06 pm
Re: Opening a folder takes forever
Wayne:
On my system mail.server.default.autosync_offline_stores is not defined. I also don't have a mail.server.serverX.autosync_offline_stores defined, where X is the gmail server id . mail.imap.use_status_for_biff is still true. So none of the settings needed for auto sync seem to exist. What settings do you think might be causing the problem for aik_au ?
Can you point me to the appropriate bug reports to help me update http://kb.mozillazine.org/Offline_folders#AutoSync ? The bug report you mentioned says that description is a bit dated. That comment was three years ago.
On my system mail.server.default.autosync_offline_stores is not defined. I also don't have a mail.server.serverX.autosync_offline_stores defined, where X is the gmail server id . mail.imap.use_status_for_biff is still true. So none of the settings needed for auto sync seem to exist. What settings do you think might be causing the problem for aik_au ?
Can you point me to the appropriate bug reports to help me update http://kb.mozillazine.org/Offline_folders#AutoSync ? The bug report you mentioned says that description is a bit dated. That comment was three years ago.