Download full IMAP message body when new mail arrives

User Help for Mozilla Thunderbird
Post Reply
esperling
Posts: 7
Joined: January 14th, 2019, 10:27 pm

Download full IMAP message body when new mail arrives

Post by esperling »

When thunderbird discovers a new message in an IMAP folder, it downloads the header immediately and notifies of new mail, as expected.

As I'm on rather slow internet, I'd like the full message to also be downloaded immediately, so that I don't have to wait 10 seconds whenever I click on a message. So far, I haven't found the correct way.

What I've tried:

- All relevant folders are subscribed
- All relevant folders have "Select this folder for offline use" checked
- All relevant folders have "When getting new messages for this account, always check this folder" checked
- All accounts have "Keep messages in all folders for this account on this computer" checked
- All accounts have "Synchronize all messages locally regardless of age" selected
. mail.imap.mime_parts_on_demand = false
- mail.imap.use_status_for_biff = false (Tried both settings, no difference)
- mail.server.default.check_all_folders_for_new = true
- mail.server.default.mime_parts_on_demand = false
- mailnews.offline_sync_mail = true

When I manually choose Sync Now, the message bodies are downloaded. What I'm trying to achieve is to have the message bodies downloaded automatically when the message arrives.

Server's Exchange 2016, accessed via IMAP.

Any suggestion on how I can enable this behavior, or if it "should work" given my current settings, what can I do to track down why it doesn't work for me?
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Download full IMAP message body when new mail arrives

Post by tanstaafl »

http://kb.mozillazine.org/Entire_messag ... AP_message

Thunderbird can cache a message in a IMAP account but in practice that has very little effect on performance, and wouldn't effect the initial delay in fetching the message from the IMAP server. You disabled MIME Parts on Demand, which optimizes how much of a message is fetched when you open it. If you have a slow Internet connection you should keep it enabled. For example it is very useful to avoid fetching an attachment until you actually want to open the attachment. That's lets you open a message with a large attachment as soon as it fetched the message body (without waiting for it to fetch the attachment).

Checking "Select this folder for offline use" etc. means that it will create a local copy of all of the messages on your hard disk. These are in mbox files, and normally referred to as offline folders. It will do that in the background. If global search/indexing is enabled it will use those mbox files, otherwise it fetches all of your mail and stores it in search database. Neither the search database or offline folders make re-opening a message quicker, and can add significant overhead as they run in the background at the same time you're trying to open a new message. I recommend you disable both features as they make opening a new message slower.

You said you have a rather slow Internet. How fast a download connection does a Internet connection speed test say it has?

If you have a really slow Internet connection you could experiment with lowering the mail.imap.mime_parts_on_demand_threshold , but it you have a long delay fetching 30KB you would probably be better off using a POP account configured to initially only fetch the headers. If you did that it would provide a dummy message body with a link that you could click on to fetch the real message body. While a IMAP account is usually a much better choice if you have a really slow Internet connection or a limit on how much bandwidth you can use a POP account might be a better choice since it avoids re-fetching the message every single time you open it.

Hopefully re-enabling MIME parts of Demand, UNchecking "keep messages in all folders for this account on the computer" and UNchecking "Enable global search and indexer" will make the performance acceptable.

If none of this helps I suggest you sign up for a free Gmail account, and add a Gmail IMAP account to Thunderbird. See what problems it has. That will help identify if the problem is your connection speed and/or your settings. See http://kb.mozillazine.org/Gmail
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Download full IMAP message body when new mail arrives

Post by tanstaafl »

"When I manually choose Sync Now, the message bodies are downloaded"

I'm confused. Is this an option from file -> work offline -> download/sync now? That is the old (pre-version 3) way of syncing offline folders. You're already using the newer version of syncing, which is done automatically for the selected folders. Please elaborate on where/how you chose Sync Now.
esperling
Posts: 7
Joined: January 14th, 2019, 10:27 pm

Re: Download full IMAP message body when new mail arrives

Post by esperling »

The exact problem I have is: I check thunderbird, see that there is new mail, and click on a message. At that point, thunderbird starts to download the message. Without attachments, it can take up to 10 seconds to retrieve the message body, due to high latency (satellite internet, VPN, bouncing from Africa via Emirates to VPN endpoint in Germany and then the US).

If I run "File, Offline, Download/Sync Now" manually, disable the "Work offline" switch, then wait for thunderbird to complete this process, it behaves like I want - it shows message bodies immediately when I select a message.

To be specific, I don't have an issue with internet bandwidth or quotas etc, I only have an issue with latency. Speed tests reach up to 10 Mbit, but takes 30-40 seconds to really get going again due to the latency.

So, my goal is to make thunderbird download complete message bodies automatically, for most of my folders, thus enabling "Select this folder for offline use" and disabling "Mime parts on demand". What I lack is the trigger to tell thunderbird to actually just go ahead and download the message when it arrives at the server.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Download full IMAP message body when new mail arrives

Post by tanstaafl »

It doesn't make sense to use two different schemes for syncing folders. Stick with the newer one - "synchronization & storage".

I don't think enabling MIME Parts on Demand would effect whats in the offline folder. Messages will get downloaded to that folder even if you never opened them. It would effect what gets cached, but the cache is pretty useless anyways. So you might as well take advantage of the optimizations of MIME parts on Demand.

You might want to simplify things and dispense with offline folders all together. Use a message filter for the inbox to automatically copy mail to the inbox, and a message filter for the sent folder (check "After sending" in the filters) to automatically copy the sent message to a sent folder in "Local Folders". This would be in addition to whatever copy is made due to "copies & folders" settings.

"(satellite internet, VPN, bouncing from Africa via Emirates to VPN endpoint in Germany and then the US)."
You might consider using something like a Gmail IMAP account that is configured to automatically fetch email from your other accounts and store it in the webmail inbox. That could replace your other accounts in Thunderbird. By adding multiple identities you will be able to send/reply as if you were still using those other accounts. Hopefully that will reduce latency because you're making a single VPN connection to a much closer server and letting it fetch the mail from the USA before notifying you that it exists.

The main issue would be how to have the nearby IMAP server make a VPN connection to the mail server(s) in the USA. I'd assume its a common problem for businesses. https://support.google.com/a/answer/2634553?hl=en talks about adding a third party VPN to G Suite (the enterprise version of Gmail).

See http://kb.mozillazine.org/Replace_sever ... ne_account if you'e interested in doing something like that.
esperling
Posts: 7
Joined: January 14th, 2019, 10:27 pm

Re: Download full IMAP message body when new mail arrives

Post by esperling »

I'd use any sync scheme I can get working - I don't know which settings are relevant for each scheme though.

I'm aware of the various alternative ways of setting up mail servers, I just finished migrating from our dovecot/exim setup to a managed exchange solution. GMail would've been the choice if they had real folders instead of the tag-based pseudo folders, but well, they don't. Local folders isn't an option given the large amount of mail being placed in folders with server rules, I really wouldn't want to keep that in sync manually.

In short, I'm not really looking for alternative ways of achieving this specific user experience, I'm just trying to figure out how to tell thunderbird to automatically download message bodies - is this possible? Could it be done with an extension if it's not natively supported?
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Download full IMAP message body when new mail arrives

Post by tanstaafl »

"I'm just trying to figure out how to tell thunderbird to automatically download message bodies - is this possible?"

The problem is you said message-bodies (i.e. more than the currently selected message). POP has a download-centric view of the world. IMAP has a client server-centric view of the world. That is why IMAP accounts don't do what you want.

You are probably better off with a POP account because the automatic downloading of the message body for all of your mail in a IMAP account only occurs for offline folders. Are you willing to read your mail while working offline? You could compose a reply and use Send Later to send it when you switch back to working online.
esperling
Posts: 7
Joined: January 14th, 2019, 10:27 pm

Re: Download full IMAP message body when new mail arrives

Post by esperling »

I'm also fully aware of the differences between POP3 and IMAP, and POP isn't an option for me.

Thunderbird is already capable of downloading all message bodies, and when I manually tell it to do so it works just fine. The *only* question I have is whether it's possible to trigger this automatically - I don't really need workarounds or alternative setups, I just want to figure out if I can make thunderbird do this with config, or if I have to start looking at extension APIs.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Download full IMAP message body when new mail arrives

Post by tanstaafl »

The add-on API situation is chaos right now. Every new major version something breaks due to changes driven by Firefox. It looks like Legacy API's might be gone as soon as version 68.

If you created an add-on to download the message bodies you'd still have to switch to working offline to see the contents (since IMAP accounts show the contents of remote folders when working online). Why do that when you can switch between online and offline mode with the click of a mouse (icon in lower left corner, main window), and leverage the new way of syncing (which is less likely to have regression bugs)?

If you're willing to write an add-on to solve your problem perhaps it would be worthwhile to see if Movemail accounts haven't been disabled under Windows. I know they used to work under Windows even after they removed support for it from the new account wizard, but that was a long time ago. You'd have to use a POP account and edit mail.server.server#.type from POP3 to Movemail. You could probably use Fetchmail from cygwin to create a spool file in the accounts local directory.
esperling
Posts: 7
Joined: January 14th, 2019, 10:27 pm

Re: Download full IMAP message body when new mail arrives

Post by esperling »

tanstaafl, I do understand that you're trying to help, and I'm grateful. But really all I'm looking for is an answer to the actual question - can thunderbirds fully functional imap caching be triggered automatically when a message arrives, or not?
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Download full IMAP message body when new mail arrives

Post by tanstaafl »

The answer is yes, but that is not sufficient to solve your problem.

Its not caching. Its offline folders. There is a tremendous difference. The cache is basically worthless. The offline folders aren't used as a cache when working online, like Apple Mail does. You either see the contents of the remote folders (work online) or the contents of the offline folders (work offline) unless you've discovered some workaround I'm not aware of.

If you are using the new way to sync, when new mail is detected Thunderbird should automatically update the offline folders with the complete contents of any new mail messages. The problem is it does you no good (can't see it) unless you switch to working in offline mode.
esperling
Posts: 7
Joined: January 14th, 2019, 10:27 pm

Re: Download full IMAP message body when new mail arrives

Post by esperling »

Ok, so what is happening when I trigger "File,Offline,Download/Sync now" without checking the "Work offline ..." checkbox? Because that gives the behavior I need, immediate display when clicking on any message (and the message eventually, after 10-30 seconds, also gets marked as read on the server). Is this new/old/offline/cache, and can I not make this happen in response to message arrival?
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Download full IMAP message body when new mail arrives

Post by tanstaafl »

You can download the contents of a remote folder to a offline folder while working online. However, AFAIK when working online you don't get to see the contents of offline folders.

As an experiment I created a test profile with just a Outlook IMAP account. I disabled the new way of sync (synchronization & storage), verified there were no mbox files in the local directory of the account, and used file -> offline -> download/sync now to download just the inbox while remaining online (I UNchecked "work offline once download and/or sync is completed") . I exited Thunderbird and used a text editor to edit both the Subject and part of the message body of the first message in the mbox file. I then started Thunderbird and looked at the first message.

* In online mode it showed me the unedited subject and unedited message body. The folder listing showed the unedited subject.
* In offline mode it showed me the EDITED subject and EDITED message body. The folder listing showed the UNedited subject.
* The folder listing never changed as I switched between different messages and/or working online/offline.

I know that global search/indexing downloading mail to build a search index (if there is no mbox file) leaks notification of any new mail to other components because I wrote a bug report about it and got confirmation from the architect that its by design. I'm not aware of any way for downloaded message bodies from the current session (haven't exited Thunderbird yet) to leak to other components. If they do on your configuration, good for you. But the simplest explanation is that either you didn't UNcheck the checkbox to work offline after downloading or the remote folders were able to fetch the message bodies very quickly afterwards due to the messages being cached by a proxy server.

I don't think I can help you.
esperling
Posts: 7
Joined: January 14th, 2019, 10:27 pm

Re: Download full IMAP message body when new mail arrives

Post by esperling »

I'm quite certain that I uncheck the check box to work offline after sync - just retried it. I'm also 100% certain there's no caching in between my client and the US server.

However, I've found the download_on_biff toggle, which seems to improve if not resolve this, I'll look more into what that actually changes.

It's a bit curious that thunderbird can't be told to just cache everything - every other client I've used supports this out of the box, it provides a much more responsive UI.
Post Reply