IMAP: why all the unnecessary transfers when sending?

[Lemon Juice]

I have recently decided to switch to IMAP from POP3 and see how I like it. I use Thunderbird 3beta4 on one computer and Seamonkey 2.0 on another. Everything works fine except that I find the email program makes unnecessary transfers of messages to and from the server when sending. This becomes a bit annoying when I'm sending large attachments, let's say 20MB. I have all folders set for offline storage so that I don't have to re-download each received message and I'm using a proper commercial IMAP account (not Gmail). According to preferences, sent mail should be copied to the Sent folder. So this is what happens:

1. I compose the message, add attachment and click send. TB transfers the whole 20MB attachment to the server.
2. After that the progress windows changes to "Copying message to the Sent folder". It takes a while and I realize that TB is transferring the whole 20MB message again to the server to the Sent folder.
3. A moment later I want to check what exactly I sent so I open the message in the Sent folder and TB downloads the whole 20MB message back from the server before it is available for viewing.

This whole procedure results in 60MB of transfer and I am asking why? I would expect this to happen:

1. I compose the message, add attachment and click send. TB transfers the whole 20MB attachment to the server.
2. The message is saved in the Sent folder ready for offline use and the server only receives information that the message just sent should be stored in the Sent folder.
3. When I open the Sent folder, TB only checks if the message on the server is the same as the one on disk. And when I open the message it is loaded from offline storage.

Is it an inherent IMAP flaw that messages in this scenario need to be transferred 3 times back and forth instead of just once? Or is it that TB still hasn't implemented offline storage well enough? Or maybe there is a way to reduce the transfers that I don't know of? So far IMAP is wonderful for small messages but when dealing with large attachments I wish I used the traditional POP3.

[DanRaisch]

Yes. You can changed the target for the Sent folder if you don't need to access sent messages from a computer other than the on on which you composed them. Look under Tools->Account Settings->Copies & Folders. Change the selection to a folder under Local Folders so that the save action is only to the local hard drive and not to the server.

[Lemon Juice]

Yes. You can changed the target for the Sent folder if you don't need to access sent messages from a computer other than the on on which you composed them. Look under Tools->Account Settings->Copies & Folders. Change the selection to a folder under Local Folders so that the save action is only to the local hard drive and not to the server.

Ok, good to know I can do it but still I would prefer my sent mail to be stored online - this is why I use IMAP. I just don't know why TB cannot simply send the message to the server with a command to store it in the Sent folder? Can't IMAP servers understand such commands? Still if they can't, TB could at least store the sent message for offline use immediately and avoid having to download it. I realize there might be no solutions for this problem now but am wondering if there's any hope for a fix in a future TB or is it simply the nature of IMAP that cannot be changed?

[rsx11m]

These are two different (i.e., completely independent!) servers:

1. For sending, you have to connect to an SMTP server on your provider's side. This will take care of transporting the message to your recipient(s).
2. To put a copy into the Sent folder of your IMAP account, Thunderbird has to talk to the IMAP server and deposit a copy of the message there.

Again, these are two different processes involving different servers, even though they usually use the same authentication (but don't have to). As a workaround, you can disable "Save a copy" and instead "Bcc" yourself (that's in Copies & Folders too, just underneath the Sent folder). In this way, the message sent to the SMTP server should be delivered to the IMAP server as incoming mail, i.e., you would see it in the Inbox and marked as Unread. It still means that the same message has to be downloaded locally if you grab it off the IMAP server, enabling offline copies (synchronization) may prevent a repeated download though.

[Lemon Juice]

These are two different (i.e., completely independent!) servers:

1. For sending, you have to connect to an SMTP server on your provider's side. This will take care of transporting the message to your recipient(s).
2. To put a copy into the Sent folder of your IMAP account, Thunderbird has to talk to the IMAP server and deposit a copy of the message there.

That makes sense, I haven't thought of that. I thought SMTP and IMAP servers can somehow communicate on the server side to each other - I thing Gmail is configured that way that after you send a message through SMTP it appears in your Sent mail without mail client having to put it there but as I understand this is non-standard behaviour.

But what about my last point - after TB has sent mail through SMTP and put it in the Sent folder on the server through IMAP, is re-downloading it necessary when I open it in TB?

[tanstaafl]

"But what about my last point - after TB has sent mail through SMTP and put it in the Sent folder on the server through IMAP, is re-downloading it necessary when I open it in TB?"

It needs to be fetched to read it again in Thunderbird. Its stored in a remote folder after all. Downloading implies it also gets stored on the hard disk. That wouldn't occur unless you configure the folder as a offline folder.

Having to re-fetch messages to read them again is one of the tradeoffs of IMAP. Some email clients transparently support a cache that reduces or eliminates the need for such re-fetching. Thunderbird 2 doesn't but the optional disk cache added in Thunderbird 3 will. Thunderbird 3 is expected to be released before the end of the year.

http://kb.mozillazine.org/Thunderbird_3 ... Disk_Cache

[Lemon Juice]

It needs to be fetched to read it again in Thunderbird. Its stored in a remote folder after all. Downloading implies it also gets stored on the hard disk.

If you say so then let it be. It's true that it is stored in a remote folder but it's also true that the exact same copy is available without downloading because I have just composed the message on my computer. Then why not simply put it in the Sent folder straight away? But perhaps there are some technical reasons I don't know why downloading is essential here.

Having to re-fetch messages to read them again is one of the tradeoffs of IMAP. Some email clients transparently support a cache that reduces or eliminates the need for such re-fetching. Thunderbird 2 doesn't but the optional disk cache added in Thunderbird 3 will. Thunderbird 3 is expected to be released before the end of the year.

http://kb.mozillazine.org/Thunderbird_3 ... Disk_Cache

I'm using Thunderbird 3 (beta) but the disk cache doesn't deal with moving sent messages to the Sent folder without network transfer so that it is available for offline viewing immediately.

[rsx11m]

The disk cache may not cache outgoing connections, and it would be a different request anyway (the cache works on the low-level networking side, thus doesn't know that the message deposited in the Sent folder is actually the same one that you want to grab later). Also, the disk cache - while available and active - still needs some improvement with a proper expiration policy. I've seen attachments being refetched even though they were present in the cache, for some reason they were treated as expired and revalidated. The offline synchronization mechanism is more intelligent as it works on the IMAP level, but I don't know if uploading a message onto the remote Sent folded will automatically result in an offline copy of it as well.

[tanstaafl]

I don't know if uploading a message onto the remote Sent folded will automatically result in an offline copy of it as well

rsx11m knows far more about this than me but given that the disk cache is general purpose (not IMAP-centric) I'd be surprised if they complicated the design by adding side effects such as adding something to the cache when uploading.

Also keep in mind that Thunderbird 3 supports the ability to automatically send messages in the background. If you're sending messages in the background you'd expect that it would upload your copy of the sent message in the background too, making the "unnecessary transfer" (as you describe it) a moot point for most users.

[Lemon Juice]

rsx11m knows far more about this than me but given that the disk cache is general purpose (not IMAP-centric) I'd be surprised if they complicated the design by adding side effects such as adding something to the cache when uploading.

That is okay because I don't think disk cache should do that. The point here is not to store sent messages in some temporary cache but store them permanently in the Sent folder for offline use after they have been sent - just like it happens with POP3 accounts. I wonder if it's a reasonable feature request.

Also keep in mind that Thunderbird 3 supports the ability to automatically send messages in the background. If you're sending messages in the background you'd expect that it would upload your copy of the sent message in the background too, making the "unnecessary transfer" (as you describe it) a moot point for most users.

My concern was not so much about the window and progress bar being open while the message is being sent but rather having to spent more $$ (and often much more time) when connected through a provider that charges per used transfer, for example a mobile phone. Sure, if we assume everyone is on fast LAN then the problem doesn't exist.

[tanstaafl]

If your email provider supports custom sieve scripts (sort of like message filters that are automatically run on the server when it receives a message, not by the email client) you could configure the account to not save a copy in the sent folder, to always bcc yourself, and use the custom sieve script to automatically move the bcc'd copy to your sent folder. That way you get a copy of the message in the sent folder without having to upload it via IMAP. If you tried to do the same thing using message filters it would fetch and then move the message, making you pay for two additional copies of the message.

http://www.fastmail.fm, runbox and tuffmail are examples of commercial IMAP providers that support this functionality though fastmail has the best Sieve support (they support many extensions to the standard). I've paid for a full account with fastmail for about 6 years because I find the extra features they provide very useful. I don't know of any free IMAP accounts that provide this.

http://en.wikipedia.org/wiki/Sieve_%28m ... anguage%29 has an overview of the Sieve mail filtering language. Its available for many types of IMAP servers but since it was developed as part of the Cyrus IMAP server you're more likely to find a email provider that supports it that is using Cyrus.

[Lemon Juice]

Thanks for the info, I didn't know this feature was called 'sieve scripts'. I think gmail provides this feature, I don't know if they specifically use sieve scripts but you can create filters that run on the server side before they reach your inbox. Anyway, my email provider only allows me to create filters that run only in webmail so I am left to using filters in TB.

However, I did some testing and found an interesting behaviour:

If your email provider supports custom sieve scripts (sort of like message filters that are automatically run on the server when it receives a message, not by the email client) you could configure the account to not save a copy in the sent folder, to always bcc yourself, and use the custom sieve script to automatically move the bcc'd copy to your sent folder. That way you get a copy of the message in the sent folder without having to upload it via IMAP. If you tried to do the same thing using message filters it would fetch and then move the message, making you pay for two additional copies of the message.

I set TB to not move the message to Sent folder and to bcc to myself. I also created a filter in TB to move messages from me to the Sent folder. The interesting thing was that after sending the message it appeared in my Sent folder after a short while but TB didn't download it, it looked like only the headers were fetched in spite of all of the folders being set to offline storage. When I clicked on the message then only the text portion was downloaded and displayed, and the rest of the message (attachment) was downloaded only when I chose to save the attachment. The weird thing was that without the filter the message was downloaded entirely right away even before I opened the inbox or clicked the message. So only the fact of setting up the filter prevented TB from fetching the message immediately. Well, even with the filter the message was eventually downloaded even without me doing anything but it took a minute or two. I don't know if it's intended or not but certainly good in this case because I don't need sent messages to be available immediately and they can be downloaded in the background. However, the headers appear immediately.

All in all, after using your trick with Bcc to myself I was able to reduce 1 unnecessary transfer, it went from 3 to 2. And it looks like I don't need sieve scripts for that - how would they help me in this case? I know it would be more convenient to set up all my filters on the server instead of in each of my email clients but apart from that what benefit would they give me? I guess even with sieve scripts 2 transfers would be necessary - one for sending and one for fetching the message to the offline Sent folder.

http://www.fastmail.fm, runbox and tuffmail are examples of commercial IMAP providers that support this functionality though fastmail has the best Sieve support (they support many extensions to the standard).

They look impressive in their feature sets but are quite pricey - which is understandable.

Thanks for your suggestions.

[tanstaafl]

With Sieve scripts the bcc'd message is moved directly to the remote Sent folder. This will occur without your PC even being powered on. The Sieve scripts are triggered by the server receiving a message for you, not your logging in. That means you'd only be paying for the original message being sent to the SMTP server unless you decide to read the copy in your remote Sent folder.

I'm so used to Sieve scripts that I forgot Gmail has its own equivalent of message filters. They're not based on the Sieve mail filtering language. Gmail's filters don't support copy or move, you'd have to set a label instead. When the filter sets the label from the IMAP servers viewpoint that copies the message to a folder. I don't see why that would cause Thunderbird to fetch the message. It should just notify you of new mail and update the headers if you're running Thunderbird.

GMX.com offers free IMAP accounts with a large mailbox and provides its own type of message filters. Its filters support moving or copying a message. It even supports regular expressions.

The weird thing was that without the filter the message was downloaded entirely right away even before I opened the inbox or clicked the message.

I don't understand that unless you're using AutoSync (a type of offline folder). It downloads messages for offline use whenever you open a folder or Thunderbird detects new mail. See http://kb.mozillazine.org/Offline_folders for details.

[Lemon Juice]

With Sieve scripts the bcc'd message is moved directly to the remote Sent folder. This will occur without your PC even being powered on. The Sieve scripts are triggered by the server receiving a message for you, not your logging in.

Certainly sieve scripts are best in this case but I don't really mind if my messages are moved by Thunderbird filter, my experience is practically the same. And from what I see, Thunderbird moves a message immediately after receiving its headers and it just tells IMAP to move it, which happens very fast. Network usage also seems to be minimal when moving so I don't mind.

That means you'd only be paying for the original message being sent to the SMTP server unless you decide to read the copy in your remote Sent folder.

Still I would need to pay for 2 transfers because I want my sent messages to be available for offline use, in which case they need to be fetched from the server even if I use sieve scripts.

I liked Gmail filters but there were other things that didn't work well like deleting messages, deleting attachments, saving drafts, buggy attachment restrictions. Better to look for IMAP elsewhere. I haven't tried GMX but they don't seem to offer addresses in your own domain.

[tanstaafl]

If you want a copy for offline use configure your sent folder to be in local folders. That doesn't require any network traffic. You could rely upon setting bcc and setting the label in a Gmail filter to copy the message to a remote Sent folder.

If you're using two computers at home you could configure one of them to offer the local folders directory as a file share, and configure Thunderbird on the other PC to use that local folders directory per http://kb.mozillazine.org/Moving_your_m ... hunderbird . That way regardless of which PC you sent a message from both would have a local copy of the message. The drawback is that you'd either have to keep the PC offering the file share running all the time or periodically sync the two local folders directories.

See http://kb.mozillazine.org/Synchronizing ... _computers