Thunderbird showing duplicate messages that don't exist

[djbon2112]

This is a bit of a strange one, so first I'll describe my setup, then the problem.

I run my own mail server using Postfix+Dovecot IMAP, using a standard Maildir backend store. This setup works wonderfully and for at least 2 years I had no problems whatsoever. I make extensive use of Thunderbird as an IMAP client with this account, and again for years it was working perfectly. The Dovecot setup consists of two instances behind an HAProxy load balancer - this might be the cause of my trouble, but I'm not seeing the following behaviour elsewhere (e.g. Android client, or Webmail which uses the load balancer), and I have modified this config several times in an attempt to correct this, but at this time I'm convinced the problem is Thunderbird.

The problems I'm seeing started approximately 6 months ago - I'm not 100% sure if there was an update around the time (I didn't notice one) but this seems to be a recent issue that I never experienced before.

What's happening, is that I'm getting "duplicates" of certain messages. I quoted duplicates, because from what I can see on the server side, there is *not* a duplicate message being stored. Additionally, the "duplicate" in Thunderbird usually has one of two things wrong with it: one, the message is "blank" - no content, no headers, nothing, and refuses to load; or two, the message that doesn't work shows an attachment (which doesn't exist). Sometimes after a little while, the second message does begin loading (identical to the first), but I still only see one copy on the server side.

I've tried all the standard options for correcting this (compacting regularly, removing the mailbox index files, etc.) but nothing seems to stop it happening. Sometimes it looks like this happens if I mark the mail "read" on my phone before accessing it in Thunderbird, but I haven't been able to correlate this definitively. I've also tried (on the server/sysadmin side) changing various settings about the load balancer, but this seems to happen regardless of the settings I'm using there.

Here's an example message from today (sent to a public mailing list) that shows the two entries, one (the bad one) with the "Attachment" flag set (the first) and the actual valid message (the second): Imgur Album. I can absolutely confirm that only a single file exists in the Maildir on the server side.

So, I'm wondering if anyone else has seen this behaviour, and if so, if you've found a way to prevent it?

[tanstaafl]

If you're the only user why do you have two instances behind a load balancer?

You could enable IMAP logging per http://kb.mozillazine.org/Session_logging_for_mail/news and examine the log file to see exactly what happens when Thunderbird fetches a non-existent duplicate but I suggest you start by testing using a single instance (and no load balancer) and see if the problem disappears. I suggest you also view the raw message source using view -> message source or Control-U and check whether the non-existent duplicate has the same value for the Message-ID header. That header is used to uniquely identify a message.

In general Dovecot works very well with Thunderbird.

[djbon2112]

I'm not the only user, but I'm both the admin, and the only one who runs Thunderbird. It's load-balanced for redundancy and not performance, though I'm pretty convinced the load balancer has nothing to do with this, since the problem occurs even if I remove the load balancing (i.e. have one HAProxy backend active)

I'm still digging to see if I can find the common denominator and have enabled session logging, and so far what I see are constant IMAP calls whenever I open the "duplicate" message, but not when I open the main one.

**SOLVED - EDIT** (to avoid bumping unnecessarily)

I was pretty much able to track this down to Dovecot and NFS interactions, not Thunderbird itself nor the HAProxy setup. This mailing list post pretty much sums it up: https://www.dovecot.org/list/dovecot/20 ... 17454.html The easy fix is in the NFS mount options, should anyone Googling for this problem in the future turn up here: Add 'lookupcache=positive' to your NFS mount option on the email server and you should see the corruption stop.

The reason I thought it was Thunderbird was due to, well, me using Thunderbird exclusively on desktop, and that being the #1 place I saw this. It looks like *most* other mail clients handle this sort of corruption transparently, but in Thunderbird's case it would just display a second (or, more than once, third) copy of the mail. Once the corrupt indexs on the Dovecot side cleared, the multiple copies Thunderbird saw were again all valid so I ended up with "duplicates" on the client side.