Wrong date being used for some messages

[aajax]

Just installed a new instance of Thunderbird (TB) on a Windows 10 system. It is Version 68.1.1. For the most part it looks to have been successful. However, there is a conspicuous and at least mysterious problem. I've noticed a set of 27 email messages, all from the same sender, that display the same wrong date. This wrong date is also being used to determine sort order. As it happens these messages originally appeared at the top of the descending list of messages. The wrong date correlates very closely to the time when the subject email account was first defined and TB began retrieving information from the IMAP server. Furthermore, while I cannot say for sure that such a defect is NOT present anywhere else in that Inbox I have verified that when message headers are viewed what I'd think to be the correct date is what is contained in the header. Also, the date displayed in the sorted list of messages appears nowhere in the header.

How did TB acquire and choose to use this wrong date? How can this problem be corrected?

[DanRaisch]

What is that "wrong" date?

[aajax]

10/10/2019 3:50PM

[DanRaisch]

Have you checked with the sender of those emails to make sure their system clock is set properly?

[aajax]

These are old emails that have been sent over a period of many months at many different times. The date displays correctly on another instance of Thunderbird. When they were received Thunderbird was running on V60. This appears to occur only on a new instance of Thunderbird running on V68, where the emails had to be re-discovered on the IMAP server.

Do you think it possible to send 27 emails all at exactly the same time? The date in the message header appears to be correct. That is NOT the date being displayed on the list of messages.

[tanstaafl]

Try rebuilding the folder listing by right clicking on the folder in the folder pane, select properties, and then press the repair folder button. If that doesn't help I suggest you login to webmail using a browser and see if it has the same problem.

[aajax]

To clarify further, I've been using TB for many years on a laptop computer. I recently noticed the existence of plugins that can be used to synchronize calendars & address books. This caused me to want to add TB to one of my desktops that I spend a lot of time on. I'm also thinking that I should be able to synchronize calendar & contacts with my wife's email client. I haven't reached any conclusion but would prefer to setup a different, than TB, email client for my wife.

My laptop was running TB V60 which according to that version of TB was up to date. However, it seems that the synchronization plugins preferred TB V68. Therefore, that is what I installed on my desktop but synchronization involves multiple computers which caused me to also upgrade my laptop to V68. Not sure why TB previously said it was up to date with V60.

When the laptop updated, which I'd point out involved continued use of the same profile, the date problem mentioned herein did not and never has existed. However, when I created the new profile on the desktop which hadn't previously had TB installed this date problem appeared right away (i.e., has always existed). I even performed a do over and created another profile with the same result.

As suggested when accessing the same email via a web client there is NO indication of such a problem.

As suggested, I performed the "Repair Folder" operation on the desktop. Sure enough the problem is recreated but now the wrong date has been changed to coincide with the date/time that the repair was undertaken. Clearly there is something about these emails that TB is failing to handle correctly.

[aajax]

A few more observations as follows:

1. When a message is selected for display in a separate tab (i.e., double click) the date is normally displayed even with View>Headers>Normal. However, the problematic messages display nothing where the date would normally be shown.
2. With View>Headers>All an inspection of various messages reveals that there are lots of tags that contain a value for date. For example, one of these tags is like this -> "Date Sat, 28 Sep 2019 15:31:55 -0400". Others that include a date value and possibly other information are as follows:

• "Received from ...; Sat, 28 Sep 2019 13:32:19 -0600" Note: There may be multiple instances of this tag.
  "Delivery-date Sat, 28 Sep 2019 13:32:19 -0600"

3. The "Date" type of tag is missing from the problematic messages but the other types of tags are plentiful.

This finding leads to speculation that TB V68 needs to find a tag of type "Date" in order to display the date when View>Headers>Normal is specified whereas V60 did not mind if it were missing. V60 appears to be able to use the value for date found in some other tag and consider it the date to display whereas V68 appears to end up with NULL for the date to display and uses the date/time when the message list is constructed as a substitute in that list.

[tanstaafl]

"Date Sat, 28 Sep 2019 15:31:55 -0400" is not a tag, its a Date: header. "Received from ...; Sat, 28 Sep 2019 13:32:19 -0600" is an example of a Received: header. While it contains a time/date stamp, that information is supposedly never used unless you set mailnews.use_received_date true. I know that setting used to be ignored for IMAP accounts while working for POP accounts in many versions. I don't know its status for version 68.*

The timestamp data is normally obtained from the Date: header. If it doesn't exist and there is no IMAP INTERNALDATE message attribute (whether that exists depends what type of IMAP server is used and whether its configured to support that) Thunderbird normally defaults to using the oldest possible time/date (1/1/1970 GMT, adjusted for your time zone), based on Internet conventions for date/time stamps. See http://kb.mozillazine.org/Invalid_date_in_IMAP_messages .

See if the problem occurs in safe mode (help -> restart with add-ons disabled). This problem might be a side effect of a new WebExtensions based add-on written for version 68. If the problem exists in safe mode, use view -> message source or Control-U and look at the raw message source of a message with the wrong date and see if the Date: header exists.

There have been several different problems in version 68.* due to IMAP messages being fetched in chunks. Use the config editor at tools -> options -> advanced and see if mail.imap.mime_parts_on_demand defaults to true. If it does try toggling the value to false (disabling MIME parts on demand disable fetching the message in chunks), and restarting Thunderbird.

If you copy a message with the wrong date from a IMAP account to a folder in "Local Folders" does it display the correct timestamp in "Local Folders"?

I found a bug report received date does not work on IMAP because Received: header is not fetched by BODY.PEEK[HEADER.FIELDS (always the same as date) bug report, but if you occasionally get the correct date in a account you don't have that problem.

http://kb.mozillazine.org/Entire_messag ... AP_message
http://kb.mozillazine.org/Tags

[aajax]

Sorry about my failure to use proper terminology. That comes with ignorance.

See if the problem occurs in safe mode (help -> restart with add-ons disabled). This problem might be a side effect of a new WebExtensions based add-on written for version 68.

No change!

If the problem exists in safe mode, use view -> message source or Control-U and look at the raw message source of a message with the wrong date and see if the Date: header exists.

There is NO "Date: Header"
Also, I did find something that seems to confirm such as follows:

***Begin content found in message source***
Content analysis details: (1.0 points, 4.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/Dns ... nsbl-block
for more information.
[URIs: ***Removed to keep private***]
-0.5 SPF_PASS SPF: sender matches SPF record
0.0 HTML_MESSAGE BODY: HTML included in message
0.0 MIME_HTML_MOSTLY BODY: Multipart message mostly text/html MIME
1.4 MISSING_DATE Missing Date: header
0.1 MISSING_MID Missing Message-Id: header
0.0 T_REMOTE_IMAGE Message contains an external image
X-Spam-Flag: NO
***End content found in message source***

... Use the config editor at tools -> options -> advanced and see if mail.imap.mime_parts_on_demand defaults to true. If it does try toggling the value to false (disabling MIME parts on demand disable fetching the message in chunks), and restarting Thunderbird.

No change!

If you copy a message with the wrong date from a IMAP account to a folder in "Local Folders" does it display the correct timestamp in "Local Folders"?

No change!

I found a bug report received date does not work on IMAP because Received: header is not fetched by BODY.PEEK[HEADER.FIELDS (always the same as date) bug report, but if you occasionally get the correct date in a account you don't have that problem.

So far I've only detected this problem for email received over a period of time from the same correspondent (i.e., the same mail server). That party has their own domain name that I suspect could easily be related to one of the many independent hosting services. I'm pretty sure that I could find out what mail service is being used, if such is desired.

Also, I haven't received any new emails from this correspondent since posting this problem. However, that will change before long. I'm very curious to see what date might appear in that case when no folder refresh (repair) is involved.

[tanstaafl]

A Message-ID: header is required, its used to uniquely identify a message. Its omission is worse than a missing Date: header.

It sounds like your problem is due to one sender, who badly mismanaged setting up email on their host. I suggest you send them a message telling them they need to fix their configuration as its missing at least two (Date: and Message-ID:) important headers.

I found https://github.com/mageplaza/magento-2-smtp/issues/103 which talks about a module used by some stores web sites to send mail which has this bug. Perhaps your correspondent is using some specialized host software rather than just being yet another user who has a Godaddy, HostGator etc. account.

[aajax]

Interesting!!! I'll try to learn what I can about the offending mail service.

However, I have to point out that this hadn't been a problem, at least for me, until I installed TB V68.

[aajax]

Now I have received a new email from the problematic sender. Now that I have 2 computers running TB that email was processed/received at a different times on each of those computers. As a result, the date displayed in the message list for the inbox is different on each system. This makes it look like TB substitutes the date/time when the incoming message is first processed for the date that is missing form the message header. It would seem to me that it would be better if TB used the Delivery-date element that can be found in these problematic messages. At least then the date would be uniform for all clients that receive the message which is NOT the case now.

Also, I haven't spent the time needed to test this theory on TB V60 but present thinking is that the only difference between V60 & V68, for this case, is that when I installed V68 on a system with no prior TB installation all of the emails had to be read and processed to create the new profile. Then as mentioned it looks like TB substitutes the time when the client processes the the incoming email. That did NOT happen when I upgraded from V60 to V68 on the computer previously used for receiving email.

Therefore, I now think there is NO reason to think V68 is any different than V60 in this respect. However, I would like to point out that this bit of testing I've been doing has involved experimenting with some other email clients. Neither the Windows 10 Mail App nor eM Client App for Windows 10 suffer from this problem. While they do seem to recognize that the Date: header is missing, rather than NULL they display "Date Missing" where the date should be displayed and their equivalent message list appears to use the Delivery-date: element from the header. I offer that this method is much less deceiving than what TB is doing.

Why doesn't TB use the "Delivery-date:" header as a substitute for missing "Date:" header rather than an arbitrary value that differs from system to system?

[tanstaafl]

"Why doesn't TB use the "Delivery-date:" header as a substitute for missing "Date:" header rather than an arbitrary value that differs from system to system?"

There is a difference between sent date and received date. Sent date is normally taken from the Date: header. The received date is handled differently. Click on the weird little widget at the far right of the folder listing columns and select "Received" to add the received date to the folder listing.

None of my email providers add a "Delivery-date:" header. I think that's a non-standard optional header, so the name should really have begun with a X. A delivery-date: header sounds like a received date for when the IMAP server got it, not a sent date. What I think you are really asking is if there is no sent date why doesn't Thunderbird display the first received date (only one hop from the sender) instead. That's a good idea, but I suspect it doesn't occur often enough (or cause enough grief) for a developer to worry about it. Given the ridiculously large backlog of untriaged bug reports unless it becomes some developers pet peeve or there is some other reason for them to pay attention to it, lots of potential small improvements get overlooked. Hopefully things will slowly improve now that's there's a larger paid staff (plus volunteers).

Received: headers have a timestamp, and are specified in a rfc. One of the KB articles mentions that Thunderbird does support getting the date from a IMAP INTERNALDATE message attribute. I think the article is wrong (I can say this since I wrote it over a decade ago) in that it says it uses it as fallback to the Date: header. Its used for the Received date instead. The value is supposed to be the same as whats in the last Received: header. I found https://bugzilla.mozilla.org/show_bug.cgi?id=570355 which talks about "Thunderbird takes the "Received date" value from the message's Received: header (for both IMAP and POP),". If access to the wiki gets restored I'll do some testing and update the KB article accordingly.

http://www.emaildiscussions.com/showthread.php?t=59640

[aajax]

I did think that these different headers are produced at different times.

Curiously, when I do what you suggest and add "Received" date to the list I find that there is NO difference from "Date". A quick scan of some 3800 emails presently in my inbox suggests a pretty good variety of both clients and servers are involved. I'd also point out that this includes that new problematic email which has a different date for when each instance of TB was run.

If "Delivery-date" is known to be undependable I understand why something else would be desired. I haven't inspected my whole inbox for that one. It is present in everything I have inspected so far while dealing with this issue. It would seem to me that when the Date: header is missing and some value is needed for sorting that list it becomes necessary to use something that is less than ideal. The idea of using a value that varies depending on when TB was launched is a pretty poor choice.

If "Received" headers are more reliably found than "Delivery-date" then I'd say that is a better choice. It looks to me like there are many "Received" headers suggesting that each server which receives the message adds such a header. I suppose that would create the dilemma of deciding which one to use for a substitute. My thinking would be that the first one (i.e., first server to receive the message) should be pretty close to the date the client sent the message. That would make it a better choice than "Delivery-date" in my way of thinking.

Another consideration would be adding some kind of indication (e.g., an asterisk) that it is NOT authentic. This would apply to both the list as well as the message display where at present it just leaves it blank which is quite deceptive.

Have I discovered another problem by turning on "Received" for the list option and finding NO difference from Date?