Error in date sorting

[Macavity]

Hello, I successfully imported all my old messages from Outlook Express to Mozilla. I have a problem though... The first time I clicked on an imported folder, it took some time to generate the list (it contains hundreds of messages); then, I saw all the messages. But at the bottom of the list (I had sorted them by date, with the newest at the bottom) I found 20 or so messages, with an indicated date of 06/02/01, 06:28. If it was the real date, those messages should have gone at the *top* of the list, but they were at the bottom. And the displayed date was wrong; all those messages were from different dates and times; in fact, if I open those messages, in the "Date" field in the lower window, I get the correct date.
What has hapened? Why doesn't the date in the messages list match the date of the actual messages? And how do I fix that?

Thanks,
Cristian

[Macavity]

Update: Not only the date is wrong, also the subject is missing. And I know for sure that those messages had a subject.
I checked, with a text editor, the actual mail file, and I found these messages: there IS the correct date and subject there. But it isn't imported correctly in Mozilla. I also tried deleting the folders and re-importing them, but I had the same problem.

This is not system-dependant: I just installed Mozilla on my sister's PC, and she had the same problem (even though only with one message): a date of 06/02/01, 06:28 ; no subject; and the message appears at the bottom of the list.

Any ideas? Is this a known bug?

Cristian

[Macavity]

So, can't anybody help me? I would like to "jump" from Outlook Express to Mozilla - as I sincerely like it - but I must sort out this problem... I just can't ignore it. I don't even know if this problem only arises when importing an OE mailbox, or also when actually receiving new emails...

Cristian

[Macavity]

Update: I also tried Mozilla Thunderbird. I imported my Outlook Express mail, and the same messages show the same problem (wrong date and wrong sorting and no subject). I wonder where does that weird date come from? (06/02/01, 06:28)

Really, didn't anyone see this before? If so, maybe I've found a new bug; where should I report this?

Cristian

[Ogre]

I have never imported Outlook messages, so I don't know if this occurs always.
Does that also happen when you receive new mail?

Before you file a bug, post something in the Mozilla Bug Forum

[Macavity]

Ogre,

It doesn't happen on each message. In the folder I imported, there are about 1500 messages, and only 20 or so are affected. But always the same messages are affected every time I try an import (even on different systems). I don't know if it also happens with new emails, because until I can be sure that Mozilla imports/receives mail correctly, I can't switch to it - I have to continue receiving mail with Outlook Express...

Thanks,
Cristian

[Ogre]

Go to the account settings / server settings and activate the "leave messages on server" option. So you can try and see what mozilla does to your new messages. If there are still problems, you can still stick to OE.

Besides that, I don't know what else could be done.

[Macavity]

Thanks, I'll try that. But maybe the problem would only arise after months of use, because as I said it only happens on a tiny fraction of the messages (about 1%).
And anyway, I would need also to be able to import correctly the old messages, not only the new ones...


Oh, on a side note - your Fermat's quote is great!

Thanks again
Cristian

[Ogre]

Thanks, I'll try that. But maybe the problem would only arise after months of use, because as I said it only happens on a tiny fraction of the messages (about 1%).

I can almost assure you that the problem will only arise while importing mail, not while receiving new one. I have never had any problems with new mails and I use the mozilla mail client since 1.0.

Keep trying!

Oh, on a side note - your Fermat's quote is great!

Thanks!

[Macavity]

I have tried using Mozilla to receive emails for some days, and it did not have problems. I only received 15-20 messages in this time though, so it's not a "proof" that it will always work; but anyway, I'll keep going.
But, is there a way to correct the badly imported messages (besides deleting them)? I could also hand-edit the files if someone knows how to do it, as there are less than 20 messages to edit.
I need to fix them, because they're annoying because they appear at the bottom of the list of messages; so, to find the newest messages, I have to scroll up. I am always presented the "wrong" messages first... :-\

Thanks,
Cristian

[herman]

I have tried using Mozilla to receive emails for some days, and it did not have problems. I only received 15-20 messages in this time though, so it's not a "proof" that it will always work; but anyway, I'll keep going.

Fine, and I think you won´t have problems, or at least see them directly.

But, is there a way to correct the badly imported messages (besides deleting them)? I could also hand-edit the files if someone knows how to do it, as there are less than 20 messages to edit.

You´ve got some hundreds of emails reproducibly converted well,
and some twenty reproducibly not converted.
There must be some difference, can you spot it?
Did you look at the headers of the mails?
To do this, you must set View->Headers to ALL, and then click on the [+] preceding the Subject of a mail, to see the full header.
Do they have a special mail client in common?
Is the From: line correctly placed, see the link below.

http://searchcio.techtarget.com/sDefini ... 56,00.html


A mbox file is a collection of single mails, separated by newlines.
The mails must have a format according to rfc822 standard.
Found some links by using google, see below.

If you want to edit, editing the folders seems to be easy, the mbox format specifies that the messages are separated by a newline.
And I´m sure you´re not editing the originals, only a copy ;-)

http://www.die.net/doc/linux/man/man5/mbox.5.html

http://resin.csoft.net/cgi-bin/man.cgi? ... topic=mbox

Reading and writing mbox style mailbox files.
http://aspn.activestate.com/ASPN/Cookbo ... ipe/157437

The mails must be in RFC822 format, maybe the unconverted mails had some flaws according to RFC822

http://www.faqs.org/rfcs/rfc822.html

I need to fix them, because they're annoying because they appear at the bottom of the list of messages; so, to find the newest messages, I have to scroll up. I am always presented the "wrong" messages first... :-\

If you click on the Date field, the sorting changes,
from newest ->oldest
to oldest -> newest and vice versa.


And please, if you found out what was wrong, report it here.

thanks,
herman

[Macavity]

Herman, thanks for your informative answer! I will try what I can when I get back home this evening.

You´ve got some hundreds of emails reproducibly converted well,
and some twenty reproducibly not converted.
There must be some difference, can you spot it?

I will extract the headers of a "good" and of a "bad" message and post them here, but I couldn't see nothing strange. Especially the date seemed correct.

If you click on the Date field, the sorting changes,
from newest ->oldest
to oldest -> newest and vice versa.

No, I can't do that. Even if the date itself appears to be old (the displayed date is 2001, while the actual date is 2002 or early 2003), Mozilla shows them as if they were the newest. I usually have Mozilla (and OE) set up to show newest messages at the bottom: and the "offending" messages are below all the others. If I reverse the sorting orders, new messages are on top, but the offending messages are above all the others.

Anyway, I'll check the headers, and post here the examples as soon as possible...

Thanks,
Cristian

[Macavity]

Herman, I had a look to the headers of the messages... I took two messages, one good and one bad, to see if there were differences. (by the way, not all the bad messages are from the same sender, and not all the messages sent by the senders of the bad messages are bad).

And I found something!! It seems that all the bad messages have a VERY long "References" line (what are the "References" anyway?); this line is so long that it gets corrupted (see below), and it seems that this messes things up a lot.

What is strange though, is that if I open a bad message from within Mozilla, and look at the header, everything is OK: the date, the subject, everything. But in the mesages list it has no subject and wrong date (and wrong sorting order). Weird!

I tried fixing a message, by opening the messages file in a text editor, and fixing the References line; but it doesn't work, because it's too late - the msf file is already built, and messed up. I should fix the msf too, but I don't know how. If I delete the msf file, Mozilla creates a new one, "remembering" the old messages, even the old deleted ones! (?!?)

So, maybe this is really what is causing the problem? Can someone experiment with this? Why do some messages have such a long References line?

And, how can I fix this?

Anyway, here I add the headers for two messages. Bad message:

From - Mon Jan 1 00:00:00 1965
Return-Path: <ccc10156@mail.dk>
Delivered-To: macavity80@softhome.net
Received: (qmail 26261 invoked by uid 417); 12 Mar 2003 00:04:59 -0000
Received: from pfepb.post.tele.dk (193.162.153.3)
by 192.168.0.30 with SMTP; 12 Mar 2003 00:04:59 -0000
Received: from athlon (0x50c48a63.adsl-fixed.tele.dk [80.196.138.99])
by pfepb.post.tele.dk (Postfix) with SMTP id 47E945EE138
for <macavity80@softhome.net>; Wed, 12 Mar 2003 01:04:56 +0100 (CET)
Message-ID: <001301c2e82b$0430cce0$0801a8c0@athlon>
From: "Christian Sorensen" <ccc10156@mail.dk>
To: "Cristian Arezzini" <macavity80@softhome.net>
References: <001e01c2d11a$3bae0390$0801a8c0@athlon> <005b01c2db91$8238da20$c3692dd5@athlon> <001e01c2dd86$01a79030$0801a8c0@athlon> <000f01c2dda4$3f4a3c80$20e5abd4@athlon> <002501c2deb4$5eb7f970$0801a8c0@athlon> <000e01c2dfd6$d93acce0$99d52dd5@athlon> <001401c2dfda$d9126080$0801a8c0@athlon> <001e01c2dffb$d4f7f340$90d52dd5@athlon> <001001c2e0bc$b3bb1150$0801a8c0@athlon> <001a01c2e105$810e9da0$38e5abd4@athlon> <001401c2e17f$93286730$0801a8c0@athlon> <001101c2e189$37955d60$7fd52dd5@athlon> <000e01c2e18f$2b6e0a40$0801a8c0@athlon> <002f01c2e194$f1cc4260$71d52dd5@athlon> <001201c2e1c9$18573640$0801a8c0@athlon> <003801c2e1db$f40438c0$08e5abd4@athlon> <000b01c2e237$6e18c170$0801a8c0@athlon> <005001c2e24b$c2293420$1ae5abd4@athlon> <000b01c2e36f$40a25290$0801a8c0@athlon> <000601c2e3d8$69b4e8a0$7ad52dd5@athlon> <000c01c2e3fa$a80e6500$0801a8c0@athlon> <002601c2e4dd$4b9ed140$b7c4b650@athlon> <003201c2e645$46fa35d0$0801a8c0@athlon

on> <000e01c2e684$07a7bc60$95c4b650@athlon>
Subject: Status Report
Date: Wed, 12 Mar 2003 00:04:56 -0000
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0010_01C2E82B.03D391B0"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2720.3000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000


You can see there's an interruption, after the last "athlon" there is a new line and then "on" is repeated.
Here is a "good" message, from the same sender:


From - Mon Jan 1 00:00:00 1965
Return-Path: <ccc10156@mail.dk>
Delivered-To: macavity80@softhome.net
Received: (qmail 23200 invoked by uid 417); 23 Feb 2003 10:51:56 -0000
Received: from pfepb.post.tele.dk (193.162.153.3)
by 192.168.0.30 with SMTP; 23 Feb 2003 10:51:56 -0000
Received: from athlon (0x50c48a63.adsl-fixed.tele.dk [80.196.138.99])
by pfepb.post.tele.dk (Postfix) with SMTP id 96D585EE19B
for <macavity80@softhome.net>; Sun, 23 Feb 2003 11:51:52 +0100 (CET)
Message-ID: <000901c2db29$915f9e20$0801a8c0@athlon>
From: "Christian Sorensen" <ccc10156@mail.dk>
To: "Cristian Arezzini" <macavity80@softhome.net>
References: <001e01c2d11a$3bae0390$0801a8c0@athlon> <000301c2d149$80cf9220$28e5abd4@athlon> <001e01c2d159$875319e0$0801a8c0@athlon> <000f01c2d1f7$da330820$80d52dd5@athlon> <000901c2d1fc$394b86d0$0801a8c0@athlon> <008401c2da66$daed4f60$c8e12dd5@athlon> <000c01c2da84$5f60e8c0$0801a8c0@athlon> <003501c2daaa$38c31640$3ae5abd4@athlon> <000e01c2dac8$9005aa80$0801a8c0@athlon> <001501c2db22$dc963e00$1de5abd4@athlon>
Subject: Re: Mainboard update
Date: Sun, 23 Feb 2003 10:51:49 -0000
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0006_01C2DB29.911F87E0"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2720.3000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000


Here the "references" line is much shorter, and not corrupted. And some other "good" messages don't have a "references" line at all.

Any thoughts?

Thanks,
Cristian

[herman]

Herman, I had a look to the headers of the messages... I took two messages, one good and one bad, to see if there were differences. (by the way, not all the bad messages are from the same sender, and not all the messages sent by the senders of the bad messages are bad).

And I found something!! It seems that all the bad messages have a VERY long "References" line (what are the "References" anyway?); this line is so long that it gets corrupted (see below), and it seems that this messes things up a lot.

What is strange though, is that if I open a bad message from within Mozilla, and look at the header, everything is OK: the date, the subject, everything. But in the mesages list it has no subject and wrong date (and wrong sorting order). Weird!

I tried fixing a message, by opening the messages file in a text editor, and fixing the References line; but it doesn't work, because it's too late - the msf file is already built, and messed up. I should fix the msf too, but I don't know how. If I delete the msf file, Mozilla creates a new one, "remembering" the old messages, even the old deleted ones! (?!?)

So, maybe this is really what is causing the problem? Can someone experiment with this? Why do some messages have such a long References line?

And, how can I fix this? ;-)


Any thoughts? ;-)

Thanks,
Cristian

Nice catch!
File a bug ;-)

Send me your mail address using the [Pri.Msg.] field below,
this is getting too long to discuss here.

Edit: Just saw that you can´t use this function, as you are not a member. Can I use the softhome.net address to send you a mail?

I also don´t know what references are, or what RFC822 specifies on this, but I assume, References is specific to Outlook, and they are breaking some spec by making it too long. But what are specs, when there is Microsoft?

http://www.faqs.org/rfcs/rfc822.html
is a standard from 1982, for email, but I don´t think, they specified Outlook in there ;-)

Start reading at
3. LEXICAL ANALYSIS OF MESSAGES

3.1. GENERAL DESCRIPTION

A message consists of header fields and, optionally, a body.
The body is simply a sequence of lines containing ASCII charac-
ters. It is separated from the headers by a null line (i.e., a
line with nothing preceding the CRLF).

3.1.1. LONG HEADER FIELDS

Each header field can be viewed as a single, logical line of
ASCII characters, comprising a field-name and a field-body.
For convenience, the field-body portion of this conceptual
entity can be split into a multiple-line representation; this
is called "folding".

...

3.4.6. BRACKETING CHARACTERS
...
There are three types of brackets which must occur in matched
pairs, and which may NOT be nested:

o Colon/semi-colon (":" and ";") are used in address
specifications to indicate that the included list of
addresses are to be treated as a group.

o Angle brackets ("<" and ">") are generally used to
indicate the presence of a one machine-usable refer-
ence (e.g., delimiting mailboxes), possibly including
source-routing to the machine.

The bad headers didn´t have the last pair of Angle Brackets as a pair, as the pair included a newline or some other whitespace.

I would suggest:
Backup the original data, so you can file a bug later on.
Fix the problem.

Can you export/save as the bad mails from Outlook, i.e. use Outlook to make a single file for each one of some of the bad mails, icncluding the headers? I don´t know/have Outlook or OE, so I don´t know if it can be done, or what must be configured to include headers, as headers are normally invisible.
Maybe you can cut them directly from the MBOX files, if Outlook uses MBOX, I don´t know.
If you can do the same with the Mozilla generated mail, File->Save As File, you get a mail.eml file, and can compare.
I assume the reference will be shorter in the mozilla generated mail.

Re: msf files.

MBOX stores all mail stacked like a stack of dishes.
As you don´t want to move a lot of bytes around if you want to delete short mail in the big file, ( try to remove the second dish in a stack of 30 dishes, you can´t just pull it out and hope that gravity will do the rest of the work ;-)
Therefore deleted files are just marked deleted in the .msf file, which also holds flags like read, or others.
The deleted files are removed from the mbox, when you compact folders. This is a time-consuming process, you can do it for all folders, using File->Compact Folders, or individually by selecting a folder, right-click, and then Compact This Folder.
Maybe that after compacting the bad headers are also fixed.

If you the delete the .msf files, they are still rebuild, marking all mail as unread, but at least you don´t have to read and delete that mails again, you had deleted, as they are gone.

I don´t know how long this takes, it will be some minutes depending on the size of the folder and the speed of the computer and the speed of the harddisk, but I assume this process shouldn´t be interrupted.

Do this only, when you still have a backup of your original MBOX files, or can reimport from Outlook, as you need the bad mails to file a bug.

herman

[Macavity]

Hi Herman!

First of all, of course you can email me at my softhome address... (by the way, I now realize that maybe I should have "masked out" the addresses in the headers, to avoid spam... Oh well.)

I'm a complete newbie about the internal structure of email messages - I grew up with Microsoft, and they don't want you to look in the "guts" of the data!! That's why I want to change client.
I looked in the RFC 822 document, and found the following:

4.6.3. REFERENCES

The contents of this field identify other correspondence
which this message references. Note that if message identif-
iers are used, they must use the msg-id specification format.

So, basically those bad messages make a reference to previous mail exchanges between us. Maybe that field builds up when we reply to a message instead of creating a new one.
So it seems that field can be safely edited to make it shorter... This means two things though:
1- Mozilla can't correctly deal with long References fields. I don't know though if Mozilla or Microsoft are out of spec.
2- As it is possible that someone will send me emails with long references in the future, the problem will possibly happen again... Unless the bug is only in the "import" code, and not in the code which decodes the headers.

My Outlook data is all backed up... I made a backup before trying to import data with Mozilla, just in case! So the bug is reproducible. I don't know if this is limited to Windows, or also to other OSes... I have Linux and would like to try, but from Linux I can't access my Windows data, so I can't make an import.

Outlook can export messages as .eml files. I thought they were "encrypted", but maybe they aren't... I'll check this evening. If they are not encrypted, I could export the bad messages from within Outlook - after locating them, :-o - then hand-edit the headers, then import them in Mozilla... Could work. I will try that.

What if I receive a "bad" message from within Mozilla? How could I fix it? Should I do the following:
- Put it in a temp folder
- exit Mozilla
- Hand-edit the temp folder mail file, and delete the msf file
- restart Mozilla, select the folder and compact it (this should fix the error?)
- Put the message back in the original folder, then delete the temp folder

Would this work?

About compacting folders: this doesn't fix the problem in a folder with bad messages, I already tried that.

So, if you email me and tell me how to file a bug, I will do it! In the meanwhile I'll try the import experiment ASAP.

Many thanks!

Cristian