Message Database "Corruption" with TB 1.5.0.8

Discussion of bugs in Mozilla Thunderbird
soulek
Posts: 75
Joined: November 7th, 2005, 4:18 am

Message Database "Corruption" with TB 1.5.0.8

Post by soulek »

Yesterday, I upgraded from TB 1.5.0.7 to 1.5.0.8 (via uninstall/reinstall). I've discovered a serious bug, I believe, in 1.5.0.8.

I have been using TB (and it's predecessors, Mozilla and Netscape) for a number of years. My messages go back to April 1998. I have them sorted into folders, and use filters to auto-sort them upon arrival.

One of my folders is named "SoftwareCo" ... a company for which I do a lot of beta (and am currently actively corresponding with). The folder currently has 343 messages, and the file for the folder is 7,351 KB.

My messages to that folder go first to GMail, so I have a GMail archive of all my messages, and are then downloaded from GMail via POP3. After I installed 1.5.0.8, I noticed that when a new message would come in (in GMail), TB would download it, and a "new message arrow" would appear on the SoftwareCo folder, but the message itself was nowhere to be found. It wasn't in the folder, or the Inbox, or Junk, or the Trash ... it was just gone.

So today, I started working with the message database, and at first thought it was corrupt. When TB starts, I can see all 343 SoftwareCo messages. If I delete the SoftwareCo.msf file, and allow it to regenerate upon TB restart, it shows 271 messages, rather than 343. If I do a CTLR-A (on the original 343 messages), and try to copy (or move) the messages to a different folder, if they moved at all, it is 271 ... in many cases, they wouldn't move at all (and sometimes the attempt freezes TB). I tried various tools to "reset" the TB message bits, but the results are the same. Compacting the folders is not an issue.

Then, on a hunch, I reinstalled 1.5.0.7, and my fully functionality was restored (except that the messages that were lost yesterday are still lost). With 1.5.0.7, I can move ALL the messages from the SoftwareCo folder to another folder and then back again, copy any of them, and if I delete the SoftwareCo.msf file, it regenerates with the full 343 messages, not 271. Clearly, a major bug in 1.5.0.8.

It appears that 1.5.0.8 knocks out all of the messages that were sent to a mailing list, or that were generated by an auto-responder (we acknowledge your bug report, number XXX). The messages from an individual Tech to me survive. The date that a particular message was sent seems irrelevant.

I probably had (or would have had) similar problems with messages in other folders, but most of yesterday's traffic was with SoftwareCo.

Who do I write about this regression? (I could file a Bugzilla report, but I'm not sure I have enough info).
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Post by tanstaafl »

Moved from the Thunderbird bugs forum to the Thunderbird support forum

See if http://kb.mozillazine.org/Disappearing_mail applies. The mozillaZine topic notification messages have a empty To: field. Do they get lost?

I'm using a nightly build of 2.0 and it doesn't have this problem. I haven't tried 1.5.0.8.
soulek
Posts: 75
Joined: November 7th, 2005, 4:18 am

Post by soulek »

tanstaafl wrote:Moved from the Thunderbird bugs forum to the Thunderbird support forum

See if http://kb.mozillazine.org/Disappearing_mail applies.

http://kb.mozillazine.org/Disappearing_mail is no solution. So far, the issue applies to but a single folder (out of 52 in the Inbox, and 31 in the Sent folders). I deleted the SoftwareCo.msf, and allowed it to regenerate. No difference.

tanstaafl wrote:The mozillaZine topic notification messages have a empty To: field. Do they get lost?

My original post was imprecise. I wrote:
soulek wrote:It appears that 1.5.0.8 knocks out all of the messages that were sent to a mailing list, or that were generated by an auto-responder (we acknowledge your bug report, number XXX). The messages from an individual Tech to me survive. The date that a particular message was sent seems irrelevant.

In looking at the messages that are adversely affected in the SoftwareCo folder, they appear to be limited to either (i) mass mails, or (ii) auto-responder generated. Mass/anonymous mail in *other* folders does not appear to be affected. There may be something unique in the SoftwareCo messages. Thus, the fact that mass mails you have received do not suffer a similar fate is not dispositive.

tanstaafl wrote:I'm using a nightly build of 2.0 and it doesn't have this problem. I haven't tried 1.5.0.8.

I don't know how you could possibly test this w/o a copy of my SoftwareCo file ... particularly since the issue appears to be unique to that folder. Very bold of you to summarily state "it doesn't have *this* problem." How do you know? (See my subsequent post for a test on the "nightly" trunk build.)

I absolutely disagree that this is a "support" issue.

1.5.0.7 has no issues ... 1.5.0.8 does ... return to 1.5.0.7 and all is fine. There's a message there.

If you insist on a "pocket veto", I'm headed to Bugzilla.
Last edited by soulek on November 8th, 2006, 6:16 pm, edited 1 time in total.
soulek
Posts: 75
Joined: November 7th, 2005, 4:18 am

Post by soulek »

I just downloaded the most recent "nightly" ... thunderbird-3.0a1.en-US.win32.installer.exe (Nov 8, 2006 @ 12:42), and installed it. Same result as 1.5.0.8 ... a portion of my SoftwareCo database disappears.

If I uninstall the nightly, and reinstall 1.5.0.7, the SoftwareCo messages are still lost (possibly because, while the nightly was installed, I copied the files from the SoftwareCo folder to an empty folder, and then back again). If I then close TB 1.5.0.7, delete "SoftwareCo" and "SoftwareCo.msf" from the Inbox folder, and restore both of those files from a Ghost image (taken at 4:30a this morning), then all the SoftwareCo messages are back.
User avatar
Daifne
Moderator
Posts: 123071
Joined: July 31st, 2005, 9:17 pm
Location: Where the Waters Meet, Wisconsin

Post by Daifne »

Sounds to me like the folder is corrupted. Have you been compacting regularly? http://kb.mozillazine.org/Compacting_folders
soulek
Posts: 75
Joined: November 7th, 2005, 4:18 am

Post by soulek »

Daifne wrote:Sounds to me like the folder is corrupted. Have you been compacting regularly? http://kb.mozillazine.org/Compacting_folders

I agree ... sounds like it ... but it doesn't appear to be (otherwise, it wouldn't display perfectly in 1.5.0.7).

For compacting, I use the Xpunge extension ... it is set to compact on startup, and every 4 hours thereafter. That's pretty "regular."
User avatar
Daifne
Moderator
Posts: 123071
Joined: July 31st, 2005, 9:17 pm
Location: Where the Waters Meet, Wisconsin

Post by Daifne »

Since deleting the .msf file sort of fixed it, I would say that it is still some sort of corruption. Sometimes an upgrade can magnify the existing corruption. We could try the sledge hammer approach:

If compacting and deleting didn't work, you will need to try the sledge hammer approach:

1. Move any mail in your Inbox - that you want to retain - to subfolders of Local Folders. Backup your profile. http://kb.mozillazine.org/Profile_backup You really shouldn't be storing them there anyway. The inbox is too busy in any email program and is therefore more prone to corruption. If you cannot see the messages to move them, close Thunderbird and rename the effected inbox file with no extension to something like InboxOld

2. Close down Thunderbird

3. Navigate to your Thunderbird Profile Folder on your system and DELETE the 'Inbox' and 'Inbox.msf' files for this account, or for Local Folders if you are using the global inbox. They will be rebuilt when you relaunch Thunderbird. If you have more than one account, make sure you do this in the proper account. All messages left in the inbox will be deleted in this process. If you do make a mistake, I hope you made the backup I suggested in step 1.

4. If InboxOld does not show at this point, let me know.

To avoid this happening again, see the tips in these articles:
http://kb.mozillazine.org/Keep_it_worki ... derbird%29
http://kb.mozillazine.org/Performance_%28Thunderbird%29
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Post by tanstaafl »

I absolutely disagree that this is a "support" issue.
The support forum is the forum to get help. You were looking for help and I wanted your thread to get more exposure since the bugs forum is a backwater.

This web site is not run by or formally associated with Mozilla. We're a independent user community. I don't see what the fuss is about. However, since you feel strongly enough to have sent me a private message I'll move this thread back to the bugs forum.

The best place to report a bug is at https://bugzilla.mozilla.org/ . That will guarantee that a developer will both read the bug AND track it. They're unlikely to read it in either forum since the forums (except for the Builds forums) do a poor job of meeting their needs. Its not clear you found a bug yet. Given how long (sometimes years) it takes for a developer to respond I'd think it would be in your best interest to get some help identifying whether your "bug" is a bug or just a problem with your profile before reporting it in bugzilla.
soulek
Posts: 75
Joined: November 7th, 2005, 4:18 am

Post by soulek »

Daifne wrote:Since deleting the .msf file sort of fixed it, I would say that it is still some sort of corruption.

I don't see how you concluded from what I wrote that deleting the .msf file fixed anything. If I delete the SoftwareCo *msf file. and let it regenerate in 1.5.0.7, I see 343 messages (the original number); if I let it regenerate in 1.5.0.8, I end up with 271.

Daifne wrote:Sometimes an upgrade can magnify the existing corruption. We could try the sledge hammer approach:

I'm not ready for that, but I've done more testing, which I believe gives greater credibility (proof?) to my belief that what I am experiencing is a bug/regression.

Working in both 1.5.0.7 and 1.5.0.8, I was able to identify the 72 "problematic" messages out of the 343 in the SoftwareCo folder. While in 1.5.0.7, I created a new folder, and moved the 72 problematic messages there. I left the 271 "non-problematic" messages in the original folder. While still in 1.5.0.7, I verified the integrity of the messages; I moved the problematic messages to a new folder, and moved them back; I copied 6 of them to a new folder ... and then moved them to a new folder and moved them back; and I moved the non-problematic messages to a new folder, and moved them back. I compacted the folders between moves. In summary, in 1.5.0.7, I was able to view and manage the messages, and the several folders as one would expect (and in 1.5.0.7, there is no distinction between problematic and non-problematic messages ... they are all the same).

I then uninstalled 1.5.0.7, and installed 1.5.0.8. I could freely move and manage the 271 non-problematic messages, but if I tried to move either the 72 problematic messages, or the subset of 6 messages in a different folder (by using CTRL-A, and then right mouse "Move", etc.), the Status Bar would show "Moving 6 of 6 messages", and I would get a *perpetual* hourglass in TB. The only escape would be to exit the program. I would then restart TB, and I would find that a single message (1 and always only 1) had been copied to the new folder, and that message had NOT been deleted in the old folder (i.e., the "move" had not been completed).

Clean Profile
++++++++

I use extensions. And though I check/clean it regularly, my profile is "seasoned." So to further isolate the issues, I created a new TB profile ... with no extensions, and no messages. I copied 3 message "hives" (is that the correct term?) or folders from MY profile ... i.e., the folder of 271 non-problematic messages, the folder of 72 problematic messages, and the folder of 6 problematic messages ... into the Inbox of the new profile. Messages only, no *msf files.

With 1.5.0.7, all was well. TB generated new .*msf files for the 3 folders, and I could view and manage the messages as I would expect. With 1.5.0.8, TB also generated new *msf files ... and I could see all of the non-problematic messages, but I could only see a *single* message in the folders with problematic messages ... that is, 1 out of 71, and 1 out of 6. If I used the *.msf files generated by 1.5.0.7 in 1.5.0.8, I could see ALL of the messages, but I had the same problem working with them that I had when I was using my own profile.

Conclusion: it isn't a profile issue, and it isn't an extension issue (because I have identical results with a clean profile and no extensions).

Message Characteristics
+++++++++++++++

So now I look at the problematic messages ... how are they different from the 271 others. All of the problematic messages were either mass-mailed to the beta group, or were sent to me using an auto-responder (after filing a bug report). All of the problematic messages after May 2006 show an X-Mailer of "Chilkat Software Inc (http://www.chilkatsoft.com)"; problematic messages before May 2006 show an X-Mailer of either JMail 4.4 by Dimac or JMail 4.5 by Dimac. The non-problematic messages typically came from a Microsoft Exchange server or, in some cases, Lotus Notes. It is curious that ALL of the Chilkat messages are problematic, and none of the Microsoft Exchange server messages are.

I don't believe that the database files themselves are corrupt, as in 1.5.0.7 I can easily (and successfully) work with all the messages. In any event, the newly-created test file of 6 problematic messages should be free of corruption, as individual messages where moved into it one by one. It is only 23 KB.

It is my sense that there is something unique about the way the problematic messages are structured that is not being handled correctly by TB 1.5.0.8 (or by the trunk nightly of November 8, 2006).

Comments?
soulek
Posts: 75
Joined: November 7th, 2005, 4:18 am

Post by soulek »

More testing ... more data ... getting much closer to the source of the problem.

In 1.5.0.8, attempt to move (by any method) the 6 test messages to a new (empty) folder. TB will "hang" (perpetual hourglass) ... so close TB and restart. You will see that it appears that a single message has been *copied* (and nothing has been "moved", because the original folder of 6 messages is the same).

Now look at the actual folder files (in my case, in Inbox.sbd). I find that the message database file for the destination folder is *identical* to the message db file of the *source* folder. Thus, the messages *were* copied. However, the .msf is not being generated correctly, so I only "see" 1 message (even though there are 6), and the error in the .msf generation prevents the "move" operation from completing (i.e., deletion of the original messages).

This behavior is consistent with my earlier finding that if 1.5.0.8 generates the .msf files for either of the problematic test folders ... whether 72 or 6 messages ... only 1 message is indexed. It is an indexing issue (and, as noted in the very first post, this issue also prevents new messages from being added to the db ... and they are lost altogether).
dr_zeus
Posts: 4
Joined: November 9th, 2006, 9:20 pm

Post by dr_zeus »

This looks like the same issue that I'm having. I'm scared to experiment because I've seen ways that many of the missing messages completely disappear.

My topic:
http://forums.mozillazine.org/viewtopic.php?t=486501
zeus
dr_zeus
Posts: 4
Joined: November 9th, 2006, 9:20 pm

Post by dr_zeus »

Additional info:

Like you, this only happened recently, I assume its because I upgraded to 1.5.0.8.

I have also seen the hanging issue when trying to move any messages to another folder.

After reading through all your posts again, I discovered that my messages had an X-Mailer of "Chilkat Software Inc" too.
zeus
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Post by tanstaafl »

I took a quick look at six problematic messages that soulek sent me (I'm still investigating). They used two mailers, "JMail 4.4 by Dimac" and "Chilkat Software".

It seems that one problem might be that the "JMail 4.4 by Dimac" mailer didn't include angle brackets in the Message-ID: header. The RFCs require the angle brackets. The Resent-Message-ID: and References: headers had the same value as the Message-ID: header but that doesn't seem to cause a problem.

I suspect many email clients are able to ignore the missing angle brackets in the Message-ID: header, but for some reason the behavior changed in 1.5.0.8 . I found that if I edited the message source (using a text editor) in the mbox file to add the missing angle brackets the problem went away. I could drag and drop the messages (or use the move/copy menu commands) with no problems using 1.5.0.7, 1.5.0.8, or a nightly build of the 2.0 branch after doing that.

The messages with a "X-Mailer: Chilkat Software Inc (http://www.chilkatsoft.com)" header had angle brackets. Perhaps there is a different problem with the headers in those messages.
dcnicholls
Posts: 35
Joined: April 27th, 2005, 9:58 pm
Location: Canberra, Oz

Post by dcnicholls »

Aha! I'm seeing the same (or a similar) problem with the Chilkat mailer. See my message under "Severely weird" thread.
dcnicholls
Posts: 35
Joined: April 27th, 2005, 9:58 pm
Location: Canberra, Oz

Post by dcnicholls »

Here's another possible clue, and certainly a new bug (in addition to this one). It might have something to do with the way filters are working in 1.5.0.8. Here's what I did.

Old filter: Messages with "xxx" in sender > Move to "forums" folder
Result: messages transferred to that folder but don't show up in the index list for "forums"

New filter:
Messages with "xxx" in sender > Copy to "forums" folder AND Move to "forums2" folder (a new one)
Result: messages don't appear in "forums" but DO appear in "forums2"
BUT: messages from main inbox get put in "forums" (bizarre) - this is the new bug.

Next step: change filter to:
Messages with "xxx" in sender > Move to "forums2"
(ie reomve the initial Copy to "forums")
Result: reverts to the original problem.

Synopsis: adding an initial Copy to allows viewing (and drap/dropping) of new filtered messages, but has odd results with normal message deletion.

I'm going to (try to) reinstall 1.5.0.7 and will report back on the results.
DN
Post Reply