Resurrected empty folders

[tanstaafl]

I'm using the 24.0a1 (2013-06-15) build under Windows 7. I have a IMAP account with Fastmail.fm that uses a Cyrus server. The INBOX.Misc.DeveloperWorks , INBOX.Misc.Java.JavaRanch , INBOX.Misc.Java.Serverside, and INBOX.Misc.purchases child folders had been deleted several months ago. I expanded the Inbox.Misc folder hierarchy this evening and noticed those folders existed, in grey italic letters. The folder pane lists those folders both with and without safe mode. The folders are empty, are listed in panacea.dat, and do not exist in webmail.

The profile is shared by both daily and 17.0.6. If I run 17.0.6 it does not list any of those folders. None of those folders have mbox files (global search/indexing and offline folders are disabled). I suspect I could get rid of the bogus listing by exiting Thunderbird and deleting the *.msf files, and if that didn't work delete panacea.dat. However, I want to understand why this happens before I do that. One reason is that several times over the last couple of months I got a empty grey italic inbox folder as a child folder of the inbox. It went away for a couple of weeks whenever I deleted it. Each time I deleted it I used file -> compact folders since the automated compacting when it will save over 1MB never runs (most of my messages are short plain text messages with no attachments, mail.imap.expunge_option is set to 2 (expunge the deleted messages when there are at least 20 messages waiting to be expunged), and the inbox is automatically compacted whenever I exit).

I have several other accounts with different IMAP servers and they have never had this problem. However, the only other account that I think I've deleted any folders on in the last couple of months was gmail.

http://kb.mozillazine.org/Grey_italic_folders states "A IMAP folder will be displayed in grey italics if the folder has a \Noselect flag. Normally this means the folder can contain child folders, but not messages." A Cyrus IMAP server has no problem with folders that contain both child folders and messages.

As an experiment I right clicked on "purchases", selected Delete, and confirmed that I wanted to delete the folder. It did. I then tried to copy a message from the inbox to Serverside using Message -> Copy To. It doesn't list that folder in the folder listing (or any of the other grey italic folders). I then tried to drag and drop the message there instead - that failed as the mouse cursor never changed from a circle with a slash through it. I browsed the folder hierarchy in the Subscribe window and it listed all of the grey italic folders. So it appears that only the folder pane and the list of subscribed folders knows about those folders.

As an experiment I created a xyzzy child folder in Inbox.Misc. , exited and restarted. That new child folder is empty but is displayed normally. I deleted the folder, exited , and restarted. The deleted folder was not resurrected.

The sharing tab in the folder permissions normally says "You have the following permissions: Read, Write, Insert (Copy Into), Set Read/Unread State, Delete Messages, Expunge, Create Subfolder, Post". For the grey italic folders it says "You have the following permissions: Full Control".

Ideas? Is this a known problem?

[tanstaafl]

I had been using my normal profile. I just tested using another profile which only has a Fastmail IMAP account and it has the same problem with both the 32 and 64 bit 6-17 daily build (but not 17.0.6). That makes it much easier to log IMAP traffic since the log file won't have intermingled entries from other accounts. The config editor has not been used to modify any of the profiles settings, though offline folders and global search/indexing are disabled.

I started Thunderbird with IMAP logging enabled, opened the subscription window for the inbox, refreshed it, closed it and then tried to open one of the grey italic folders.

Looking in the log file I noticed 3212[974df70]: 8988800:mail.messagingengine.com:A:SendData: 8 xlist "" "%.%" didn't return any of the grey italic folders but
3212[974df70]: 8988800:mail.messagingengine.com:A:SendData: 9 list (subscribed) "" "INBOX.*" did:

3212[974df70]: 8988800:mail.messagingengine.com:A:CreateNewLineFromSocket: * LIST (\NonExistent \Subscribed) "." INBOX.Misc.DeveloperWorks
3212[974df70]: 8988800:mail.messagingengine.com:A:CreateNewLineFromSocket: * LIST (\NonExistent \Subscribed) "." INBOX.Misc.Java.JavaRanch
3212[974df70]: 8988800:mail.messagingengine.com:A:CreateNewLineFromSocket: * LIST (\NonExistent \Subscribed) "." INBOX.Misc.Java.Serverside

later on the call to "discoverallandsubscribedboxes" (which I assume is from refreshing the subscribe window) also returned the last three lines.

No network calls were made by my attempt to open the non-existent developerworks folder.

My first thought was that daily is not paying attention to the \NonExistent flag while 17.0.6 does. However when I reran it using 17.0.6 the log file did not contain any \NonExistent flags or any of the folders that Daily displayed as grey italic folders.

Right after all of the XLIST commands Daily sends:
3212[974df70]: 8988800:mail.messagingengine.com:A:SendData: 9 list (subscribed) "" "INBOX.*"
this returns the data that seems to cause the problem

17.0.6 sends instead
3308[7c4f2e0]: c3f1800:mail.messagingengine.com:A:SendData: 9 lsub "" "INBOX.*"

According to RFC 3501 "The LIST command returns a subset of names from the complete set of all names available to the client." while "The LSUB command returns a subset of names from the set of names that the user has declared as being "active" or "subscribed"."

Could this be the reason for the difference in behavior? Its not clear to me what active means. I'm wondering if my deleting a remote folder just caused it to be inactive rather than physically deleted on the IMAP server. I don't have Windows Mail configured but ran eM client 5.0 instead and verified it did not have this problem.

If this makes sense is there any other information I should try to collect before filing a bug report?

[tanstaafl]

I filed https://bugzilla.mozilla.org/show_bug.cgi?id=885162 against MailNews Core

[Lee_Dailey]

howdy tanstaafl,

i'm so freaking lost when i read this ... i'm glad someone does this sort of thing, but i'm also glad that i am not doing it. [*grin*]

take care,
lee

[denali]

I had the exact same symptoms here after upgrading to the release version of Thunderbird 24, with three long-deleted folders appearing greyed out under my Fastmail Inbox. I was unable to delete or move these folders. I tried all of the usual stuff, creating new profiles and deleting index files. So convinced was I that the problem had to be local that I was considering installing TB on a virgin machine just to prove the point. But then I stumbled upon this thread which, along with the follow-ups in the Bugzilla report linked above, helped me to effect a solution.

I don't claim to understand what exact mechanisms caused this but, in case anyone else finds this thread with a search, here's the solution that worked for me.

1. Set Thunderbird to permanently delete messages instead of sending them to Trash.

This may not be needed for all users, but one of my rogue folders was a former Trash folder and I didn't want to risk sending any deleted files into it by mistake.

2. Use the Advanced Account Settings dialog to set the IMAP Server Directory to null, then restart Thunderbird.

This isn't the normal way things are set up with Fastmail, but it has the effect of "flattening" the folder tree under the Inbox. In my case there were three greyed out folders under my Inbox called Pending, Pre-Spam (from an early filtering experiment) and Trash (an old IMAP trash folder that got orphaned and deleted months ago). Once I'd complete this step the three folders disappeared from beneath the Inbox and appeared instead within their own top-level folders. Pending appeared under a new folder called INBOX (all caps) while Pre-Spam and Trash appeared under a new folder called Inbox (cased). All of these folders were greyed out.

3. Create new folders with the same names and in the same locations as the greyed folders.

Because the grey folders are flagged on the server as both Non-existent and Subscribed, this step has the effect of both overwriting them (thus making them "not Non-existent") and marking them subscribed on both the server and in Thunderbird. Note that this didn't work for my former Trash folder, which Thunderbird would not let me create. Instead I renamed this folder to Trash1 which, fortunately, had the same effect of reaffirming the subscription in Thunderbird.

4. Delete the now un-greyed folders.

To be safe I started with the nested folders and worked up, deleting the two rogue INBOX and Inbox folders last. It was probably safe to do this just by deleting the top-level folders but I was experimenting and wanted to make sure.

5. Use the Advanced Account Settings dialog to set the IMAP Server Directory back to "INBOX." and restart Thunderbird.

This puts all of the folders back where they should be on the folder tree.

6. Set Thunderbird to send messages to an appropriate Trash folder if you don't want to always delete.


As with any step-by-step guide I can only offer anecdotal evidence that this worked for me. Everyone else's experience may well be different. And as I said above, I have no idea what set of circumstances led to this happening, although it would appear that something allowed Fastmail's server to allocate both the 'Non-existent' and 'Subscribed' flags to several folders that stayed hidden for months until Mozilla changed the IMAP command for subscription list requests.

My days of parsing server logs and reading RFCs for sh*ts and giggles are long behind me, but it would seem from some of the comments on the Bugzilla page that this behaviour is perfectly compliant with the IMAP protocol. Compliant it may be, but it's certainly annoying.

I hope this helps anyone else with a similar attack of rogue grey folders in the Inbox.