Mail folders verse OS Physical path with Special Characters

[thundergoesboom]

Good Day

I'm trying to figure out how Thunderbird 31.6 handles/manages special characters in Mail folder names as physically stored in the Windows file system (to sort out some issues with a outlook express import)

As an example to test what should work:

I create the following folder under the inbox of my default mail folder "Folder\Test" which creates a physical file of "..\Inbox.sbd\Folder8b445e6f" along with the supporting MSF file

I copy this folder from the back end (via a CMD prompt) to Thunderbirds "local folder", and the correct name is shown within Thunderbird (but the OS still shows Folder8b445e6f)

All works as expected (although not 100% sure exactly how it works)


Now instead of one folder, I'm moving a lot more; but the results are different.

I've imported email (about 1000 sub mail folders with a lot of special characters) from OutlookExpress (imported email is stored in a Outlook import folder under local folders)

I then go about moving the imported email folders from the Local folder to the active email profile mail folder via the back end (OS Cut/paste), and now every "Special" mail folder shows up as how the Windows File system has them stored. IE. Folder\Test now shows up as Folder8b445e6f

I've dug around, and cant figure where the link/mappings stored between the folder within Thunderbird, and the physical folder location (It might be something in the MSF file as the original folder name is referenced sometimes; but I hope not, as I've often recommended/done a MSF purge to correct corrupt index)

Anyone able to point me in the right direction on how it works?


Sincerely

Paul

[trolly]

I'm not sure I understand you.

If you look into the msf file (in fact a text file) you can see something similar to:

(line 21 for me)
<(80=1)(81=4)(82=Testdf11db8f)(83=0)(84=5547e4ae)(85=Test\\Test)(86

That's where the folder name is translated into the OS name.

[thundergoesboom]

Hi Trolly

Thanks for the reply.

I have seen what you described; but I’ve also seen that the msf doesn't always contain the Active Thunderbird folder to OS folder Translation and that thunderbird may be caching the folder translations somewhere else.

As mentioned initially, folders imported from the outlook express import functionality don’t seem to have the 82=/=85 entry pair (the thunderbird folder to OS folder translation) in the MSF files; but the folder translation is still maintained actively within thunderbird.

As a test, I shutdown Thunderbird, deleted "..\Inbox.sbd\Folder8b445e6f.msf", restarted Thunderbird, and the correct "Folder\Test" name was maintained within Thunderbird. On checking the new Folder8b445e6f.msf, the thunderbird to OS translation was still present (the MSF file was deleted....)

I then copied the ..\Inbox.sbd\Folder8b445e6f file (without the msf) to another folder, and opening thunderbird show Folder8b445e6f in the alternate folder. I stopped thunderbird, dropped in the correct Folder8b445e6f.msf (with the correct folder to os conversion); but the directory name didn't correct itself (was still showing Folder8b445e6f)

It honestly looks like there is some sort of caching going on

I’m going to attempt to delete all the MSF files in the Outlook Express conversion folder, and rebuild them to see if the MSF files will be rebuilt with the correct Thunderbird folder to OS folder translations (which should allow me to move them via the back end without have folder name mess ups).


Aside: This raises another question….. If the MSF is the source of the translation, then how is Thunderbirds folder to OS name translation is surviving deleting all the MSF files as outlined here http://kb.mozillazine.org/Compacting_fo ... em_to_work

Sincerely

Paul

[tanstaafl]

I'm using Thunderbird 31.6.0 with a English US version of Windows 7 x64.

I created a child folder of the inbox in Local Folders called ÀÈÌÁâãäŸ using the ALT key and the numeric keypad. Its properties in the folder pane show its name to be ÀÈÌÁâãäŸ , and its location to be mailbox:///C:/Users/Eric/Profiles/Thunderbird/tanstaafl/Mail/Local Folders/Inbox.sbd/ÀÈÌÁâãäŸ . I used windows explorer to verify it created a mbox file called ÀÈÌÁâãäŸ and a index file called ÀÈÌÁâãäŸ.msf

I then tried again using characters from the Cherokee Unicode block to make certain it couldn't be mapped to a ISO/IEC 8859 character set.

I created a child folder of the inbox in Local Folders called ᏠᏡᏢᏣᏤᏥᏦᏧᏨᏩᏪᏫᏬᏭᏮᏯ by copying and pasting the characters from https://en.wikipedia.org/wiki/Cherokee_ ... e_block%29. Its properties in the folder pane show its name to be ᏠᏡᏢᏣᏤᏥᏦᏧᏨᏩᏪᏫᏬᏭᏮᏯ and its location to be mailbox:///C:/Users/Eric/Profiles/Thunderbird/tanstaafl/Mail/Local Folders/Inbox.sbd/fca44689 . I used windows explorer to verify it created a mbox file called fca44689 and a index file called fca44689.msf

I'd assume that the filename matches the folder name as long as all of the characters are part of a Windows code page such as the Windows-1252 character set. Since I'm using a NTFS file system I expected the second folder name to use UTF16 encoding. However, the name it uses has 8 hexadecimal characters instead of the 16 character long folder name I expected. The msf file includes:

<(80=1)(81=4)(82=fca44689)(83=0)(84=55482d51)(85
=$E1$8F$A0$E1$8F$A1$E1$8F$A2$E1$8F$A3$E1$8F$A4$E1$8F$A5$E1$8F$A6$E1$8F$A7$E1\
$8F$A8$E1$8F$A9$E1$8F$AA$E1$8F$AB$E1$8F$AC$E1$8F$AD$E1$8F$AE$E1$8F$AF)>
{1:^9F {(k^A0:c)(s=9u)}
[1(^AC=1)(^AD=1)(^88=4)(^A7^82)(^A3=0)(^A5^84)(^AE^85)(^B9=1)]}

The characters I used were 13E0 through 13EF (hexadecimal). I don't see any thirteens. I see what appears to E18FAX, where X goes from 0 to F. I used http://www.unicodemap.org/ to search for how that Unicode string was stored, expecting it might be UCS-2 or something like that. If you look at the search URL its http://www.unicodemap.org/search.asp?se ... E%E1%8F%AF which uses percent encoding (aka URL encoding), and matches the string in the index file (uses % instead of $). Hover your cursor over that link and you'll see the argument translated to the Unicode string.

It looks like it truncated the filename when creating a folder name with Unicode characters, but stores the correct folder name in the index file using percent encoding.

https://en.wikipedia.org/wiki/Percent-encoding

[tanstaafl]

Aside: This raises another question….. If the MSF is the source of the translation, then how is Thunderbirds folder to OS name translation is surviving deleting all the MSF files as outlined here http://kb.mozillazine.org/Compacting_fo ... em_to_work

If you delete the .msf file by pressing the repair button in the folder's properties in the folder pane its already loaded the contents of the that file, knows the filename, and normally doesn't lose anything when it recreates the index file. If you exit Thunderbird and delete the *.msf file using windows explorer its re-created based on the contents of the corresponding mbox file, which may not have everything. I noticed that the filename for a long Unicode folder name was truncated, and only the *.msf file had the full folder name. If I delete the *.msf file manually I suspect it will display a truncated Unicode folder name because it will be based on the percent encoded filename.

[thundergoesboom]

Good Day

Sorted my issue with outlook express import.

The name was being stored in the MSF file; but not in the same way as a test folder I'd manually created (was no "(85=" but the name was present further down in the msf file)

Initially the folder names were incorrect after shutting down Thunderbird, moving the folders From Import folder, then starting Thunderbird back up

But after physically clicking on the folder, and waiting a few seconds, the names from the MSF file were refreshed and showed as expected

Compacting all folders was enough to correct/refresh the folder names .

Thank you for the feed back

Sincerely