Inconsistent searching & indexing

[Kent Slimm]

The search feature built into TB is less than satisfactory, and I was disappointed to find that Windows 10 won't index and search TB message files. The good news is that a new version of a third-party program called Copernic Desktop Search finally does this -- and it does it very well. I have tens of thousands of emails in about two dozen folders and subfolders; CDS will instantly find a word that appears in any one of them.

The bad news is that one of my folders, which appears to be no different from the others, refuses to be indexed (and thus searched) for some reason. (This folder has one subfolder, which *can* be indexed. As can other folders that have both subfolders and their own messages.) There's nothing unusual about the messages in this folder when I'm reading or replying to them within Thunderbird -- or when I compare this folder to other folders in Windows File Explorer.

I have twice tried wiping out the CDS index and forcing it to reindex all my messages, but the same problem occurs with this particular folder. I've spent a great deal of time troubleshooting with a CDS tech rep in case it's an issue with their product, but it isn't; CDS works fine otherwise. I have to assume there's something I can't see that distinguishes this particular TB message folder. As an experiment, I tried copying a message that contained an unusual word into another folder and CDS immediately indexed it without a problem.

Is there any way to figure out what about this folder -- which has a perfectly common name, by the way; no unusual characters -- might be blocking the search engine?

Thanks in advance for suggestions...

[DanRaisch]

What version of Thunderbird?
How is the account set up in Thunderbird, as POP or IMAP? Find out using menu path Tools->Account Settings->Server Settings->Server Type at the top right of the dialogue.

[Kent Slimm]

v 78.2.1 -- and POP mail
But why would version number or server type affect otherwise interchangeable folders differentially, with indexing working for 19 of them and failing for 1?

[DanRaisch]

Knowing the version might let us determine if there are any known issue with that version that could be in play.
Knowing the server type tells us whether there is one or more types of folders involved. With an IMAP account, messages might be on the server rather than on the local computer as with a POP account.

It sounds as though that problem folder may be corrupted. First, backup your Profile ( http://kb.mozillazine.org/Profile_backup )

Create a new sub-folder under Local Folders. Move all of the messages currently in the problem folder to that new sub-folder.

Confirm that you have made the recommended backup of your Profile.

Close Thunderbird and navigate to the \Mail sub-folder of your Profile folder. Delete both files that have the name of the problem folder, one with the extension .msf and the second with no extension.

Restart Thunderbird and create a new folder with the same name as the problem one. Copy any messages you MUST have in that folder from the sub-folder created above to the new folder. Compact the folder after those messages are copied in.

The "Real Fix" entry in this article -- http://kb.mozillazine.org/Compacting_folders#Real_fix also describes this process.

[Kent Slimm]

Thanks for your detailed suggestion.

1. If, as you suggest, I *move* my emails from the problem folder to a new one, and then *copy* them back to the newly created folder with the same name as the problem folder, I'd end up with duplicates of all the emails, no? Or did you mean "move" rather than "copy"?

2. If I don't care about the name, is there any reason for that second step? Why not just create a new folder, move all the messages to it, and delete the old folder?

3. Is the theory here that the original folder isn't being indexed because the folder itself (apart from its contents) has somehow become corrupted -- and, specifically, corrupted in a way that doesn't show up in daily use but only when a program tries to index its contents?

[DanRaisch]

1. You would have two folders with the same messages, yes. But you would not have duplicate messages in a given folder. After everything is back to working properly, you can certainly delete that folder created in Local Folder as it was only a temporary holding area for your messages from the problem folder.

2. Deleting the files on the hard drive rather than from within Thunderbird ensures that the .msf file is also deleted so both the message storage folder and the message summary folder are created as new, un-corrupted files in the later steps.

3. That's one way to express it. Thunderbird message folders are stored as what are basically text files (with structured header data and coded data for any embedded images.) Unless you routinely read every messages in a folder you might not find the corruption but indexing does read every message and thus would encounter any corruption..

[Kent Slimm]

Clear enough on points 1 and 3, but maybe *I* wasn't clear with my question for #2. I wasn't asking why I should bother deleting the files on the hard drive, but why I should bother moving all the messages back to a (recreated) folder with the original name.

[DanRaisch]

That would be the process if you created the storage folder under Local Folders but wanted the final folder to be located in the POP account folder structure.

[Kent Slimm]

Huh. I wasn't aware of -- and even now that I look for it, can't find -- a "POP account folder structure." All my folders (except Inbox, Sent, Junk, and Trash) are under Local Folders.

Anyway, if I'm content to have the replacement folder in Local, just as the corrupted folder it's replacing was, is there some benefit to doing two moves, or can I leave the messages in the new replacement folder?

[DanRaisch]

You can leave them right in that new folder under Local Folders.

[Kent Slimm]

OK, I have done as you suggested and then forced the program to reindex my emails while I watched in real time. It indexed a few dozen of the messages out of more than 4500 in the new folder. I tried again the next day and it didn't include any more. That's a bit of progress, I suppose, but eliminating whatever was wrong with the old folder name still leaves ~99% of the emails in that folder -- and, as far as I can tell, ONLY in that folder -- unindexed and therefore unsearchable.

Any other ideas?

[DanRaisch]

How are you determining the status of the indexing process?
How many messages are in that folder and what is the size of the file on the hard drive that represents that folder?

[Kent Slimm]

Copernic Desktop Search allows users to monitor the indexing process; the names of the individual files scroll up in a window as they're indexed.
There are about 4500 messages in that folder. It takes up about 260 MB. That makes its size unremarkable (about average) compared to the roughly two dozen other email folders, all of which are indexed just fine.

[tanstaafl]

You said "a new version of a third-party program called Copernic Desktop Search finally does this". Are you talking about version 7.1.2? Are you using the commercial version (which has a free 30-day trial) or the lite version?

https://www.snapfiles.com/get/CopernicD ... earch.html is one way to download the lite version of 7.1.2. "The free Lite version has an index limit of 75,000 files and may display advertisements."

I've used an older version of Copernic Desktop Search with Thunderbird before. It had a 2GB limit on the maximum size of the search index that Copernic would create. I also found a 10 year old thread where somebody used CDS with Thunderbird. Why do you believe earlier versions didn't work?

One crude hack might be to install version 52.* in a different directory, copy the bad mbox file to Local Folders in the profile used by version 52, install the ImportExportTools add-on, use it to export all of the messages as .eml files, and then to import them into a new mail folder. That frequently cleans up problems with a folder, though you may lose a few messages that were corrupted. You can then copy the mbox file back to the profile used by version 78.

See http://kb.mozillazine.org/Go_back_to_an ... hunderbird for links to where you can get setup programs for older versions of Thunderbird.

[tanstaafl]

DocFetcher is a free desktop search engine based on lucene/solr that supports searching .eml files. The optional maildir storage format implemented In Thunderbird uses .eml files but doesn't have a file extension.

https://sourceforge.net/p/docfetcher/di ... /fc4ce158/
http://docfetcher.sourceforge.net/en/more.html

See "How can I index files that don't have a file extension?" section of http://docfetcher.sourceforge.net/wiki/doku.php?id=faq for a workaround.

I realize you want to get CDS to work well with Thunderbird but thought I'd mention a potential alternative if you get desperate.