Can't search messages with Copernic. Rebuild TB profile?

User Help for Mozilla Thunderbird
Kent Slimm
Posts: 137
Joined: December 4th, 2008, 6:08 pm

Can't search messages with Copernic. Rebuild TB profile?

Post by Kent Slimm »

For the last several months, Copernic Desktop Search has been unable to index and search the last few months' of TB emails. (It has no problem with my Word files.) When I do a search, those relatively recent messages are labeled with the day they were indexed, not their actual date; and when they're opened, I see only pages of formatting ... from a different message.

I've tried two different releases of the software (CDS 4 and 5), both free and paid versions, with two different computers (one using Windows 7, the other Windows 8). I've reinstalled CDS several times and reindexed all my emails from scratch. Nothing works. CDS is supposed to work with TB; their tech support says the problem is with my message files. (I should mention that most of my 70,000+ archived messages were originally imported from Outlook Express. But it's the more recent ones, created by TB, that seem to be the problem.)

I've never had multiple TB profiles and therefore have never needed to deal with profiles at all. My question is whether it's worth the time and possible risk to repair or completely rebuild my profile when I don't know what I'm doing. All I know about that process is what I read here: http://kb.mozillazine.org/Transferring_ ... hunderbird.

So *should* I create a new profile and then try to migrate all my messages? Do I have any other options? If not, is there anything I should know about how to do this -- and what to watch out for -- beyond what's on that wiki page?

Thanks.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by tanstaafl »

I would expect Copernic to ignore all of the files in the profile except for the mbox files. Those are the files named after the mail folder, with no file extension, that contain all of the messages for that folder. If thats true, I don't see any benefit in building new profiles. I read their help text about Thunderbird and it seems like a copout.

"inbox." is the mbox file for the inbox folder. "inbox.msf" is basically a cache of the folder listing and "inbox.sbd" a renamed directory used to organize the mail folders into a hierarchy.

One possibility is that many of those mbox files are corrupt, but that normally only occurs for inbox folders if you let them have too many deleted messages and don't compact them often enough.

As an experiment, copy all of your mbox files to a unrelated directory (you don't need to keep the folder hierarchy), tell CDS to rebuild its index, and see if it can read the messages in those files. I suspect it just knows how to read mbox files, and doesn't know about or understand the other files in the profile such as prefs.js so it should be content reading those mbox files despite them not being in a profile.

I'll download the free copy onto my Windows 10 system and see if it has a problem with my profile.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by tanstaafl »

I installed the free version of CDS version 5 but when I try to run it nothing happens. I used task manager to check that it did not create a new process (that I could not see) or add a windows service (that didn't start for some reason). I did this using both the shortcut it created and by double clicking on the executable in the installation directory. I even tried configuring the shortcut to run in Windows 7 compatibility mode.

I uninstalled it.

You might try Exalead desktop search. It used to be a commercial product, now its free/open source software. I found a review article about it that mentioned that it supported Thunderbird.
Kent Slimm
Posts: 137
Joined: December 4th, 2008, 6:08 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by Kent Slimm »

Thanks for your quick and helpful response. I've spent very little time looking under TB's hood, so I hope you'll be patient with me.

First, an observation: My problem began at about the same time on both of my computers (Windows 8 desktop + Windows 7 laptop). If TB messages and profiles are stored locally rather than in the cloud, doesn't that suggest the problem is with CDS? I mean, how likely is it that my TB files on both computers would have been corrupted simultaneously?

Second, my inbox contains few or no messages; they're promptly moved to one of about a dozen archive folders.

Third, I don't know what an mbox file is. My message folders live at \Mail\Local Folders\Outlook Express Import.sbd. They all have the suffix ".mozmsgs" except that some of the folder names seem to have duplicates, and the alternate folders end with ".sbd." (Hmmm. Could this duplication be the source of the problem?) No files, at least in this folder, end with ".mbox."

Finally, in trying to diagnose this issue, I should mention that I didn't move, rename, or otherwise modify these TB files and folders, yet CDS worked fine until a couple of months ago.

In light of all this, I'm curious about what you think I should do. And if you have a specific suggestion, step-by-step directions for how to do it would be greatly appreciated.
Kent Slimm
Posts: 137
Joined: December 4th, 2008, 6:08 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by Kent Slimm »

Just saw your second message. I will look into Exalead, which I hadn't heard of. Most desktop search programs only search *for* files, not *within* files, so I'll be curious to see whether this one does the latter. Also, a quick web search for the program turns up nothing written about it in the last 5+ years, so I wonder whether it's still being updated.

Anyway, I'm not ready to give up on CDS yet. It's weird you weren't able to run it at all; I find that it installs easily and immediately starts indexing files. I hope you can still offer advice even if you can't get it to run on your computer.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by tanstaafl »

"Third, I don't know what an mbox file is. My message folders live at \Mail\Local Folders\Outlook Express Import.sbd. They all have the suffix ".mozmsgs" except that some of the folder names seem to have duplicates, and the alternate folders end with ".sbd." (Hmmm. Could this duplication be the source of the problem?) No files, at least in this folder, end with ".mbox.""

.mozmsgs contains partial copies of messages (upto 20KB) stored as .eml files (though saved as *.wdseml files) by windows search indexing. That is enabled by tools -> options -> advanced -> general -> "allow windows search to search messages". Under OS X there is a similar feature for Spotlight integration, using *.mozeml files in a .mozmsgs subdirectory. I haven't tested windows search integration for years so some details may have changed but what hasn't changed is that Thunderbird stores messages in mbox files. They are flat plain text files named after the folder, with no file extension, that contain all of the messages for that mail folder. You can open and read them with a text editor, though its not user friendly. They have no file extension, they do NOT use a .mbox file extension.

There is also an option to store messages individually using the maildir lite format, but there is no support in the GUI for that, and AFAIK they don't have a .mozmsgs file extension.

http://kb.mozillazine.org/Files_and_fol ... hunderbird

"Just saw your second message. I will look into Exalead, which I hadn't heard of. Most desktop search programs only search *for* files, not *within* files, so I'll be curious to see whether this one does the latter. Also, a quick web search for the program turns up nothing written about it in the last 5+ years, so I wonder whether it's still being updated."

I don't think its being maintained by anybody. Desktop search programs (that understand the format of various files) seem to have fallen out of favor, though there are plenty of programs to quickly find files. I use Ultrasearch for example. "UltraSearch does not maintain an index which is stored on your hard disk. UltraSearch achieves its speed by working directly on the Master File Table (MFT) of the NTFS partitions. UltraSearch even identifies NTFS hardlinks. "
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by tanstaafl »

"If TB messages and profiles are stored locally rather than in the cloud, doesn't that suggest the problem is with CDS? I mean, how likely is it that my TB files on both computers would have been corrupted simultaneously?"

The corruption is usually a side effect of bad habits. I'd expect you to have the same habits on both machines. You could go back to an older version of Thunderbird (the problem only showed up a couple of months ago, so it might have been triggered by an update) and migrate your address books and mail folder to a new profile (as Copernic wants you to). But it doesn't make sense if CDS only pays attention to the mbox files. Their format hasn't changed AFAIK other than adding a X-Mozilla-Keys: header to support tags in version 2. I used to have several versions of Thunderbird installed in different directories (2.0.0.24, 3.1.20, 17.* and whatever the latest version was) and had no problems with the mail folders as I switched which version I used with the profile.

http://kb.mozillazine.org/Go_back_to_an ... hunderbird
Kent Slimm
Posts: 137
Joined: December 4th, 2008, 6:08 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by Kent Slimm »

Thanks for your latest.

1. While I appreciate your explanation about mboxfiles and .mozmsgs vs. .sbd, I'm still left wondering what my best move is here. Is the duplication possibly confusing CDS? Should I therefore delete one of the categories of folders, and is it completely safe to do so? If not, what *can* I do? (I can't selectively tell CDS to index only .mozmsgs rather than .sbd -- or vice versa -- because the CDS interface doesn't show me both versions of a given folder.)

2. As a Plan B, I downloaded and installed Exalead. Thanks for that suggestion. It indexed 60,000+ files in less than an hour. Unfortunately it failed to index three key folders, and I can't figure out why. Unlike CDS, it does show .mozmsgs and .sbd folders and allows me to check or uncheck each version separately, but no permutation for each of the three overlooked folders (.mozmsgs version only, .sbd version only, or both) forces the program to index it. There's no tech support for Exalead, and rather than spend more hours on this, I'd prefer just to figure out the problem with CDS.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by tanstaafl »

Try disabling windows search integration in Thunderbird and then moving .mozmsgs to removable media (such as a usb/flash drive) that Copernic won't see. Then have Copernic update its index. That will let you quickly if the duplication is causing problems, and is easily reversible.

"(I can't selectively tell CDS to index only .mozmsgs rather than .sbd -- or vice versa -- because the CDS interface doesn't show me both versions of a given folder.)"
http://support.copernic.com/en/configur ... erdossiers talks about excluding certain folders using tools -> options -> "Emails & Organizer"

The CDS web page now claims it supports Windows 10. I'll download it and try it again.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by tanstaafl »

It works. I'm using Copernic Desktop Search Lite 5.1.0 build 8331. I don't have windows search integration enabled, and have no .mozmsgs/*.wdseml files.

I got a security warning from windows firewall about it disabling some features of CDS and unchecked its ability to use private networks (public networks was already disabled) and told it to continue. I've let it index enough that I could search for certain topics and read the search results without any problems. I used process explorer to figure out that if I double click on a search result it creates a .eml temporary file and then passes its path as a command line argument to Thunderbird, to view a message. For example: "C:\Program Files (x86)\Mozilla Thunderbird\thunderbird.exe" "C:\Users\Eric\AppData\Local\Copernic\DesktopSearch\Temp\tmp-108508-0164.eml" . Otherwise I can read it in the scrollable bottom window. Some of the messages it found have 2MB embedded images which were displayed correctly in CDS. The same messages were displayed without the image if I double clicked on them. Thats a known limitation of Thunderbird's .eml support.

Task manager shows a Copernic Desktop Search (32-bit) process that runs in the background when I close the CDS window. I don't see any sign of it adding a windows service.
Kent Slimm
Posts: 137
Joined: December 4th, 2008, 6:08 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by Kent Slimm »

Please let me know if, while noodling around with CDS, you discover anything that might help me solve my problem.

Now that I look more carefully at the TB message folders, I retract my duplication theory. Where a folder (e.g., "Education") has subfolders, the collection of subfolders is called "Education.sbd." But each of the individual subfolders, like all the folders without subfolders, ends with ".mozmsgs." To delete (or temporarily remove) .mozmsgs folders would be to get rid of all my messages.

I tried deselecting TB folders from Windows search, but I'm not sure that's likely to make a difference so it would make sense to have CDS build a new index from scratch.

What next?
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by tanstaafl »

Its getting annoying that you don't believe me about Thunderbird storing messages in mbox files. Open "inbox." with a text editor, look at the first message and compare it to what you see using view -> message source in Thunderbird. Or spend some time playing with the ImportExportTools add-on.

I enabled windows search integration and let CDS update its index. My profile now has some additional directories whose name ends with .mozmsgs , which contain *.wdseml files. For example one of my IMAP accounts has a digests mail folder. The profile used to have just a "digests." (mbox file) and "digests.msf" (cache of folder listing) file for that mail folder since it doesn't have any child folders. Those two files still exist, but after enabling windows search integration it added a digests.mozmsgs directory that has *.wdseml files.

CDS doesn't seem to have a problem with having the same message in both the mbox file and a *.wdseml file. One of the messages I tested was 374KB. I scrolled to the bottom of the results and read text that I verified was not in the corresponding 49KB *.wseml file for that message. I'm surprised the *.wdseml is so large, I thought it was limited to about 20KB.

I noticed that CDS will return results from other files in the TB profile such as prefs.js or bookmarks.html. I don't believe it understands what prefs.js does it appears to treat it as just another document in a email folder. It also understands .docx and .pdf attachments in messages and treats them as separate documents.

I don't understand why you are running into the problems you are. Perhaps its time to follow Copernic's advice and create a new profile, and then transfer just your address books and mail folders (mbox files) to it. I suggest you don't enable windows search integration in your new profile, since it doesn't seem to add any value if you use CDS to search .
Kent Slimm
Posts: 137
Joined: December 4th, 2008, 6:08 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by Kent Slimm »

I appreciate your continued patience in helping me sort through this. I'm not sure I understand your first comment, though, because I didn't say anything about mbox files in my last message.

Is there anything in your first four paragraphs that supports a recommendation for something I can do to make it more likely that CDS will again be able to index and search my recent messages accurately?

As for your last paragraph, in which you suggest I create a new profile:

1) I return to the question in my original post: Is there anything I should know about how to do this -- and what to watch out for -- beyond what's on that wiki page? Please keep in mind that I'm not as familiar with these procedures as whoever wrote those instructions seems to assume.

2) You suggest that I transfer just my address books and mail folders to a new profile.
-- Do you mean temporarily or permanently?
-- What would I *not* be transferring?
-- What's the reason for transferring only those things?

Thanks!
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by tanstaafl »

I'm suggesting that if the new profile doesn't have problems with CDS, that you keep using it instead of your current profile. You would *only* transfer your address book and mail folders to it. You'd have to add your accounts again, tweak your settings the way you like them, save your passwords again, install any add-ons again etc. The basic idea is if you can't figure out what causes a serious problem, at some point your best option is to start over keeping only the most essential information and re-create everything else, rather than keep banging your head against the wall.

See http://kb.mozillazine.org/Profile_manager for how to create a new profile, and to switch between profiles as needed.
See http://kb.mozillazine.org/Importing_folders for how to import your mail folders into the new profile
See http://kb.mozillazine.org/Moving_addres ... n_profiles for how to import your address books to the new profile.
Kent Slimm
Posts: 137
Joined: December 4th, 2008, 6:08 pm

Re: Can't search messages with Copernic. Rebuild TB profile?

Post by Kent Slimm »

OK, thanks. I'll try that.
Post Reply