IMAP not syncing all messages

[Derfram]

Background:
First, when I refer to "New messages on the server", I am referring to new messages I can see on the ISP's webmail page. I realize this is not 'the server', but if I can see new messages on the webmail page I assume these messages exist as new messages on the server.

I have the last version of TB. I have tried (on different machines) a couple older versions of TB all the way back to 38.5 and have seen the same symptoms in all. My original ISP was Time Warner Cable, which of course does not exist anymore, but the accounts have all been taken over (in this area) by 'Spectrum'. With an @wi.rr.com email address, the only active Spectrum email server I can log into is still mail.twc.com. When setting up TB, it wants to auto-configure to mobile.spectrum.net mobile.charter.net, which is the same mail server that spectrum recommends, but with the legacy rr.com email account, this server does not recognize me. So I continue to use the server I have used for more than 10 years, mail.twc.com.

This issue surfaced when I bought this new Windows 10 machine to replace my -expiring- Windows 7 machine. So new machine, new copy of TB. I had been using POP3 on the Win7 machine, and had not had any problems. Set up the new machine for IMAP and have had the issue since. (I have disabled TB/POP3 on the Win7 machine, so it is not messing with the new messages)

Current troubleshooting:
I have created the logging .bat file (imap.bat) in my profile folder. I disabled all accounts in TB except my primary email account (one of the rr.com accounts). Last night, I made sure I had read all the new messages as seen on the webmail page, (so there were no existing new messages) and started TB via the .bat. The .log file appeared of zero size.

This morning, there are 10 new messages showing on the webmail page. TB has pulled 9 of them (this is good, usually it leaves more new messages on the server). Log file now shows a size of 850 KB.

What is my next step to get the log analyzed?

Update: There are now 16 new messages shown on the webmail page, but Thunderbird only lists 12. The imap.log file has climbed to 3724 KB.

[tanstaafl]

The new account wizard attempts to find the recommended settings for your email provider from the Mozilla ISP database. Most of the time if it finds something its the right settings. But not always. And you can override the choices and specify what server, port etc. you want by using the advanced and/or manual edit button. You just need to figure out what settings to use if you do that.

Are you ddeerrff in https://forums.timewarnercable.com/t5/E ... m-p/165359 ?

You said you have a Gmail, Yahoo and an Outlook account that work fine. Have you considered going to gmail webmail with a browser and configuring it to automatically fetch mail from the spectrum POP server and merge it in the inbox with the gmail messages? If you add a "multiple identity" in the Thunderbird gmail account you could both send and receive email's using the Spectrum email address. It would automatically select the right address when replying. When sending, you'd select the desired From: address from the From: list box. If you did that you wouldn't use the spectrum account in Thunderbird anymore, thus avoiding any interoperability issues.

See http://kb.mozillazine.org/Replace_sever ... ne_account for more details. You could stick your toe in the water and just configure the gmail webmail account to automatically fetch mail from spectrum and see how well that works. If it works fine then think about making the changes I suggested in Thunderbird.

"How do I attach a log file on this message board?"
You could upload it to something like dropbox, Microsoft onedrive or Google Drive, make it world readable, and post a link to it. You could password protect the link . I doubt any spiders that crawl web pages looking for email addresses would be smart enough to use that link and password and figure out how to parse a log file for a email address. It hasn't been a problem in the past AFAIK.

If you want to post it in this thread I suggest you use a text editor (NOT a word processor) to do a global replacement of your email address with a generic string like username@myisp and then copy&paste it into a new post. Save the original log file though as I suspect you will eventually need to file a bug report.

A 3274KB log file is too large (especially to post in this forum). I suggest you create a new one, reproduce the problem, and then stop logging. All you need to do is exit Thunderbird and not use the batch file again to stop logging. Don't forget that you need to mention what messages it should have gotten, and what messages it actually got, so that we can try to figure out why certain messages were ignored.

[Derfram]

The new account wizard attempts to find the recommended settings ...you can override the choices and specify what server, port etc.

Yes, that is what I have done.

Are you ddeerrff in https://forums.timewarnercable.com/t5/E ... m-p/165359 ?

yes, but my original thread in that forum seems to have been merged into another thread.

You said you have a Gmail, Yahoo and an Outlook account that work fine. Have you considered going to gmail webmail with a browser and configuring it to automatically fetch mail from the spectrum POP server and merge it in the inbox with the gmail messages?

Thanks for that suggestion but I really shouldn't have to do that.

"How do I attach a log file on this message board?"
You could upload it to something like dropbox, Microsoft onedrive or Google Drive,...If you want to post it in this thread I suggest you use a text editor (NOT a word processor) to do a global replacement of your email address with a generic string like username@myisp and then copy&paste it into a new post......
A 3274KB log file is too large ...... I suggest you create a new one, reproduce the problem, and then stop logging.

I'll give that a try. Haven't used dropbox, Microsoft onedrive or Google Drive, maybe it's time I learned how to. What are the size limits to posting a log file as text here?

[tanstaafl]

The limits for the cloud drives should far exceed any log file you upload. Most log files that I create are well under 1MB.

Dropbox for example has a 50GB maximum file size limit. In practice its limited to the maximum size of your storage space. The free version of Dropbox (Dropbox Basic) has a 2GB storage space. The maximum file size in Microsoft OneDrive depends upon a lot of factors such as how you upload it. In general though I'd you can at least upload a 15GB file. Google Drive is unusual in that most file size limits are based on the file type. It supports a 1.02 million character document. But you can also upload a 5TB file (if you have enough storage quota).

There is no specified limit for the maximum size you can post in this thread. However its usually best to post excerpts of a log file in a thread. That works well when you are logging a specific error, and you also need to see some of the traffic before and afterwards to get some context, but don't need most of the entire log file. In your case the problem is more distributed, so you can't post an excerpt. I suggest you post a link to a file you uploaded to a cloud drive. If I find something useful I'll post some excerpts.

[Derfram]

Where's the spam when you need it . After a couple of hours, one new message appeared on the webmail page, but did not appear on TB. The message timestamped at 7:03 pm on the webmail page actually showed up at 9:27 pm (03:27 UTC). The log appears to show a 'syntax error', and then perhaps some issues with logout/login. Anyway, if I did this right, you should be able to find it at https://1drv.ms/u/s!AoGTjfDYtTb8gjSVG5l ... b?e=aCzBSD .

Thanks again for your efforts and patience.

Got a second new message and this one *did* show up on TB. Pictures and imap2.log added. Timestamp in the log at 05:03 UTC.

[tanstaafl]

I only took a quick look at what you uploaded, I'll take a real look tonight. However, the internal syntax errors seems to be just for the deleted items, junk mail, and sent items folders so it shouldn't effect your problem. I also found a old bug report - https://bugzilla.mozilla.org/show_bug.cgi?id=1258429 - that mentions :
"> The comment from Gene Smith that Charter has a defective IMAP server is
> confusing, since my Android email, AquaMail, has no problem requesting and
> receiving new emails on demand from Charter's IMAP server.

Yes, some email clients and apps work OK with charter because they do a redundant SELECT command on INBOX before they attempt to FETCH new email. I have observed at least one email client doing this (kmail) while others I have observed don't detect new email and do exactly like thunderbird (evolution and claws-mail)."

I see no sign of redundant SELECT commands (that's used to specify the folder you are working on) in the log files.

[Derfram]

Reading over the linked bug report, I think you hit the nail on the head. In the bug report (and reports linked in it) there seems to be some discussions of work-arounds, but no final solution. And was this bug 'resolved' or not?

[tanstaafl]

It was supposedly fixed by patches for another bug report - https://bugzilla.mozilla.org/show_bug.cgi?id=1231592 in version 55. The fix requires you to set the maximum connection limit to 1 in the accounts server settings -> advanced per https://bugzilla.mozilla.org/show_bug.c ... 231592#c16 but they state its not a reliable fix. The other problem is that https://bugzilla.mozilla.org/show_bug.c ... 31592#c148 states its being delayed for TB 59 ESR (despite Status: RESOLVED FIXED and Milestone: Thunderbird 55.0). There is a tracking flag saying it was fixed in 57+ but its not clear that means the fix was actually released.

Several comments later there is "mobile.charter.net is never normal IMAP server. Charter net utilizes some IMAP commands but it's never RFC 3501 complient IMAP server."

They're using a Intermail IMAP server, which is supposedly no longer supported. It was originally a software.com product, and the company merged with another to form OpenWave. There were two versions of the product. One was killed off. The other was renamed to be Openwave Email MX. The company changed its name to Unwired Planet and the product was split off into another company - Openwave Messaging. That was acquired by Synchronoss Technologies in 2016 and the company started trading under Synchronoss Messaging. The email server software they sell is Synchronoss Email Suite, which includes wireless and RCS, so I'd assume that is not a later version of Intermail. I lost track of exactly when the Intermail product was no longer supported but according to one bug report four years ago your ISP was using version 9.0 of the software, and your log file shows they're currently using 9.0.1.

Some FETCH commands are obviously not working, and there is no sign of multiple SELECT commands being used . This is probably because you're not using the workaround configuration. https://bugzilla.mozilla.org/show_bug.cgi?id=1359277 summarizes the suggested changes:

The workaround is to set these preferences:
mail.server.default.use_idle = false
mail.server.serverX.max_cached_connections = 1
mail.imap.expunge_after_delete = true

To do that uncheck "Allow immediate server notifications..." in Server Settings, set maximum connections to cache to 1 in server settings -> advanced, and set mail.imap.expunge_after_delete true using the config editor.

""Move it to the Trash folder" is very similar to deleting a message in a POP account. The main difference is that if you set mail.imap.expunge_after_delete to true using the Config editor Thunderbird will compact the folder after it successfully copies the message to the trash folder. [1] The same thing occurs if you move a message to another folder since moving a message deletes a message and then copies it."

I recommend you either use a different email provider or replace the IMAP account with a POP account. POP servers are much simpler, so it might work well. Your ISP is using a unsupported IMAP server that was known to do a bad job of implementing the RFC's, and the Thunderbird developers have little motivation to try to provide a better workaround since they estimate only 0.1% of Thunderbird users are effected and they have already tried (unsuccessfully) several times to contact Charter about the problem.

[Derfram]

Thanks tanstaafl.

After reading and trying to comprehend all the interconnected links in the bug reports, I did the following earlier today:

Server settings Advanced... Changed "Maximum number of server connections" from 5 to 1 for all 3 TWC accounts.
"Allow immediate server notification when new messages arrive" unchecked for all 3 TWC accounts.

In config:
Changed mail.server.default.force_select from "auto" to "true"
Changed mail.server.server1.force_select from "no-auto" to "true"
Changed mail.server.server3.force_select from "no-auto" to "true"
Changed mail.server.server4.force_select from "no-auto" to "true"

After reading your response I also:

Changed mail.server.default.use_idle from "true" to "false"
Changed mail.imap.expunge_after_delete from "false" to "true"

Are my changes in regard to 'Force_select' reasonable (1, 3, and 4 are all TWC accounts)?
I did note in the config that mail.imap.force_select_detect seems to know we are dealing with Email MX/Openwave messaging.

What about the check box for "Clean up ("Expunge") inbox on exit" under Server Settings? Is that the same as the "mail.imap.expunge_after_delete" config entry?

[tanstaafl]

https://bugzilla.mozilla.org/show_bug.cgi?id=1360117

I would have left mail.server.default.force_select alone since you don't probably want to enable forceSelect for a server that doesn't need it, and by setting it as a default any new account would inherit it.

mail.imap.expunge_after_delete does a expunge whenever you delete a message. So there is no need for "Clean up ("Expunge") inbox on exit" , which only effects the inbox when you exit Thunderbird.

[Derfram]

OK, reverted the default.force_select to auto. Understand on the expunge. I'll have to see how it work, I may just end up going back to POP3.

Spectrum is currently specifying 'mobile.charter.net' as the mail servers, but with my legacy RoadRunner/TWC account, that server does not recognize me. I have to use mail.twc.com. I'm fearing that Charter/Spectrum is abandoning the legacy accounts as far as support and that sooner or later I'm going to lose my @xx.rr.com address. Like losing a long held phone number - there are sooo many people and businesses that need to be notified of an email address change.

Appreciate all the help and research.

[Derfram]

A quick revisit to this issue... Anyone know if Spectrum has updated their IMAP server to be RFC-3501 compatible or if Thunderbird now sends that 'redundant Fetch' command? Currently using POP3.. Don't want to waste my time reconfiguring for an IMAP account if I am just going to have the same problem I had a year ago. Thanks!