'Proper' Emails still going to Junk

Discussion of general topics about Mozilla Thunderbird
rodhudson
Posts: 285
Joined: August 28th, 2007, 2:09 pm

'Proper' Emails still going to Junk

Post by rodhudson »

I am imagining that this is a 'Filtering' problem, as I have a filter that sends anything from anyone who is not in my Address Book to a folder called 'Commercial'.

From there I skip through them, and then normally delete.

But many times valid emails that ARE in my address Book not only go to COMMERCIAL, but often straight to JUNK. Which means I now have to check ALL incoming emails, which defeats the whole purpose of the Filter. Particularly as my own Email addresses are all DOT COM..... so I now get hundreds a day from all over the world. It's becoming unmanageable.

Any suggestions ? [Not rude !!]
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: 'Proper' Emails still going to Junk

Post by tanstaafl »

What is the exact rule that you are using?
Have you verified whatever address book that you are using isn't corrupted? Rare, but it happens. Export the contents as a .csv file and open it using Notepad. Any corruption should be easy to spot.
rodhudson
Posts: 285
Joined: August 28th, 2007, 2:09 pm

Re: 'Proper' Emails still going to Junk

Post by rodhudson »

Hi. Thanks for your reply.

Regarding the 'rule that I am using' ....... I am actually typing nothing. Purely selecting from one of the options offered.

So.......if you are saying that some of them are not good, surely they shouldn't be there.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: 'Proper' Emails still going to Junk

Post by tanstaafl »

I am asking exactly what did you select. For example, "From" "isn't in my address book" "Personal Address Book" with "Move Message to" "Commercial" as the action
rodhudson
Posts: 285
Joined: August 28th, 2007, 2:09 pm

Re: 'Proper' Emails still going to Junk

Post by rodhudson »

Yes. I do understand the question. But what is happening is that people who ARE in my Address book are going to Junk. So the fact that I may have selected , e.g ...' isn't in my address book' with the action to go to 'Commercial' .... is surely irrelevant.

Also, I should have added that this has worked fine for over a year.
Yet suddenly, without any change, NOT so.

So it's the inconsistency that is the main problem that is worrying me.
rodhudson
Posts: 285
Joined: August 28th, 2007, 2:09 pm

Re: 'Proper' Emails still going to Junk

Post by rodhudson »

Another ongoing problem for me is that several of my sent emails have not been received, which is even more worrying.

I'm thinking of running one or two other Email Client programs in tandem with T-Bird, just to see if all these 'glitches' are being replicated.

At least it should tell me something !
rodhudson
Posts: 285
Joined: August 28th, 2007, 2:09 pm

Re: 'Proper' Emails still going to Junk

Post by rodhudson »

Well.......it HAS told me something, as the identical thing is happening with 2 other Client programs, in terms of some being correctly received, and others going to JUNK, from the same Address.

I don't know about the 'Outgoing', but it's probably related.

So I suppose my next task is to take this all up with my ISP ? Which is IXWEBHOSTING in the US.

Whether there are other 'in-between' servers for them to blame, I'll find out !
rodhudson
Posts: 285
Joined: August 28th, 2007, 2:09 pm

Re: 'Proper' Emails still going to Junk

Post by rodhudson »

I haven't heard anything from anyone on this for over a week now, so maybe I'm wasting my time on here !

This is not a complaint. Just a statement of fact.

I know there is no charge, but I think I would rather pay for a well supported program.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: 'Proper' Emails still going to Junk

Post by tanstaafl »

You never answered my two questions. I got fed up and decided to let somebody else help you. Sometimes the questions asked by somebody who is trying to help you seems a waste of time but its usually a good investment of your time to answer them.

I repeat my suggestion that you export your address book and open it in notepad to verify some of it is not corrupted. I'd also enable the optional filter log and see if it has any useful information. It records what the message filters did and can be useful figuring out what they recognized.

"valid emails that ARE in my address Book not only go to COMMERCIAL, but often straight to JUNK. "
Are you saying it sometimes goes to junk instead of commercial, or that two copies of a message might occur?

What are the junk mail controls set to? i.e. is enable adaptive junk mail controls enabled, your address books white listed, and "move junk mail controls to" set to junk?

"The order of processing is:

Message filters
"Trust header"
"Adaptive junk" (junk mail controls)
"

Trust header is if you told the junk mail controls to trust what SpamAssassin or SpamPal flagged as junk. I assume you didn't configure that. The reason I mention the order of processing is if a message got moved to junk by the junk mail controls that implies that the message filters didn't recognize the sender as being in the address book.

"Whether there are other 'in-between' servers for them to blame, I'll find out !"
In-between servers rarely modify the message other than adding another Received: header (indicating who passed it along and when).

Are you using a POP account?

"it HAS told me something, as the identical thing is happening with 2 other Client programs, in terms of some being correctly received, and others going to JUNK, from the same Address. "

I suggest you provide details on exactly what the message filters are in Thunderbird, and the senders address for a couple of examples of both when it is was moved correctly and when it wasn't. You could replace the '@' with _AT_ to try to hide the email address from a spider. (spammer harvesting known good email addresses).

"The message filters for each account are stored in a "msgFilterRules.dat" file in the accounts "local directory". The local directory is specified near the bottom of each accounts server settings by the browse button." You could copy and paste the contents of that file rather than try to describe what the message filters were.
rodhudson
Posts: 285
Joined: August 28th, 2007, 2:09 pm

Re: 'Proper' Emails still going to Junk

Post by rodhudson »

Thanks for your reply, advice and suggestions. I will follow through on them, and post results, and what you have asked for.

My ISP has now got heavily involved, as I was inclined to be blaming them or their servers for a lot of this, as the same 'faults' were occurring on all the new Client Programs I was using to test, and they have NO filters set up. Yet emails were going to Inbox, or Junk seemingly randomly.

They have asked for 'headers' from examples etc.

A problem now is that I have made so many changes trying to put things right, that my set-up is nowhere the same as when started !
But as it is all still happening no matter what I do, that probably doesn't matter.

Anyway - to answer a couple of your more 'specific questions'........I DID open the Addresses using NOTEBOOK, [ as I have done before], and it all looks good.

Regarding the 'valid emails that ARE in my address book not only go to commercial but often straight to Junk'......that is EITHER OR.....not both.

AS I just said, I'm ruling out the filters for the moment, as the same thing is happening on other 'TEST' Clients that have NO filters set up.

Yes - I AM using POP Accounts, wanting to keep my own back-up copies here rather than on-line, [which is getting out of hand !]

I will try and find the 'msgFilterRules.dat' file and copy and paste it here. Although the 'Filters' may be a 'red herring' as it is occurring without any.

One further point. The Spam Assassin entry which IS enable, used to appear in the Filters box..... but now doesn't, even though still enable....and deleted and re-enabled.
Is that right ? Maybe just a program change.
rodhudson
Posts: 285
Joined: August 28th, 2007, 2:09 pm

Re: 'Proper' Emails still going to Junk

Post by rodhudson »

UPDATE: It appears that the problems seem to be cause by SPAM ASSASSIN marking all these emails as JUNK.
ISP is looking into it now.

Should I stop using it, and try another one ? If so, any recommendation ?
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: 'Proper' Emails still going to Junk

Post by tanstaafl »

SpamAssassin is the standard for spam detection. However, it takes skill/work to get it to work well. My main email provider (fastmail.fm) uses it and I regularly go over a year without any spam in my inbox. I've gone as long as 4 years without any spam. However, it took them over four months of fairly frequent tweaking before it became significantly more effective than a default installation. Perhaps your ISP wasn't willing to make the same investment.

SpamAssassin can be run locally (rather than by your email provider on the server) but its a large program and you wouldn't have access to the large amount of training data your email provider does. I wouldn't recommend that. If you want to try something locally take a look at SpamPal. It relies upon DNS blacklisting (a list of IP addresses of servers that are known to be heavily used by spammers, maintained by some group) and you can customize whose DNS blacklist you use. For example, Spamhaus maintains a good list. SpamPal is a pretty lightweight program. The big problem with SpamPal is not that its old and no longer maintained (it still works fine) its that nobody is promoting it, it has a dated user interface, and doesn't claim to have any fancy technology. So most potential users either never heard of it or turn their nose up at it.

I suggest you disable the junk mail controls in Thunderbird from trusting SpamAssassins headers. Instead, add a message filter that tests whether the spam score exceeds a certain value, and only if it does, then set the message as junk. In my case I'd test X-Spam-score. The custom header that your ISP uses may be different. Use Control-U or view -> message source and you should be able to identify it. I suggest you look at the spam score of some of the messages that were mis-categorized to figure out what threshold to use. You probably want a value between 5 and 10.

I suggest you also add many of the email addresses from your Thunderbird address book as contacts in webmail. Hopefully SpamAssassin will lower the spam score of any message sent from any of those contacts, making it far less likely that they will be mis-categorized as spam.

Another solution would be to sign up for a free Gmail webmail account, configure it to automatically fetch mail from your ISPs POP server and store it in your Gmail inbox, and use a Gmail POP account instead of your existing one. I could explain how to do that in way that nobody would know that you are actually using Gmail (you'd still get the same email and could send using the ISPs From: address). Gmail doesn't use SpamAssassin but their spam detection is extremely good, they spent about $50 million buying several companies just to use their anti-spam technology. Let me know if you want to do that.

Lots of people prefer to use their ISP as their email provider. IMHO they are a poor email provider as they view email as just a cost center, and if you change ISPs you need to change your email address. Gmail seems to be the best of the free email providers. Outlook.com is good but they have periodic outages. Yahoo and AOL are poor. GMX used to be good but now seems poor. Zoho seems okay but is nothing special. That is just my personal opinion. Take it with a grain of salt.
rodhudson
Posts: 285
Joined: August 28th, 2007, 2:09 pm

Re: 'Proper' Emails still going to Junk

Post by rodhudson »

WOW. Thanks so much. Lots to think about and look into there ! I do already have a Yahoo and a GMAIL account, seldom used. But with different user names, of course.
I don't really want to change ISP's [ although I am threatening them to].... as I have 4 web-sites set up through one of their providers, which would be a mammoth task to re-create.

Still. Well see.

I will look into all you have said, which may take a while.... and report back.
Thanks again.
rodhudson
Posts: 285
Joined: August 28th, 2007, 2:09 pm

Re: 'Proper' Emails still going to Junk

Post by rodhudson »

As suggested, I AM taking the notes and suggestions ‘with a grain of salt’ …… but even a grain is of immense use if you don’t have a salt cellar.

I HAVE disabled the SPAM ASSASSIN in the Junk controls, but gather that the ISP also uses it there before dispatch.

You mentioned the ‘large amount of training data that an ISP may have’ …… but in this case, I think that is part of the problem.

I have been emailing with them about it, and apparently many emails both being sent to me, and that I am sending out, are being marked as Spam because words are printed in capitals, or because there are too many spaces !

I find this ridiculous. I am a scriptwriter, and I use Capitals and spaces on purpose, and for a reason, not to be a reason to be classed as ‘SPAM’.

I was also told that an email sent from my daughter’s Yahoo Account, which she has used to sent me email for many years, was marked as Spam because it only contained a photo of my 2 grandsons at the Christmas tree, with very little text….. and this is ‘NOT STANDARD PRACTICE’ !!

I find this ludicrous, and have told them so.

I don’t think changing my ISP as an email provider should affect my Addresses, as they are all my own DOT COMs. Which IS, of course, part of the problem in terms of a large number of emails being received from Advertisers etc.

But I think I could change the email handler without changing the ISP, so that I wouldn’t have to re-do the Web-sites.

Are there companies who just do Email handling ? At the moment I am on a ‘shared server’. Apparently if I change to a ‘dedicated server’ all these problems could be overcome….. but at a cost of around $70 a month ! A bit much, I thought. $70 a year I’d do it !

So…… it seems I want [or need] to keep the same ISP, and Thunderbird, but change my Email Handler.

Is there a way to do this ? I don’t really want to pay for 2 ISP’s !!

I will look into changing the Spam control program, but thought I’d ask about the above first.
User avatar
tanstaafl
Moderator
Posts: 49647
Joined: July 30th, 2003, 5:06 pm

Re: 'Proper' Emails still going to Junk

Post by tanstaafl »

See http://kb.mozillazine.org/Junk_Mail_Con ... s_you_sent for how to check the spam score of messages you are going to send/sent. You might want to check some of your messages against the spam score your ISP calculates. It sounds like they have some poor rules.

Some people use web hosts from DreamHost, GoDaddy, HostGator, InMotionHosting, 1&1 etc. as their email provider. Personally, unless you need to host your own web site I think its much better to use a company that is just a email provider. Most commercial email providers let you use your own domain (you pick and deal with the domain registrar, such as namecheap or gandi) and may offer the option to let them register and manage the domain for you instead. You would not be paying for a second ISP.

I pay a annual subscription to fastmail.fm, which I've used for over a decade. See https://www.fastmail.com/pricing/ for their current plans (I'm on a older legacy plan). You'd probably want the standard plan at $50/year. You can use your own domain with them. Their support is good, but it typically takes a day for them to initially respond. There is also the unofficial support forums at http://www.emaildiscussions.com/forumdi ... forumid=17 . I've found them very helpful. My only connection with fastmail is that I'm a happy customer.

Polarismail at https://www.polarismail.com/ and Runbox at https://runbox.com/price-plans/ are some other inexpensive choices. Luxsci at https://luxsci.com/ (well known for their customer support) and G Suite at https://gsuite.google.com/ (commercial version of Gmail) primarily target business users.

Zoho workplace and Microsoft Office365 provide business email, but bundle other services.
Post Reply