Test Win32 Build Available with Junk Mail Filter Changes

djbrock · Post by **djbrock** » February 22nd, 2004, 7:50 pm

I'm hitting a bit better than 50% with the Adaptive filter setting at 10. It's caught 6 Spam and missed 5 in the last day.

peaveyman · Post by **peaveyman** » February 22nd, 2004, 9:04 pm

Since I did the retraining of the filter, it has been running almost at 100 %. I have over a hundred junk emails and it has caught all but 1 or 2 of them. Not bad, I think.

Kylotan · Post by **Kylotan** » February 23rd, 2004, 4:01 am

Whereas I just got another 4 spams that Thunderbird totally missed and Popfile picked up. How can people get such totally different results?

Moonwolf · Post by **Moonwolf** » February 23rd, 2004, 4:57 am

Yesterday it was working at around 95% for me. Then I marked a false positive as not junk. This morning it missed 95% of my spam. Still some work to do I think.

Kylotan · Post by **Kylotan** » February 23rd, 2004, 5:07 am

Is the code for this build available anywhere?

Moonwolf · Post by **Moonwolf** » February 23rd, 2004, 5:10 am

Yes, it has been checked into the trunk.

Kylotan · Post by **Kylotan** » February 23rd, 2004, 5:13 am

How would I see the file(s) in question without downloading 36MB of source code?

peaveyman · Post by **peaveyman** » February 23rd, 2004, 6:11 am

And now, this morning, there were 11 spam that came in and it missed 4 of them. Was working fine yesterday.

Kylotan · Post by **Kylotan** » February 23rd, 2004, 6:39 am

Just as a hint to mscott, POPFile counts html tags as special tokens, and invalid html tags as another token type. Seeing 18 instances of 'html:invalid tag' in a single email is a great spam indicator!

Kerrick · Post by **Kerrick** » February 23rd, 2004, 10:00 am

Scott:

I notice in one particualr mailbox i get lots and lots of spam that TB steadfastly refuses to catch. Would getting your hands on a few hundred (or thousand, wouldnt take long) pieces of it help you at all in analyzing why the filter wont catch it? I'd be more than pleased to post it or something if it would help your analysis

mgl · Post by **mgl** » February 23rd, 2004, 10:43 am

Kylotan wrote:Whereas I just got another 4 spams that Thunderbird totally missed and Popfile picked up. How can people get such totally different results?

This is a classic issue with TB's junk mail filter, as detailed in several threads in the TB forums. Some people get POPFile-quality filtering from TB with very little effort, while others (like you, and me when I had this special build installed) consistently fail to achieve even a mild level of success with it. I suspect part of the answer has to do with the filter being very sensitive to the training method used--do it even slightly wrong, and you're all messed up.

For instance, I thought that training TB on my collected Junk Mail corpus of about 4,000 messages would be sure to help it learn what spam is. After all, the corpus contains just about every permutation of "Viagra" and "mortgage" you can imagine, so what better way to teach the filter than to provide with thousands of messages which are all unambiguously junk? Turns out it doesn't work that way, for some reason: it still failed to pick up even the most obvious incoming spam.

I deleted training.dat several times--I tried different strategies, marking all my good messages as Not Junk, then I tried not doing that, then I tried marking only false positives. I tried these things without training the new build on the large corpus, but only on incoming spam. None of it helped--the filter was still failing to pick up most of the incoming junk.

So I went back to the nightly, restored my old training.dat, and I'm back to 85-90% performance--good enough for me, but not competitive with POPFile etc. This thing is a mystery to me.

mscott · Post by **mscott** » February 23rd, 2004, 10:56 am

btw, none of this is checked into the trunk. It's an experiment.

Also, there were some bugs in the new algorithm that someone is fixing for me.

Kylotan · Post by **Kylotan** » February 23rd, 2004, 10:59 am

Yeah. As I understand it, a Bayesian classifier trained with 4000 junk mails should easily catch every junk mail you throw at it, except for utterly novel ones. The problem then would be false positives, but you'd quickly fix that by correcting the mistakes. The probabilistic nature of it should mean that only about 10 fixed false positives should be enough to correct this.

It's also interesting that we're encouraged to train the filter on previous mails, when in fact these systems tend to work better when only trained on errors.

Kylotan · Post by **Kylotan** » February 23rd, 2004, 11:00 am

Any idea when the next version of this experimental algorithm will be available then? I'll provide another Thunderbird vs. POPFile comparison for everyone's enjoyment.

DizzyWeb · Post by **DizzyWeb** » February 23rd, 2004, 3:07 pm

mscott wrote:btw, none of this is checked into the trunk. It's an experiment.

Also, there were some bugs in the new algorithm that someone is fixing for me.

Those bugs would be the ones where it doesn't catch anything after you mark anything as junk or not junk?