urlclassifier3.sqlite and excessive I/O and CPU

[F J Walter]

Don't know if it's worth mentioning that Bug 402469 is one of only three bugs still marked as 'blocking' which means that the Firefox team intend to hold off the release of Firefox 3 until it's fixed:
https://bugzilla.mozilla.org/show_bug.cgi?id=402469

This bug concerns itself with the slow rate at which the urlclassifier3.sqlite database is built. Its natural size is over 30 megabytes, but this is downloaded gradually from Google, meaning that full protection from attack websites will not be achieved until it has fully downloaded.

If this bug is fixed, the urlclassifier3.sqlite database will grow much faster, which will offer better protection from attack websites.

If you don't want this file to grow at all you will need to disable the "suspected attack site" warning in your Firefox preferences.

[Bluefang]

Another bug that I just recently saw is this:

Firefox 3 uses fsync excessively

Which has the potential to bring a Linux system to its knees with very large databases. I think this could potentially affect both the urlclassifier and bookmarks/history.

[Littlemutt]

And a fix for that bug has just been checked in ... Should be in the latest hourly build.

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9pre) Gecko/2008052206 Minefield/3.0pre Firefox/3.0 ID:2008052206

[WAPCE]

Well I wiped out the urlclassifier3.sqlite file and it recreated itself, jumped to about 6 MB and is now at about 12 MB and appears to be holding steady. I'm not sure what triggers it to grow or how it decides when to do so.

Considering it grew to > 54 MB in less than an hour I think something went wrong the first time.

That's the correct size.

55894016 2008-05-23 12:33 urlclassifier3.sqlite

There are a <I>lot</I> of malicious sites on the Internet.

[rickst29]

Another bug that I just recently saw is this:
Firefox 3 uses fsync excessively
Which has the potential to bring a Linux system to its knees with very large databases. I think this could potentially affect both the urlclassifier and bookmarks/history.

You're just a bit confused about 421482-- the SIZE of the database doesn't matter, merely the fact that writes are being "confirmed" by flushing to disk frequently is the cause of the problem. We're doing that <b>way, WAY more often</b> on "places.sqlite" than we do it on "urlclassifier3.sqlite". (Basically, every time you touch a new page or even reload a current one, we update your "history".) Also the "fix" which LittleMutt refered to isn't really a "fix", it just creates a preference so that us testers can flip the flushing frequency between the current default "FULL", the new default "NORMAL", and the ultra-efficient but probably dangerous value of "OFF". With a pref, we can do this on-the-fly, instead of loading up a whole new Frefox build for each test. (And of course, having the pref allows the normal Tinderboxen and Nightlies to include the capability, instead of us building custom jobbies over and over and over.)

But testing so far has seen very little difference between "FULL" and "NORMAL", that's not really a solution IMO.

402469 is actually "done", according to the text in the bug. (As you may have noticed, I was the first end-user to pounce on a Tinderbox build which had both 402469 and 430530.) Dave Camp has opened other bugs for more recent issues with the Google Servers (e.g., all the 400s we were getting a while back), but there's nothing happening on this one, I'm not sure why it's still open.

No one ever answered my question about "why is it 50+MB, when Norton Internet Security can fit all their programs AND data in less? They're not merely doing phishing and malware sites in your browsers-- they're also scanning your emails as they arrive, scanning your entire filesystems, running a pretty comprehensive firewall....)

[Bluefang]

My bad. I didn't read the entire bug, and it started with discussion of the problem becoming noticeable after the history got significantly large.

[Diago]

I thought I enabled safe browsing in security options by setting both:
- Tell me if the site I'm visiting is a suspected attack site
- Tell me if the site I'm visiting is a suspected forgery

However, on my system I cannot find the file urlclassifier3.sqlite, only a urlclassifier2.sqlite file of 0KB which was last updated on 17/08/07 11:45. Any ideas?

Seems I'm definitely suffered from the places.sqlite (24421KB) problem.

[the_dees]

Very interesting blogpost about this issue!

[ehume]

Very interesting blog indeed, coming as it does from Shaver.

I've already made my comments on incorporating sqlite into Firefox's bookmarks and history. This is just more of the same.

I'm on WinXP with a 53.4MB url classifiier file. The Firefox installation directory for 3.0RC1 is only 23.3 MB.

[asquithea]

Interesting reading indeed. I thought complaints about disk i/o were overstating the point, but I guess it's one of those times where it sucks to be a Linux user.

[Irongrip]

did any of you think of this before:
1. Delete urlclassifier3.sqlite
2. Recreate it as a 0 byte txt file with the same name
3. Deny Write/Delete Permissions to everyone

Works for me

[malliz]

The world has been waiting around for a year for that

[toods]

did any of you think of this before:
1. Delete urlclassifier3.sqlite
2. Recreate it as a 0 byte txt file with the same name
3. Deny Write/Delete Permissions to everyone

Works for me

If you had read some of the Forum Posts over the last 12 months since this thread was a active you would have realised that there are ways of avoiding 'urlclassifier3.sqlite' usage by making 'conventional' preference setting changes.

Bill.

[ehume]

Please. The guy comes up with a workable solution and you dump on him. He's not claiming he had the only solution, or the best. In fact, framing his question the way he did, he's not even saying he's the first to think of it.

Notice that this is his very first post. With responses like yours, it might be his last. Lighten up, please.

[geeknik]

Please. The guy comes up with a workable solution and you dump on him. He's not claiming he had the only solution, or the best. In fact, framing his question the way he did, he's not even saying he's the first to think of it.

Notice that this is his very first post. With responses like yours, it might be his last. Lighten up, please.

No kidding. I'm trying to get MORE people posting on MozillaZine instead of SUMO and this is counter-productive to my efforts.