urlclassifier3.sqlite and excessive I/O and CPU

Discussion about official Mozilla Firefox builds
F J Walter
Posts: 235
Joined: May 26th, 2003, 2:50 am

Post by F J Walter »

Don't know if it's worth mentioning that Bug 402469 is one of only three bugs still marked as 'blocking' which means that the Firefox team intend to hold off the release of Firefox 3 until it's fixed:
https://bugzilla.mozilla.org/show_bug.cgi?id=402469

This bug concerns itself with the slow rate at which the urlclassifier3.sqlite database is built. Its natural size is over 30 megabytes, but this is downloaded gradually from Google, meaning that full protection from attack websites will not be achieved until it has fully downloaded.

If this bug is fixed, the urlclassifier3.sqlite database will grow much faster, which will offer better protection from attack websites.

If you don't want this file to grow at all you will need to disable the "suspected attack site" warning in your Firefox preferences.
User avatar
Bluefang
Posts: 7857
Joined: August 10th, 2005, 2:55 pm
Location: Vermont
Contact:

Post by Bluefang »

Another bug that I just recently saw is this:

Firefox 3 uses fsync excessively

Which has the potential to bring a Linux system to its knees with very large databases. I think this could potentially affect both the urlclassifier and bookmarks/history.
There have always been ghosts in the machine... random segments of code that have grouped together to form unexpected protocols. Unanticipated, these free radicals engender questions of free will, creativity, and even the nature of what we might call the soul...
User avatar
a;skdjfajf;ak
Posts: 17002
Joined: July 10th, 2004, 8:44 am

Post by a;skdjfajf;ak »

And a fix for that bug has just been checked in ... Should be in the latest hourly build.

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9pre) Gecko/2008052206 Minefield/3.0pre Firefox/3.0 ID:2008052206
WAPCE
Posts: 34
Joined: July 18th, 2007, 9:06 pm

Post by WAPCE »

Morac wrote:Well I wiped out the urlclassifier3.sqlite file and it recreated itself, jumped to about 6 MB and is now at about 12 MB and appears to be holding steady. I'm not sure what triggers it to grow or how it decides when to do so.

Considering it grew to > 54 MB in less than an hour I think something went wrong the first time.

That's the correct size.
55894016 2008-05-23 12:33 urlclassifier3.sqlite

There are a <I>lot</I> of malicious sites on the Internet.
rickst29
Posts: 182
Joined: January 22nd, 2003, 10:45 pm

various replies...

Post by rickst29 »

Bluefang wrote:Another bug that I just recently saw is this:
Firefox 3 uses fsync excessively
Which has the potential to bring a Linux system to its knees with very large databases. I think this could potentially affect both the urlclassifier and bookmarks/history.


You're just a bit confused about 421482-- the SIZE of the database doesn't matter, merely the fact that writes are being "confirmed" by flushing to disk frequently is the cause of the problem. We're doing that <b>way, WAY more often</b> on "places.sqlite" than we do it on "urlclassifier3.sqlite". (Basically, every time you touch a new page or even reload a current one, we update your "history".) Also the "fix" which LittleMutt refered to isn't really a "fix", it just creates a preference so that us testers can flip the flushing frequency between the current default "FULL", the new default "NORMAL", and the ultra-efficient but probably dangerous value of "OFF". With a pref, we can do this on-the-fly, instead of loading up a whole new Frefox build for each test. (And of course, having the pref allows the normal Tinderboxen and Nightlies to include the capability, instead of us building custom jobbies over and over and over.)

But testing so far has seen very little difference between "FULL" and "NORMAL", that's not really a solution IMO.

402469 is actually "done", according to the text in the bug. (As you may have noticed, I was the first end-user to pounce on a Tinderbox build which had both 402469 and 430530.) Dave Camp has opened other bugs for more recent issues with the Google Servers (e.g., all the 400s we were getting a while back), but there's nothing happening on this one, I'm not sure why it's still open.

No one ever answered my question about "why is it 50+MB, when Norton Internet Security can fit all their programs AND data in less? They're not merely doing phishing and malware sites in your browsers-- they're also scanning your emails as they arrive, scanning your entire filesystems, running a pretty comprehensive firewall....)
Last edited by rickst29 on May 24th, 2008, 1:34 pm, edited 1 time in total.
Make a fire FOR a man, he'll be warm for the rest of the night. Set a man ON fire, he'll be warm for the rest of his life!
User avatar
Bluefang
Posts: 7857
Joined: August 10th, 2005, 2:55 pm
Location: Vermont
Contact:

Post by Bluefang »

My bad. I didn't read the entire bug, and it started with discussion of the problem becoming noticeable after the history got significantly large.
There have always been ghosts in the machine... random segments of code that have grouped together to form unexpected protocols. Unanticipated, these free radicals engender questions of free will, creativity, and even the nature of what we might call the soul...
Diago
Posts: 51
Joined: November 5th, 2002, 12:48 am

Post by Diago »

I thought I enabled safe browsing in security options by setting both:
- Tell me if the site I'm visiting is a suspected attack site
- Tell me if the site I'm visiting is a suspected forgery

However, on my system I cannot find the file urlclassifier3.sqlite, only a urlclassifier2.sqlite file of 0KB which was last updated on 17/08/07 11:45. Any ideas?

Seems I'm definitely suffered from the places.sqlite (24421KB) problem.
the_dees
Posts: 512
Joined: March 28th, 2007, 3:54 pm

Post by the_dees »

Thank you. Have a nice day.
User avatar
ehume
Posts: 6743
Joined: November 17th, 2002, 12:33 pm
Location: Princeton, NJ, USA

Post by ehume »

Very interesting blog indeed, coming as it does from Shaver.

I've already made my comments on incorporating sqlite into Firefox's bookmarks and history. This is just more of the same.

I'm on WinXP with a 53.4MB url classifiier file. The Firefox installation directory for 3.0RC1 is only 23.3 MB.
Firefox: Sic transit gloria mundi.
User avatar
asquithea
Posts: 1533
Joined: March 17th, 2003, 2:43 pm
Location: Guildford, UK

Post by asquithea »

Interesting reading indeed. I thought complaints about disk i/o were overstating the point, but I guess it's one of those times where it sucks to be a Linux user.
Irongrip
Posts: 1
Joined: May 10th, 2009, 2:57 am

Re: urlclassifier3.sqlite and excessive I/O and CPU

Post by Irongrip »

did any of you think of this before:
1. Delete urlclassifier3.sqlite
2. Recreate it as a 0 byte txt file with the same name
3. Deny Write/Delete Permissions to everyone

Works for me 8-)
User avatar
malliz
Folder@Home
Posts: 43796
Joined: December 7th, 2002, 4:34 am
Location: Australia

Re: urlclassifier3.sqlite and excessive I/O and CPU

Post by malliz »

The world has been waiting around for a year for that :eyeroll2:
What sort of man would put a known criminal in charge of a major branch of government? Apart from, say, the average voter.
"Terry Pratchett"
User avatar
toods
Posts: 255
Joined: December 22nd, 2004, 9:33 am
Location: Cornwall, UK

Re: urlclassifier3.sqlite and excessive I/O and CPU

Post by toods »

Irongrip wrote:did any of you think of this before:
1. Delete urlclassifier3.sqlite
2. Recreate it as a 0 byte txt file with the same name
3. Deny Write/Delete Permissions to everyone

Works for me 8-)

If you had read some of the Forum Posts over the last 12 months since this thread was a active you would have realised that there are ways of avoiding 'urlclassifier3.sqlite' usage by making 'conventional' preference setting changes.

Bill.
Slackware, 12.1
User avatar
ehume
Posts: 6743
Joined: November 17th, 2002, 12:33 pm
Location: Princeton, NJ, USA

Re: urlclassifier3.sqlite and excessive I/O and CPU

Post by ehume »

Please. The guy comes up with a workable solution and you dump on him. He's not claiming he had the only solution, or the best. In fact, framing his question the way he did, he's not even saying he's the first to think of it.

Notice that this is his very first post. With responses like yours, it might be his last. Lighten up, please.
Firefox: Sic transit gloria mundi.
geeknik
Posts: 960
Joined: February 25th, 2003, 10:11 pm
Location: USA

Re: urlclassifier3.sqlite and excessive I/O and CPU

Post by geeknik »

ehume wrote:Please. The guy comes up with a workable solution and you dump on him. He's not claiming he had the only solution, or the best. In fact, framing his question the way he did, he's not even saying he's the first to think of it.

Notice that this is his very first post. With responses like yours, it might be his last. Lighten up, please.


No kidding. I'm trying to get MORE people posting on MozillaZine instead of SUMO and this is counter-productive to my efforts. ](*,)
Locked