urlclassifier3.sqlite and excessive I/O and CPU

Discussion about official Mozilla Firefox builds
User avatar
Aus
Posts: 183
Joined: September 6th, 2004, 1:59 pm

urlclassifier3.sqlite and excessive I/O and CPU

Post by Aus »

I've had some problems recently with 3.0pre nightly builds on Windows XP. Shortly after opening fx, I get excessive disk I/O and high CPU usage. I finally tracked the problem down to the urlclassifier3.sqlite database, which on my system is almost 32Mb. If I delete the file, a new file is created 9kb in size and the I/O and CPU problems go away.

I noticed there have been reports of problems in this area from people using linux builds, but I couldn't find anything specific about Windows. Is this a known problem?

Aus
User avatar
a;skdjfajf;ak
Posts: 17002
Joined: July 10th, 2004, 8:44 am

Post by a;skdjfajf;ak »

The total file is about 40meg, and deleting it will only restart the download all over again. I've not seen any performance hits on Vista with the new larger file.
pytechd
Posts: 25
Joined: December 11th, 2007, 5:31 am

Post by pytechd »

How "recent", Aus? Build ID?
User avatar
Aus
Posts: 183
Joined: September 6th, 2004, 1:59 pm

Post by Aus »

To be fair, I was running on a build from a few days ago but with a profile from a few weeks ago. I just replaced that with today's build and a new profile, so we'll see what happens when the file is built.

Aus
rickst29
Posts: 182
Joined: January 22nd, 2003, 10:45 pm

Post by rickst29 »

There's been multiple problems/fixes in this area-- the big for Linux people was bug 430530. pytechd was right to ask exactly which Build, because the fix for 430530 only landed last Sunday AM.

402469 and it's follow-up "relatives" (bug 432490 bug 431563) are all still open, and they're all blockers. These effect all safebrowsing users-- but they're issues with the Google-based Servers, we won't know what Google has done until they post, and the timing of the fix(es) won't be an exact match one of our FF BuildIDs.

Because we (now I'm talking you and I, not mozilla.org) are using these builds as 'volunteer testers', it wouldn't be nice to turn off phishing protection-- Mozilla needs our bug reports. So please leave it active. I think that bug 402469 is the most general of the "urlclassifier3.sqlite" bugs which is still open, so post any misbehavior in that one-- and let Dave Camp split it off if it turns out to be a specific issue still present in our code on Windows.
Make a fire FOR a man, he'll be warm for the rest of the night. Set a man ON fire, he'll be warm for the rest of his life!
sheng
Posts: 8
Joined: May 10th, 2008, 9:22 pm

Post by sheng »

Firefox 3 RC1 still has this issue.
OS: Windows XP Pro SP3

For someone, who uses notebooks on batteries, there is no other solution but to turn off phishing protection, because this excessive CPU/IO activity is draining the battery. Also urlclassifier3.sqlite file is growing bigger, last time I saw it, it was about 53MB. How large it can grow(?) I'm beginning to think that Firefox 3 is not intended for mobile users.
rickst29
Posts: 182
Joined: January 22nd, 2003, 10:45 pm

Post by rickst29 »

I agree about the file size-- why should FF3 need more disk space than an ENTIRE installation of Norton Internet Security? (That's *all* of NIS-- the programs, the data, the updates, everything.) It sounds like you've got the entire file, though-- mine is also 53.2 MB, and I've been running FF3 for many months.

Do you see lots of I/O even when you have phishing protection turned off? I've got a bad feeling that maybe, you're not killing the battery just via "urlclassifier3.sqlite". Could also be bug 421482, a places/history database fiasco. I just scribbled a note into that bug (on your behalf) that battery life is a BIG issue. They've so far been focusing primarily on GUI "lock-up".
Make a fire FOR a man, he'll be warm for the rest of the night. Set a man ON fire, he'll be warm for the rest of his life!
sheng
Posts: 8
Joined: May 10th, 2008, 9:22 pm

Post by sheng »

rickst29 wrote:Do you see lots of I/O even when you have phishing protection turned off? I've got a bad feeling that maybe, you're not killing the battery just via "urlclassifier3.sqlite". Could also be bug 421482, a places/history database fiasco. I just scribbled a note into that bug (on your behalf) that battery life is a BIG issue. They've so far been focusing primarily on GUI "lock-up".

With phishing protection turned off I/O activity is low, almost none. After that, some small I/O activity appears only while browsing, editing/removing/adding bookmarks, clearing history. But CPU load is also noticeable at these moments. I guess excessive I/O and also CPU load (as stated in the post title) both at the same time appear only while updating/working with urlclassifier3.sqlite, it was periodically updated (while browsing or while browser was idle) with large amounts of data: after update file size increased by 2-3MB or more. I guess SQLite engine is not suitable for such kind of transactions, or it is not tuned for best performance at the moment, or update operations are made not in an optimal way.
User avatar
Aus
Posts: 183
Joined: September 6th, 2004, 1:59 pm

Post by Aus »

I've just installed RC1 with a new profile. The size of urlclassifier3.sqlite is currently 54.544Mb and it's killing my CPU and Disk.

Unfortunately, it looks like I too will have to forego phishing protection until this issue is fixed.
User avatar
SickSadWorld
Posts: 116
Joined: February 23rd, 2006, 8:15 pm

Post by SickSadWorld »

Is there a way to stop Fx3 from creating the file and downloading the data?

I have disabled "safebrowsing" and it doesn't make any difference.
User avatar
Bluefang
Posts: 7857
Joined: August 10th, 2005, 2:55 pm
Location: Vermont
Contact:

Post by Bluefang »

You have to disable both the phishing filter and the malicious site filter.
There have always been ghosts in the machine... random segments of code that have grouped together to form unexpected protocols. Unanticipated, these free radicals engender questions of free will, creativity, and even the nature of what we might call the soul...
User avatar
SickSadWorld
Posts: 116
Joined: February 23rd, 2006, 8:15 pm

Post by SickSadWorld »

Done, thanks.
User avatar
Morac
Posts: 2519
Joined: February 9th, 2004, 8:20 pm
Contact:

Post by Morac »

My urlclassifier3.sqlite in Firefox 3.0 RC1 on Windows was at 53,440,512 and it was being rebuilt in the temp directory I shutdown Firefox when it reached around 90 MB. This seems a lot larger than it should have been.

I did try to shut down Firefox when the urlclassifier3.sqlite was first being build (Firefox didn't shut down for a few minutes and was using 99% of the CPU during that time).

Is it possible the urlclassifier3.sqlite file could have become corrupted?
User avatar
MechR
Posts: 1286
Joined: July 30th, 2003, 4:13 pm
Location: Earth

Post by MechR »

On Windows XP I've been seeing periods of disk activity with RC1 as well (several seconds at a time intermittently), but my urlclassifier3.sqlite is only 32KB. OTOH, places.sqlite is 1.75MB. Not sure if that might have anything to do with it.
User avatar
Morac
Posts: 2519
Joined: February 9th, 2004, 8:20 pm
Contact:

Post by Morac »

Well I wiped out the urlclassifier3.sqlite file and it recreated itself, jumped to about 6 MB and is now at about 12 MB and appears to be holding steady. I'm not sure what triggers it to grow or how it decides when to do so.

Considering it grew to > 54 MB in less than an hour I think something went wrong the first time.
Locked