urlclassifier3.sqlite gets absurdly large?

Discussion of general topics about Mozilla Firefox
wellofsouls
Posts: 168
Joined: March 22nd, 2008, 10:12 pm

urlclassifier3.sqlite gets absurdly large?

Post by wellofsouls »

For Fx2, the urlclassifier2.sqlite never got bigger than 10M even after months, but now the urlclassifier3.sqlite in Fx3 RC1 gets to over 90M under weeks. I wonder if there's any settings about this, or any option to purge the file? thanks.
suser
Posts: 113
Joined: May 18th, 2008, 7:40 am

Re: urlclassifier3.sqlite gets absurdly large?

Post by suser »

wellofsouls wrote:For Fx2, the urlclassifier2.sqlite never got bigger than 10M even after months, but now the urlclassifier3.sqlite in Fx3 RC1 gets to over 90M under weeks. I wonder if there's any settings about this, or any option to purge the file? thanks.

Thanks for mentioning it, urlclassifier2.sqlite of my Firefox 2 profile - which is actually my work profile - is 4,7 MB large, urlclassifier3.sqlite of my test Firefox 3 installation is 53,2 MB.
User avatar
Durian_CS
Posts: 108
Joined: March 29th, 2005, 11:53 am
Location: The Netherlands

Re: urlclassifier3.sqlite gets absurdly large?

Post by Durian_CS »

wellofsouls wrote:For Fx2, the urlclassifier2.sqlite never got bigger than 10M even after months, but now the urlclassifier3.sqlite in Fx3 RC1 gets to over 90M under weeks. I wonder if there's any settings about this, or any option to purge the file? thanks.


You can purge the file, if you want, but this file is downloaded gradually from Google, meaning that full protection from attack websites implies a large file size, see http://forums.mozillazine.org/viewtopic ... 33#3381133
Durian, King of Fruits
My Minimal Theme
User avatar
RobertJ
Moderator
Posts: 10880
Joined: October 15th, 2003, 7:40 pm
Location: Chicago IL/Oconomowoc WI

Post by RobertJ »

I've been using FF3 b4/b5/rc1 and don't have a urlclassifier3.sqlite file. Any suggestions? I use Noscript and Adblock Plus. I can't find any setting in FF3 preferences to allow Google's list.

EDIT: FOUND MY urlclassifier3.sqlite FILE thanks to Harlequin99.
Last edited by RobertJ on June 1st, 2008, 10:36 am, edited 1 time in total.
FF 92.0 - TB 78.13 - Mac OSX 10.13.6
suser
Posts: 113
Joined: May 18th, 2008, 7:40 am

Re: urlclassifier3.sqlite gets absurdly large?

Post by suser »

Durian_CS wrote:You can purge the file, if you want, but this file is downloaded gradually from Google, meaning that full protection from attack websites implies a large file size, see http://forums.mozillazine.org/viewtopic ... 33#3381133

But why has it been a tenth in size in Firefox 2?
User avatar
sysKin
Posts: 902
Joined: March 17th, 2004, 9:09 pm
Location: Adelaide, Australia

Post by sysKin »

Probably because it contains malware protection, on top of phishing protection.
suser
Posts: 113
Joined: May 18th, 2008, 7:40 am

Post by suser »

sysKin wrote:Probably because it contains malware protection, on top of phishing protection.

Thanks sysKin, but I can hardly believe that those additional URLs are nine times in size of the existing phishing URLs in Firefox 2.
User avatar
Bockworscht
Posts: 159
Joined: May 19th, 2008, 1:57 pm
Contact:

Post by Bockworscht »

The Bible in .txt format takes 1,64 MB

How many MILLION Sites were in this file?
And how many thousand people were working on this list, on order to add so many sites in such a short time?

What is in 1 year?
The filesize is then 150 MB?
wellofsouls
Posts: 168
Joined: March 22nd, 2008, 10:12 pm

Post by wellofsouls »

is there any documentation on the content of the file? what's moz_classifier, moz_subs, and moz_tables?
IdahoEv
Posts: 3
Joined: June 18th, 2008, 5:18 pm

Re: urlclassifier3.sqlite gets absurdly large?

Post by IdahoEv »

One of my users' urlclassifier2.sqlite file grew to 775GB (yes, GB, that's not a typo) in one day yesterday. We discovered it when our nightly incremental backup suddenly reported that it was going to take 46 hours to complete.

I'd say there's a problem.

I've seen one other report of these files growing to several hundred gigs, but it's on the members-only paysite experts-exchange.com, so I couldn't see what the resolution was.
User avatar
polidobj
Posts: 3147
Joined: March 31st, 2004, 9:10 am
Location: Maryland USA - im in ur tinderbox, crashtesting ur firefox

Re:

Post by polidobj »

suser wrote:
sysKin wrote:Probably because it contains malware protection, on top of phishing protection.

Thanks sysKin, but I can hardly believe that those additional URLs are nine times in size of the existing phishing URLs in Firefox 2.
After seeing many articles like this one:
http://www.zdnet.com.au/news/security/s ... 654,00.htm

Combined with what Google Security said back in February which is they have over 3 million malware sites:
http://googleonlinesecurity.blogspot.co ... to-us.html

So it seems believable that malware is several times as bad a phishing.
Brian J Polidoro - Today's bugs brought to you by Raid. :P
Windows7 - Firefox user since ~Feb 2002
rickst29
Posts: 182
Joined: January 22nd, 2003, 10:45 pm

Re: urlclassifier3.sqlite gets absurdly large?

Post by rickst29 »

wellofsouls wrote:For Fx2, the urlclassifier2.sqlite never got bigger than 10M even after months, but now the urlclassifier3.sqlite in Fx3 RC1 gets to over 90M under weeks. I wonder if there's any settings about this, or any option to purge the file? thanks.


The new database isn't merely handling "another kind" of bad site (it's WAY more comprehensive). But WoS, I think that you SHOULD delete the file from your profile (while Firefox is NOT running). And do that AFTER updating from RC1 to "final". There weren't any changes affecting phishing/malware DB operations between RC2 and Final, but IIRC there were changes between RC1 and RC2. The correct size is only about 54MB, and erasing it will allow the new file to be built "properly". If you've been using Firefox "Beta" or "Nightly" versions from even earlier than RC1, there could be quite a lot of "grunge" from older implementations, doing absolutely nothing except consuming disk space.

As for the 54 megabytes-- I agree, it's a little odd that the DB is bigger than an entire install of something like Norton Internet Security (including all the programs, and Norton does much more-- Firewall, email scanning, disk scanning, etc.). I'm not saying that I like NIS, not at all-- but the humongous size of the DB is kinda surprising. Still, yours (at 90+MB) is definitely not right, you're nearly double what it should take.
Make a fire FOR a man, he'll be warm for the rest of the night. Set a man ON fire, he'll be warm for the rest of his life!
1fud
Posts: 1
Joined: September 20th, 2008, 10:59 am

Re: urlclassifier3.sqlite gets absurdly large?

Post by 1fud »

I found this forum just looking for info on urlclassifier3.sqlite. I had noticed that "Sweepi" a free cleanup application from http://sweepi.yooapps.com/?l=E takes a while (maybe 45 seconds) sometimes to delete it. I gather fro the entries in this forum that deleting it hurts nothing and in fact is a really good idea. So is this program, I've been using it daily with XP for quite a while with no ill effects. Solves this problem, that's for sure
User avatar
steviex
Moderator
Posts: 28902
Joined: August 12th, 2006, 8:27 am
Location: Middle England

Re: urlclassifier3.sqlite gets absurdly large?

Post by steviex »

Remember this... Every time you delete urlclassifier3.sqlite, Firefox 3 will redownload it....

Make sure that you have the available Download Capacity in your agreement with your ISP to download that file every day......
Only two things are infinite, the universe and human stupidity, and I'm not sure about the former. -Albert Einstein

Please DO NOT PM me for support... Lets keep it on the board, so we can all learn.
VanillaMozilla
Posts: 13808
Joined: November 7th, 2005, 11:26 am

Re: urlclassifier3.sqlite gets absurdly large?

Post by VanillaMozilla »

1fud wrote:I had noticed that "Sweepi" a free cleanup application

You really shouldn't use unknown apps--or any unnecessary 3rd party app--for maintenance--and certainly not just to delete a file. The correct size, as all agree, is about 55 MB. Delete it ONCE and that should be all you ever have to do. If it isn't, then something is wrong, and you have something else tinkering around with that file that shouldn't be there. If you really want to bother, there is also a sqlite utility that will "vacuum" the file and compact it by roughly 2 x. But I wouldn't use that if the file isn't already the correct size.
Locked