MozillaZine

urlclassifier3.sqlite gets absurdly large?

Discussion of general topics about Mozilla Firefox
wellofsouls
 
Posts: 168
Joined: March 22nd, 2008, 10:12 pm

Post Posted June 1st, 2008, 7:30 am

For Fx2, the urlclassifier2.sqlite never got bigger than 10M even after months, but now the urlclassifier3.sqlite in Fx3 RC1 gets to over 90M under weeks. I wonder if there's any settings about this, or any option to purge the file? thanks.

suser
 
Posts: 113
Joined: May 18th, 2008, 7:40 am

Post Posted June 1st, 2008, 8:58 am

wellofsouls wrote:For Fx2, the urlclassifier2.sqlite never got bigger than 10M even after months, but now the urlclassifier3.sqlite in Fx3 RC1 gets to over 90M under weeks. I wonder if there's any settings about this, or any option to purge the file? thanks.

Thanks for mentioning it, urlclassifier2.sqlite of my Firefox 2 profile - which is actually my work profile - is 4,7 MB large, urlclassifier3.sqlite of my test Firefox 3 installation is 53,2 MB.

Durian_CS

User avatar
 
Posts: 108
Joined: March 29th, 2005, 11:53 am
Location: The Netherlands

Post Posted June 1st, 2008, 9:17 am

wellofsouls wrote:For Fx2, the urlclassifier2.sqlite never got bigger than 10M even after months, but now the urlclassifier3.sqlite in Fx3 RC1 gets to over 90M under weeks. I wonder if there's any settings about this, or any option to purge the file? thanks.


You can purge the file, if you want, but this file is downloaded gradually from Google, meaning that full protection from attack websites implies a large file size, see http://forums.mozillazine.org/viewtopic ... 33#3381133
Durian, King of Fruits
My Minimal Theme

RobertJ
Moderator

User avatar
 
Posts: 10660
Joined: October 15th, 2003, 7:40 pm
Location: Chicago IL/Oconomowoc WI

Post Posted June 1st, 2008, 9:43 am

I've been using FF3 b4/b5/rc1 and don't have a urlclassifier3.sqlite file. Any suggestions? I use Noscript and Adblock Plus. I can't find any setting in FF3 preferences to allow Google's list.

EDIT: FOUND MY urlclassifier3.sqlite FILE thanks to Harlequin99.
Last edited by RobertJ on June 1st, 2008, 10:36 am, edited 1 time in total.
FF 62.0.3 - FF 63b14 - FF 64a - TB 60.2.1 - Mac OSX 10.13.6
Computers I've used: IBM 7094/UNIVAC 1108/IBM 360/DEC PDP11/DEC VAX-11 780/DEC VAXstation 8000/Sun SPARCstation 2/Mac from 1984 to 2018

suser
 
Posts: 113
Joined: May 18th, 2008, 7:40 am

Post Posted June 1st, 2008, 9:48 am

Durian_CS wrote:You can purge the file, if you want, but this file is downloaded gradually from Google, meaning that full protection from attack websites implies a large file size, see http://forums.mozillazine.org/viewtopic ... 33#3381133

But why has it been a tenth in size in Firefox 2?

sysKin

User avatar
 
Posts: 893
Joined: March 17th, 2004, 9:09 pm
Location: Adelaide, Australia

Post Posted June 1st, 2008, 9:59 am

Probably because it contains malware protection, on top of phishing protection.

suser
 
Posts: 113
Joined: May 18th, 2008, 7:40 am

Post Posted June 1st, 2008, 10:20 am

sysKin wrote:Probably because it contains malware protection, on top of phishing protection.

Thanks sysKin, but I can hardly believe that those additional URLs are nine times in size of the existing phishing URLs in Firefox 2.

Bockworscht

User avatar
 
Posts: 159
Joined: May 19th, 2008, 1:57 pm

Post Posted June 1st, 2008, 10:30 am

The Bible in .txt format takes 1,64 MB

How many MILLION Sites were in this file?
And how many thousand people were working on this list, on order to add so many sites in such a short time?

What is in 1 year?
The filesize is then 150 MB?

wellofsouls
 
Posts: 168
Joined: March 22nd, 2008, 10:12 pm

Post Posted June 1st, 2008, 9:03 pm

is there any documentation on the content of the file? what's moz_classifier, moz_subs, and moz_tables?

IdahoEv
 
Posts: 3
Joined: June 18th, 2008, 5:18 pm

Post Posted June 18th, 2008, 5:29 pm

One of my users' urlclassifier2.sqlite file grew to 775GB (yes, GB, that's not a typo) in one day yesterday. We discovered it when our nightly incremental backup suddenly reported that it was going to take 46 hours to complete.

I'd say there's a problem.

I've seen one other report of these files growing to several hundred gigs, but it's on the members-only paysite experts-exchange.com, so I couldn't see what the resolution was.

polidobj

User avatar
 
Posts: 3147
Joined: March 31st, 2004, 9:10 am
Location: Maryland USA - im in ur tinderbox, crashtesting ur firefox

Post Posted June 18th, 2008, 5:42 pm

suser wrote:
sysKin wrote:Probably because it contains malware protection, on top of phishing protection.

Thanks sysKin, but I can hardly believe that those additional URLs are nine times in size of the existing phishing URLs in Firefox 2.
After seeing many articles like this one:
http://www.zdnet.com.au/news/security/s ... 654,00.htm

Combined with what Google Security said back in February which is they have over 3 million malware sites:
http://googleonlinesecurity.blogspot.co ... to-us.html

So it seems believable that malware is several times as bad a phishing.
Brian J Polidoro - Today's bugs brought to you by Raid. :P
Windows7 - Firefox user since ~Feb 2002

rickst29
 
Posts: 182
Joined: January 22nd, 2003, 10:45 pm

Post Posted June 18th, 2008, 8:05 pm

wellofsouls wrote:For Fx2, the urlclassifier2.sqlite never got bigger than 10M even after months, but now the urlclassifier3.sqlite in Fx3 RC1 gets to over 90M under weeks. I wonder if there's any settings about this, or any option to purge the file? thanks.


The new database isn't merely handling "another kind" of bad site (it's WAY more comprehensive). But WoS, I think that you SHOULD delete the file from your profile (while Firefox is NOT running). And do that AFTER updating from RC1 to "final". There weren't any changes affecting phishing/malware DB operations between RC2 and Final, but IIRC there were changes between RC1 and RC2. The correct size is only about 54MB, and erasing it will allow the new file to be built "properly". If you've been using Firefox "Beta" or "Nightly" versions from even earlier than RC1, there could be quite a lot of "grunge" from older implementations, doing absolutely nothing except consuming disk space.

As for the 54 megabytes-- I agree, it's a little odd that the DB is bigger than an entire install of something like Norton Internet Security (including all the programs, and Norton does much more-- Firewall, email scanning, disk scanning, etc.). I'm not saying that I like NIS, not at all-- but the humongous size of the DB is kinda surprising. Still, yours (at 90+MB) is definitely not right, you're nearly double what it should take.
Make a fire FOR a man, he'll be warm for the rest of the night. Set a man ON fire, he'll be warm for the rest of his life!

1fud
 
Posts: 1
Joined: September 20th, 2008, 10:59 am

Post Posted September 20th, 2008, 11:07 am

I found this forum just looking for info on urlclassifier3.sqlite. I had noticed that "Sweepi" a free cleanup application from http://sweepi.yooapps.com/?l=E takes a while (maybe 45 seconds) sometimes to delete it. I gather fro the entries in this forum that deleting it hurts nothing and in fact is a really good idea. So is this program, I've been using it daily with XP for quite a while with no ill effects. Solves this problem, that's for sure

steviex
Moderator

User avatar
 
Posts: 28902
Joined: August 12th, 2006, 8:27 am
Location: Middle England

Post Posted September 20th, 2008, 12:31 pm

Remember this... Every time you delete urlclassifier3.sqlite, Firefox 3 will redownload it....

Make sure that you have the available Download Capacity in your agreement with your ISP to download that file every day......
Only two things are infinite, the universe and human stupidity, and I'm not sure about the former. -Albert Einstein

Please DO NOT PM me for support... Lets keep it on the board, so we can all learn.

VanillaMozilla
 
Posts: 13808
Joined: November 7th, 2005, 11:26 am

Post Posted September 24th, 2008, 5:42 am

1fud wrote:I had noticed that "Sweepi" a free cleanup application

You really shouldn't use unknown apps--or any unnecessary 3rd party app--for maintenance--and certainly not just to delete a file. The correct size, as all agree, is about 55 MB. Delete it ONCE and that should be all you ever have to do. If it isn't, then something is wrong, and you have something else tinkering around with that file that shouldn't be there. If you really want to bother, there is also a sqlite utility that will "vacuum" the file and compact it by roughly 2 x. But I wouldn't use that if the file isn't already the correct size.

Return to Firefox General


Who is online

Users browsing this forum: No registered users and 0 guests