MozillaZine

Why are .sqlite files so much larger in Firefox 4 ?

Discussion about official Mozilla Firefox builds
abhinavk
 
Posts: 40
Joined: July 7th, 2008, 1:19 pm
Location: India

Post Posted March 12th, 2011, 11:41 pm

You know, even if you create a text file of just 1 character (1 byte) in your 500 GB HDD, it is going to take 4KB (64 KB on FAT32) on your file system. That's increase by a factor of a whooping 4095.

Don't you think that's bloat too?

Nightwish

User avatar
 
Posts: 169
Joined: September 8th, 2003, 5:26 am

Post Posted March 13th, 2011, 6:23 am

Guest wrote:
Tony-E wrote:In Firefox 4 the places.sqlite file will increase in size in 10MB blocks, this is to avoid file fragmentation. As soon as the file uses the 10MB it will automatically be increased to 20MB. The cookies.sqlite file grows in 512KB increments.
Bug 581606 – Avoid sqlite fragmentation via SQLITE_FCNTL_CHUNK_SIZE

ARGH ! Don't people defrag their systems ? This is wasting disk space unnecessarily and to top it off SQLite Database Browser can't read the padded files.

You are confusing things here. The database fragmentation has absolutely NOTHING to do with filesystem fragmentation. You can defrag a million times a day and it's not going to solve the fragmentation issues of a database with a small chunk size.

Filesystem defrag will only optimize entire files on disk, it's not going to defrag the internal file structure.

The change did make sense, a lot of actually.

lithopsian
 
Posts: 3252
Joined: September 15th, 2010, 9:03 am

Post Posted March 13th, 2011, 12:40 pm

No cache either I hope, since that is tens of times larger than the history database. My places.sqlite has clicked over past 10MB so it is now 20MB. We've had people on here with it larger than that. Another fatty is the google dodgy sites database, I think it was 55MB last time I looked.

hahnchen
 
Posts: 1
Joined: March 18th, 2011, 3:47 am

Post Posted March 18th, 2011, 3:59 am

I'm disappointed with the dismissive nature of the responses here, as if the only concern is hard disk size. It isn't.

The maximum Windows roaming profile quota size limit, if one is set, is 30mb. Firefox 4 wants to fill a third of my roaming profile with a history database, even though I disable the history. It wants to copy a 10mb empty database across a network every time I log in and out.

Our corporate network is full of laptops, it has people connecting over VPN on 3G networks. I don't want to waste my time every day waiting for this to happen. I don't want to have to clear out my user profile manually (which most users won't even know how to) in order to sync on log off.

I am not unique in this position.

----

Note - I am not the guest that started the thread. My Firefox profile is 18.6mb, this is ridiculous.

Ver Greeneyes
 
Posts: 834
Joined: June 28th, 2008, 4:57 am

Post Posted March 18th, 2011, 5:06 am

That sounds like good reasoning for opening a new bug (though it won't make FF4, at this point).

4td8s
 
Posts: 486
Joined: June 24th, 2009, 1:07 pm

Post Posted March 18th, 2011, 9:15 am

malliz wrote:So guest lets get this straight. You are running a 10 to 20gig HD and you have no mp3s, video clips or unnecessary programs like a word processor?
In all honesty when was the last time you ran disk cleanup


wow one of the "guests" is using a 10gb or 20gb hard drive? geez, replace either one of them with a bigger and faster one if your computer can handle larger hard drives. :)

Firefox 4 isn't alone with "large" .sqlite files. The Seamonkey 2.1 alphas & betas I've tested also use "large" .sqlite files and I don't worry about them a lot.

btw, my old HP Pavilion computer is using a 30gig hard disk drive and WinXP SP3.

KWierso
 
Posts: 8818
Joined: May 7th, 2006, 10:29 pm
Location: California

Post Posted March 18th, 2011, 9:21 am

http://forums.techarena.in/active-directory/784816.htm
Anyway, the restriction you see in the Group Policy Editor can be
circumvented by alter the system.adm file. It is just a "default" value
that Microsoft has put into this. Maybe they were thinking that no one
needs a larger profile than this ;)

You can dig into %windir%\inf - open the file called "system.adm" and
search for the string called "MaxProfileSize".

You see those three lines there:

VALUENAME "MaxProfileSize"
DEFAULT 30000
MAX 30000
MIN 300

Change the MAX - line to MAX 100000
for 100MB or whatever you need as the maximum. The value is given Kilobytes.

dsmith
 
Posts: 182
Joined: November 5th, 2002, 1:00 pm

Post Posted March 18th, 2011, 11:01 am

Which only solves the problem of the size limit, not the excessive network load/store, if applicable.


I don't have that particular problem (though I can certainly see how it would be annoying), but I did a quick check of my own places.sqlite file and it's at 20 MB. So, at some point in my usage I've gone past the 10 MB starting limit, which indicates that it is indeed possible in 4.0.

Note that I only do about half my browsing in 4.0; the other half I do in 3.6. My cookies.sqlite in 3.6 is 512KB (which does seem mildly excessive, but depends on how much space each cookie takes up; I have perhaps a few hundred cookies), while places.sqlite is 21,548KB. Basically in line with the 4.0 file sizes.

Given that I use the browser a *lot* more than the average user, setting the limits as they did likely gives the average user a huge amount of room to prevent issues of database fragmentation while also keeping file growth for heavy users to a minimum.

Anonymosity
 
Posts: 8166
Joined: May 7th, 2007, 12:07 pm

Post Posted March 18th, 2011, 12:06 pm

Nightwish wrote:You are confusing things here. The database fragmentation has absolutely NOTHING to do with filesystem fragmentation. You can defrag a million times a day and it's not going to solve the fragmentation issues of a database with a small chunk size.

Filesystem defrag will only optimize entire files on disk, it's not going to defrag the internal file structure.

The change did make sense, a lot of actually.

I must be missing a point somewhere. What difference does it make if the file is composed of one 10 MB piece or, say, three 2 MB pieces? If one is continually adding things and removing other things from the database, there will be gaps in there anyway. Even if the file did have three blocks, they would still be mapped on the drive as one file, right?

michaell
 
Posts: 2417
Joined: November 4th, 2002, 4:47 pm
Location: London, UK

Post Posted March 18th, 2011, 12:55 pm

Anonymosity wrote:I must be missing a point somewhere. What difference does it make if the file is composed of one 10 MB piece or, say, three 2 MB pieces? If one is continually adding things and removing other things from the database, there will be gaps in there anyway. Even if the file did have three blocks, they would still be mapped on the drive as one file, right?

Well, firstly, it wasn't usually 2MB pieces, but more like 128KB pieces, and files could be a lot more than 10MB, so it's not three pieces but dozens of pieces. It all still appears as one file in the file system, yes, but the hard drive has to read them all, unless they are cached. When you load Firefox, the whole file is read, the bits generally aren't cached, and the disk will be busy anyway, so making the hard drive read in dozens of pieces instead of one piece could make loading Firefox take 15-20 seconds instead of 5 seconds.

Ed1
 
Posts: 754
Joined: January 30th, 2005, 2:33 pm

Post Posted March 18th, 2011, 7:10 pm

Is the change in the minimum size of FF4 .sqlite files relevant to whether a user can just copy these files from his old FF3 profile and paste them into a new FF4 profile and have cookies, places, history, bookmark toolbar, etc. display in the same way and work correctly as they did in FF3?

patrickjdempsey

User avatar
 
Posts: 22782
Joined: October 23rd, 2008, 11:43 am
Location: Asheville NC

Post Posted March 18th, 2011, 7:30 pm

There shouldn't be. The first time you run Firefox 4.0 with the 3.x places.sqlite in the profile it will convert it to the new format and expand the size. It is after-all designed to install over Firefox 3.x and use the same profile.
Last edited by patrickjdempsey on March 18th, 2011, 7:31 pm, edited 1 time in total.
Tip of the day: If it has "toolbar" in the name, it's crap.
What my avatar is about: https://addons.mozilla.org/en-US/seamonkey/addon/sea-fox/

Bluefang

User avatar
 
Posts: 7847
Joined: August 10th, 2005, 2:55 pm
Location: Vermont

Post Posted March 18th, 2011, 7:30 pm

If you copy a places.sqlite in to a Firefox 4 profile, it should do its standard database migration, which IIRC will also pad the file as well.

Generally, it's not an issue using a file from an older version of Firefox in a newer version. But it can be a problem when you try to use a file from a newer version of Firefox in an older version.
There have always been ghosts in the machine... random segments of code that have grouped together to form unexpected protocols. Unanticipated, these free radicals engender questions of free will, creativity, and even the nature of what we might call the soul...

Anonymosity
 
Posts: 8166
Joined: May 7th, 2007, 12:07 pm

Post Posted March 19th, 2011, 11:20 am

michaell wrote:
Anonymosity wrote:I must be missing a point somewhere. What difference does it make if the file is composed of one 10 MB piece or, say, three 2 MB pieces? If one is continually adding things and removing other things from the database, there will be gaps in there anyway. Even if the file did have three blocks, they would still be mapped on the drive as one file, right?

Well, firstly, it wasn't usually 2MB pieces, but more like 128KB pieces, and files could be a lot more than 10MB, so it's not three pieces but dozens of pieces. It all still appears as one file in the file system, yes, but the hard drive has to read them all, unless they are cached. When you load Firefox, the whole file is read, the bits generally aren't cached, and the disk will be busy anyway, so making the hard drive read in dozens of pieces instead of one piece could make loading Firefox take 15-20 seconds instead of 5 seconds.

You misunderstood. With my drive partitioned the way it is with Windows, the clusters are 16K in size, and with the Mac they are always 4K, but that is not the point. I am talking about database fragmentation, not file fragmentation. What I am trying to understand is what difference it makes whether the file is made from one 10 MB piece or three 2 MB pieces, because in either case the database is going to get fragmented anyway.

michaell
 
Posts: 2417
Joined: November 4th, 2002, 4:47 pm
Location: London, UK

Post Posted March 19th, 2011, 11:32 am

Anonymosity wrote:You misunderstood. With my drive partitioned the way it is with Windows, the clusters are 16K in size, and with the Mac they are always 4K, but that is not the point. I am talking about database fragmentation, not file fragmentation. What I am trying to understand is what difference it makes whether the file is made from one 10 MB piece or three 2 MB pieces, because in either case the database is going to get fragmented anyway.

Ah... the larger file sizes are all about reducing file system fragmentation. In terms of database fragmentation, it makes no difference how the file is on the disk, as you said.

Return to Firefox Builds


Who is online

Users browsing this forum: Grantius and 5 guests