Why are .sqlite files so much larger in Firefox 4 ?

Discussion about official Mozilla Firefox builds
abhinavk
Posts: 40
Joined: July 7th, 2008, 1:19 pm
Location: India
Contact:

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by abhinavk »

You know, even if you create a text file of just 1 character (1 byte) in your 500 GB HDD, it is going to take 4KB (64 KB on FAT32) on your file system. That's increase by a factor of a whooping 4095.

Don't you think that's bloat too?
User avatar
Nightwish
Posts: 171
Joined: September 8th, 2003, 5:26 am

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by Nightwish »

Guest wrote:
Tony-E wrote:In Firefox 4 the places.sqlite file will increase in size in 10MB blocks, this is to avoid file fragmentation. As soon as the file uses the 10MB it will automatically be increased to 20MB. The cookies.sqlite file grows in 512KB increments.
Bug 581606 – Avoid sqlite fragmentation via SQLITE_FCNTL_CHUNK_SIZE

ARGH ! Don't people defrag their systems ? This is wasting disk space unnecessarily and to top it off SQLite Database Browser can't read the padded files.

You are confusing things here. The database fragmentation has absolutely NOTHING to do with filesystem fragmentation. You can defrag a million times a day and it's not going to solve the fragmentation issues of a database with a small chunk size.

Filesystem defrag will only optimize entire files on disk, it's not going to defrag the internal file structure.

The change did make sense, a lot of actually.
lithopsian
Posts: 3664
Joined: September 15th, 2010, 9:03 am

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by lithopsian »

No cache either I hope, since that is tens of times larger than the history database. My places.sqlite has clicked over past 10MB so it is now 20MB. We've had people on here with it larger than that. Another fatty is the google dodgy sites database, I think it was 55MB last time I looked.
hahnchen
Posts: 2
Joined: March 18th, 2011, 3:47 am

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by hahnchen »

I'm disappointed with the dismissive nature of the responses here, as if the only concern is hard disk size. It isn't.

The maximum Windows roaming profile quota size limit, if one is set, is 30mb. Firefox 4 wants to fill a third of my roaming profile with a history database, even though I disable the history. It wants to copy a 10mb empty database across a network every time I log in and out.

Our corporate network is full of laptops, it has people connecting over VPN on 3G networks. I don't want to waste my time every day waiting for this to happen. I don't want to have to clear out my user profile manually (which most users won't even know how to) in order to sync on log off.

I am not unique in this position.

----

Note - I am not the guest that started the thread. My Firefox profile is 18.6mb, this is ridiculous.
Ver Greeneyes
Posts: 1030
Joined: June 28th, 2008, 4:57 am

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by Ver Greeneyes »

That sounds like good reasoning for opening a new bug (though it won't make FF4, at this point).
4td8s
Posts: 784
Joined: June 24th, 2009, 1:07 pm

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by 4td8s »

malliz wrote:So guest lets get this straight. You are running a 10 to 20gig HD and you have no mp3s, video clips or unnecessary programs like a word processor?
In all honesty when was the last time you ran disk cleanup


wow one of the "guests" is using a 10gb or 20gb hard drive? geez, replace either one of them with a bigger and faster one if your computer can handle larger hard drives. :)

Firefox 4 isn't alone with "large" .sqlite files. The Seamonkey 2.1 alphas & betas I've tested also use "large" .sqlite files and I don't worry about them a lot.

btw, my old HP Pavilion computer is using a 30gig hard disk drive and WinXP SP3.
KWierso
Posts: 8829
Joined: May 7th, 2006, 10:29 pm
Location: California

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by KWierso »

http://forums.techarena.in/active-directory/784816.htm
Anyway, the restriction you see in the Group Policy Editor can be
circumvented by alter the system.adm file. It is just a "default" value
that Microsoft has put into this. Maybe they were thinking that no one
needs a larger profile than this ;)

You can dig into %windir%\inf - open the file called "system.adm" and
search for the string called "MaxProfileSize".

You see those three lines there:

VALUENAME "MaxProfileSize"
DEFAULT 30000
MAX 30000
MIN 300

Change the MAX - line to MAX 100000
for 100MB or whatever you need as the maximum. The value is given Kilobytes.
dsmith
Posts: 246
Joined: November 5th, 2002, 1:00 pm

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by dsmith »

Which only solves the problem of the size limit, not the excessive network load/store, if applicable.


I don't have that particular problem (though I can certainly see how it would be annoying), but I did a quick check of my own places.sqlite file and it's at 20 MB. So, at some point in my usage I've gone past the 10 MB starting limit, which indicates that it is indeed possible in 4.0.

Note that I only do about half my browsing in 4.0; the other half I do in 3.6. My cookies.sqlite in 3.6 is 512KB (which does seem mildly excessive, but depends on how much space each cookie takes up; I have perhaps a few hundred cookies), while places.sqlite is 21,548KB. Basically in line with the 4.0 file sizes.

Given that I use the browser a *lot* more than the average user, setting the limits as they did likely gives the average user a huge amount of room to prevent issues of database fragmentation while also keeping file growth for heavy users to a minimum.
Anonymosity
Posts: 8779
Joined: May 7th, 2007, 12:07 pm

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by Anonymosity »

Nightwish wrote:You are confusing things here. The database fragmentation has absolutely NOTHING to do with filesystem fragmentation. You can defrag a million times a day and it's not going to solve the fragmentation issues of a database with a small chunk size.

Filesystem defrag will only optimize entire files on disk, it's not going to defrag the internal file structure.

The change did make sense, a lot of actually.

I must be missing a point somewhere. What difference does it make if the file is composed of one 10 MB piece or, say, three 2 MB pieces? If one is continually adding things and removing other things from the database, there will be gaps in there anyway. Even if the file did have three blocks, they would still be mapped on the drive as one file, right?
michaell522
Posts: 2417
Joined: November 4th, 2002, 4:47 pm
Location: London, UK
Contact:

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by michaell522 »

Anonymosity wrote:I must be missing a point somewhere. What difference does it make if the file is composed of one 10 MB piece or, say, three 2 MB pieces? If one is continually adding things and removing other things from the database, there will be gaps in there anyway. Even if the file did have three blocks, they would still be mapped on the drive as one file, right?

Well, firstly, it wasn't usually 2MB pieces, but more like 128KB pieces, and files could be a lot more than 10MB, so it's not three pieces but dozens of pieces. It all still appears as one file in the file system, yes, but the hard drive has to read them all, unless they are cached. When you load Firefox, the whole file is read, the bits generally aren't cached, and the disk will be busy anyway, so making the hard drive read in dozens of pieces instead of one piece could make loading Firefox take 15-20 seconds instead of 5 seconds.
Ed1
Posts: 1059
Joined: January 30th, 2005, 2:33 pm

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by Ed1 »

Is the change in the minimum size of FF4 .sqlite files relevant to whether a user can just copy these files from his old FF3 profile and paste them into a new FF4 profile and have cookies, places, history, bookmark toolbar, etc. display in the same way and work correctly as they did in FF3?
User avatar
patrickjdempsey
Posts: 23686
Joined: October 23rd, 2008, 11:43 am
Location: Asheville NC
Contact:

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by patrickjdempsey »

There shouldn't be. The first time you run Firefox 4.0 with the 3.x places.sqlite in the profile it will convert it to the new format and expand the size. It is after-all designed to install over Firefox 3.x and use the same profile.
Last edited by patrickjdempsey on March 18th, 2011, 7:31 pm, edited 1 time in total.
Tip of the day: If it has "toolbar" in the name, it's crap.
What my avatar is about: https://addons.mozilla.org/en-US/seamonkey/addon/sea-fox/
User avatar
Bluefang
Posts: 7857
Joined: August 10th, 2005, 2:55 pm
Location: Vermont
Contact:

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by Bluefang »

If you copy a places.sqlite in to a Firefox 4 profile, it should do its standard database migration, which IIRC will also pad the file as well.

Generally, it's not an issue using a file from an older version of Firefox in a newer version. But it can be a problem when you try to use a file from a newer version of Firefox in an older version.
There have always been ghosts in the machine... random segments of code that have grouped together to form unexpected protocols. Unanticipated, these free radicals engender questions of free will, creativity, and even the nature of what we might call the soul...
Anonymosity
Posts: 8779
Joined: May 7th, 2007, 12:07 pm

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by Anonymosity »

michaell wrote:
Anonymosity wrote:I must be missing a point somewhere. What difference does it make if the file is composed of one 10 MB piece or, say, three 2 MB pieces? If one is continually adding things and removing other things from the database, there will be gaps in there anyway. Even if the file did have three blocks, they would still be mapped on the drive as one file, right?

Well, firstly, it wasn't usually 2MB pieces, but more like 128KB pieces, and files could be a lot more than 10MB, so it's not three pieces but dozens of pieces. It all still appears as one file in the file system, yes, but the hard drive has to read them all, unless they are cached. When you load Firefox, the whole file is read, the bits generally aren't cached, and the disk will be busy anyway, so making the hard drive read in dozens of pieces instead of one piece could make loading Firefox take 15-20 seconds instead of 5 seconds.

You misunderstood. With my drive partitioned the way it is with Windows, the clusters are 16K in size, and with the Mac they are always 4K, but that is not the point. I am talking about database fragmentation, not file fragmentation. What I am trying to understand is what difference it makes whether the file is made from one 10 MB piece or three 2 MB pieces, because in either case the database is going to get fragmented anyway.
michaell522
Posts: 2417
Joined: November 4th, 2002, 4:47 pm
Location: London, UK
Contact:

Re: Why are .sqlite files so much larger in Firefox 4 ?

Post by michaell522 »

Anonymosity wrote:You misunderstood. With my drive partitioned the way it is with Windows, the clusters are 16K in size, and with the Mac they are always 4K, but that is not the point. I am talking about database fragmentation, not file fragmentation. What I am trying to understand is what difference it makes whether the file is made from one 10 MB piece or three 2 MB pieces, because in either case the database is going to get fragmented anyway.

Ah... the larger file sizes are all about reducing file system fragmentation. In terms of database fragmentation, it makes no difference how the file is on the disk, as you said.
Post Reply