Why are .sqlite files so much larger in Firefox 4 ?

[Guest]

With a new profile places.sqlite is 10MB and cookies.sqlite is 512KB, that's a rather big difference from Firefox 3.x, what gives ? Also is there a way to reduce the sizes ?

[Daifne]

Problems with the test builds need to be discussed in the Firefox Builds forum. Please register and post back here. I will then move this for you.

[Guest]

Problems with the test builds need to be discussed in the Firefox Builds forum. Please register and post back here. I will then move this for you.

I'm using 4.0 RC1 which will likely become 4.0.0 so is this really considered a test build like the betas ? Anyway it's not an actual problem per se though it is curious as to why there's such a huge jump in file sizes.

[sdwilsh]

These files will grow to roughly these sizes in normal use, so they start out that big to reduce fragmentation issues, which end up making startup slower.

[Guest]

These files will grow to roughly these sizes in normal use, so they start out that big to reduce fragmentation issues, which end up making startup slower.

I've never had those files grow to these sizes and I regularly defrag my system. Is there a way to configure the initial sizes to something more reasonable ?

[Tony-E]

In Firefox 4 the places.sqlite file will increase in size in 10MB blocks, this is to avoid file fragmentation. As soon as the file uses the 10MB it will automatically be increased to 20MB. The cookies.sqlite file grows in 512KB increments.
Bug 581606 – Avoid sqlite fragmentation via SQLITE_FCNTL_CHUNK_SIZE

[Guest]

In Firefox 4 the places.sqlite file will increase in size in 10MB blocks, this is to avoid file fragmentation. As soon as the file uses the 10MB it will automatically be increased to 20MB. The cookies.sqlite file grows in 512KB increments.
Bug 581606 – Avoid sqlite fragmentation via SQLITE_FCNTL_CHUNK_SIZE

ARGH ! Don't people defrag their systems ? This is wasting disk space unnecessarily and to top it off SQLite Database Browser can't read the padded files.

[sdwilsh]

I've never had those files grow to these sizes and I regularly defrag my system. Is there a way to configure the initial sizes to something more reasonable ?

No. The amount of data that is stored in places.sqlite has also increased for Firefox 4.0, so while you may not have hit that in the past, you are more likely to with Firefox 4.0.

[KWierso]

There's a difference between file system fragmentation and SQLite database fragmentation.

[Anonymosity]

That much space seems extremely excessive to me. I have been doing some browsing, and my cookies.sqlite file size is 8 KB, while my places.sqlite file is 596 KB. I do, after all, default to blocking cookies, and I block favicons, and I do not keep months of history.

[malliz]

Excessive? For petes sake how big is your HD?

[Guest]

I've never had those files grow to these sizes and I regularly defrag my system. Is there a way to configure the initial sizes to something more reasonable ?

No. The amount of data that is stored in places.sqlite has also increased for Firefox 4.0, so while you may not have hit that in the past, you are more likely to with Firefox 4.0.

I seriously doubt that, these bundle sizes are excessive.

cookies.sqlite: 512KB - in 3.x my file rarely surpasses 16KB; I reject third-party cookies, have some filtered out via permissions and periodically delete crap cookies

places.sqlite: 10MB - in 3.x my files rarely surpasses 2MB; I'm aware that bookmarks are now stored here in 4.0 but that's not going to make a significant difference. What else is stored here that will ?

Also much like with omni.jar this change introduces an incompatibility with other utilities, SQLite Database Browser http://sqlitebrowser.sourceforge.net doesn't like the padded files and can't process them correctly.

[sdwilsh]

I seriously doubt that, these bundle sizes are excessive.

Maybe not for you specifically, but Mozilla doesn't design things specifically for you. The data is based on a common case.

cookies.sqlite: 512KB - in 3.x my file rarely surpasses 16KB; I reject third-party cookies, have some filtered out via permissions and periodically delete crap cookies

The maximum number of cookies (per host and global) limits have changed in 4.0.

places.sqlite: 10MB - in 3.x my files rarely surpasses 2MB; I'm aware that bookmarks are now stored here in 4.0 but that's not going to make a significant difference. What else is stored here that will ?

History, bookmarks, favicons, and what you've typed in the location bar (specific letters that map to what you then visited). The amount of history Firefox keeps has been increased, and what data it stores for both history and bookmarks has increased.

Also much like with omni.jar this change introduces an incompatibility with other utilities, SQLite Database Browser http://sqlitebrowser.sourceforge.net doesn't like the padded files and can't process them correctly.

Mozilla doesn't design their browser to not break random third party tools designed to do unsupported things. Mozilla designs it to be performant for most of its users.

With that being said, your characterization that the larger file size is bigger is not what is causing these tools to not work. Firefox uses a new journal format in SQLite that isn't backwards compatible with older versions. The tool is using an older version.

[Guest]

Maybe not for you specifically, but Mozilla doesn't design things specifically for you. The data is based on a common case.

Why do people on forums always have to be smart alecs ? Anyway if these sizes are truly based upon common case then can you point me to a site with some statistics showing this ?

History, bookmarks, favicons, and what you've typed in the location bar (specific letters that map to what you then visited). The amount of history Firefox keeps has been increased, and what data it stores for both history and bookmarks has increased.

History was stored there in 3.x so that's not new. So it appears that places.sqlite in 4.0 combines some stuff. I still would like to see some evidence since in my usage all of the 3.x sqlite files combined never reached even 2.5MB so yes I consider 10MB by default excessive.

Mozilla doesn't design their browser to not break random third party tools designed to do unsupported things. Mozilla designs it to be performant for most of its users.

Viewing or editing an sqlite database is NOT an unsupported thing.

With that being said, your characterization that the larger file size is bigger is not what is causing these tools to not work. Firefox uses a new journal format in SQLite that isn't backwards compatible with older versions. The tool is using an older version.

That I wasn't aware of, I'll see if the beta version of SQLite Database Browser or some other utility supports the new format though curiously it's only places.sqlite and cookies.sqlite that have a problem, the others sqlite files are handled just fine.

[sdwilsh]

Viewing or editing an sqlite database is NOT an unsupported thing.

As a peer of Firefox's history and bookmarking engine, and the owner of its SQLite wrapper, I can say with 100% certainty that it's not something that Mozilla supports.

That I wasn't aware of, I'll see if the beta version of SQLite Database Browser or some other utility supports the new format though curiously it's only places.sqlite and cookies.sqlite that have a problem, the others sqlite files are handled just fine.

Those are the only two databases that current use the WAL journal file format. The tool needs to use SQLite 3.7.0 or newer to open them.