Why are .sqlite files so much larger in Firefox 4 ?
-
- Guest
Why are .sqlite files so much larger in Firefox 4 ?
With a new profile places.sqlite is 10MB and cookies.sqlite is 512KB, that's a rather big difference from Firefox 3.x, what gives ? Also is there a way to reduce the sizes ?
- Daifne
- Moderator
- Posts: 123071
- Joined: July 31st, 2005, 9:17 pm
- Location: Where the Waters Meet, Wisconsin
Re: Why are .sqlite files so much larger in Firefox 4 ?
Problems with the test builds need to be discussed in the Firefox Builds forum. Please register and post back here. I will then move this for you.
-
- Guest
Re: Why are .sqlite files so much larger in Firefox 4 ?
Daifne wrote:Problems with the test builds need to be discussed in the Firefox Builds forum. Please register and post back here. I will then move this for you.
I'm using 4.0 RC1 which will likely become 4.0.0 so is this really considered a test build like the betas ? Anyway it's not an actual problem per se though it is curious as to why there's such a huge jump in file sizes.
-
- Posts: 563
- Joined: November 6th, 2005, 9:46 pm
- Location: California
Re: Why are .sqlite files so much larger in Firefox 4 ?
These files will grow to roughly these sizes in normal use, so they start out that big to reduce fragmentation issues, which end up making startup slower.
Problem Solver
-
- Guest
Re: Why are .sqlite files so much larger in Firefox 4 ?
sdwilsh wrote:These files will grow to roughly these sizes in normal use, so they start out that big to reduce fragmentation issues, which end up making startup slower.
I've never had those files grow to these sizes and I regularly defrag my system. Is there a way to configure the initial sizes to something more reasonable ?
- Tony-E
- Posts: 8778
- Joined: November 5th, 2004, 11:28 am
Re: Why are .sqlite files so much larger in Firefox 4 ?
In Firefox 4 the places.sqlite file will increase in size in 10MB blocks, this is to avoid file fragmentation. As soon as the file uses the 10MB it will automatically be increased to 20MB. The cookies.sqlite file grows in 512KB increments.
Bug 581606 – Avoid sqlite fragmentation via SQLITE_FCNTL_CHUNK_SIZE
Bug 581606 – Avoid sqlite fragmentation via SQLITE_FCNTL_CHUNK_SIZE
-
- Guest
Re: Why are .sqlite files so much larger in Firefox 4 ?
Tony-E wrote:In Firefox 4 the places.sqlite file will increase in size in 10MB blocks, this is to avoid file fragmentation. As soon as the file uses the 10MB it will automatically be increased to 20MB. The cookies.sqlite file grows in 512KB increments.
Bug 581606 – Avoid sqlite fragmentation via SQLITE_FCNTL_CHUNK_SIZE
ARGH ! Don't people defrag their systems ? This is wasting disk space unnecessarily and to top it off SQLite Database Browser can't read the padded files.
-
- Posts: 563
- Joined: November 6th, 2005, 9:46 pm
- Location: California
Re: Why are .sqlite files so much larger in Firefox 4 ?
Guest wrote:I've never had those files grow to these sizes and I regularly defrag my system. Is there a way to configure the initial sizes to something more reasonable ?
No. The amount of data that is stored in places.sqlite has also increased for Firefox 4.0, so while you may not have hit that in the past, you are more likely to with Firefox 4.0.
Problem Solver
-
- Posts: 8829
- Joined: May 7th, 2006, 10:29 pm
- Location: California
Re: Why are .sqlite files so much larger in Firefox 4 ?
There's a difference between file system fragmentation and SQLite database fragmentation.
-
- Posts: 8779
- Joined: May 7th, 2007, 12:07 pm
Re: Why are .sqlite files so much larger in Firefox 4 ?
That much space seems extremely excessive to me. I have been doing some browsing, and my cookies.sqlite file size is 8 KB, while my places.sqlite file is 596 KB. I do, after all, default to blocking cookies, and I block favicons, and I do not keep months of history.
- malliz
- Folder@Home
- Posts: 43796
- Joined: December 7th, 2002, 4:34 am
- Location: Australia
Re: Why are .sqlite files so much larger in Firefox 4 ?
Excessive? For petes sake how big is your HD?
What sort of man would put a known criminal in charge of a major branch of government? Apart from, say, the average voter.
"Terry Pratchett"
"Terry Pratchett"
-
- Guest
Re: Why are .sqlite files so much larger in Firefox 4 ?
sdwilsh wrote:Guest wrote:I've never had those files grow to these sizes and I regularly defrag my system. Is there a way to configure the initial sizes to something more reasonable ?
No. The amount of data that is stored in places.sqlite has also increased for Firefox 4.0, so while you may not have hit that in the past, you are more likely to with Firefox 4.0.
I seriously doubt that, these bundle sizes are excessive.
cookies.sqlite: 512KB - in 3.x my file rarely surpasses 16KB; I reject third-party cookies, have some filtered out via permissions and periodically delete crap cookies
places.sqlite: 10MB - in 3.x my files rarely surpasses 2MB; I'm aware that bookmarks are now stored here in 4.0 but that's not going to make a significant difference. What else is stored here that will ?
Also much like with omni.jar this change introduces an incompatibility with other utilities, SQLite Database Browser http://sqlitebrowser.sourceforge.net doesn't like the padded files and can't process them correctly.
-
- Posts: 563
- Joined: November 6th, 2005, 9:46 pm
- Location: California
Re: Why are .sqlite files so much larger in Firefox 4 ?
Guest wrote:I seriously doubt that, these bundle sizes are excessive.
Maybe not for you specifically, but Mozilla doesn't design things specifically for you. The data is based on a common case.
Guest wrote:cookies.sqlite: 512KB - in 3.x my file rarely surpasses 16KB; I reject third-party cookies, have some filtered out via permissions and periodically delete crap cookies
The maximum number of cookies (per host and global) limits have changed in 4.0.
Guest wrote:places.sqlite: 10MB - in 3.x my files rarely surpasses 2MB; I'm aware that bookmarks are now stored here in 4.0 but that's not going to make a significant difference. What else is stored here that will ?
History, bookmarks, favicons, and what you've typed in the location bar (specific letters that map to what you then visited). The amount of history Firefox keeps has been increased, and what data it stores for both history and bookmarks has increased.
Guest wrote:Also much like with omni.jar this change introduces an incompatibility with other utilities, SQLite Database Browser http://sqlitebrowser.sourceforge.net doesn't like the padded files and can't process them correctly.
Mozilla doesn't design their browser to not break random third party tools designed to do unsupported things. Mozilla designs it to be performant for most of its users.
With that being said, your characterization that the larger file size is bigger is not what is causing these tools to not work. Firefox uses a new journal format in SQLite that isn't backwards compatible with older versions. The tool is using an older version.
Problem Solver
-
- Guest
Re: Why are .sqlite files so much larger in Firefox 4 ?
sdwilsh wrote:Maybe not for you specifically, but Mozilla doesn't design things specifically for you. The data is based on a common case.
Why do people on forums always have to be smart alecs ? Anyway if these sizes are truly based upon common case then can you point me to a site with some statistics showing this ?
History, bookmarks, favicons, and what you've typed in the location bar (specific letters that map to what you then visited). The amount of history Firefox keeps has been increased, and what data it stores for both history and bookmarks has increased.
History was stored there in 3.x so that's not new. So it appears that places.sqlite in 4.0 combines some stuff. I still would like to see some evidence since in my usage all of the 3.x sqlite files combined never reached even 2.5MB so yes I consider 10MB by default excessive.
Mozilla doesn't design their browser to not break random third party tools designed to do unsupported things. Mozilla designs it to be performant for most of its users.
Viewing or editing an sqlite database is NOT an unsupported thing.
With that being said, your characterization that the larger file size is bigger is not what is causing these tools to not work. Firefox uses a new journal format in SQLite that isn't backwards compatible with older versions. The tool is using an older version.
That I wasn't aware of, I'll see if the beta version of SQLite Database Browser or some other utility supports the new format though curiously it's only places.sqlite and cookies.sqlite that have a problem, the others sqlite files are handled just fine.
-
- Posts: 563
- Joined: November 6th, 2005, 9:46 pm
- Location: California
Re: Why are .sqlite files so much larger in Firefox 4 ?
Guest wrote:Viewing or editing an sqlite database is NOT an unsupported thing.
As a peer of Firefox's history and bookmarking engine, and the owner of its SQLite wrapper, I can say with 100% certainty that it's not something that Mozilla supports.
Guest wrote:That I wasn't aware of, I'll see if the beta version of SQLite Database Browser or some other utility supports the new format though curiously it's only places.sqlite and cookies.sqlite that have a problem, the others sqlite files are handled just fine.
Those are the only two databases that current use the WAL journal file format. The tool needs to use SQLite 3.7.0 or newer to open them.
Problem Solver