huge places.sqlite

[shoofy]

Somehow after removing all the favicons and compacting, the places.sqlite file got bigger. I have no way of explaining that at all, but for the moment I'm going to move away from the disabling favicons path, since there's no indication that that's what's causing the problem.

[alterna]

Instead of working directly with places.sqlite, did you try my procedure, creating a new places.sqlite?

By the way, it's a good idea to remove the bookmark backups .jsons before proceeding.
I might also recommend removing places.sqlite, but that would remove your history.

[shoofy]

Again, I would only want to do that as a last resort, since there is no good way of exporting and importing the history and I would lose so much if I had to rebuild that from scratch. I'm actually surprised that there is no way to import and export history built in, and even more surprised that nobody has made an extension to do it. I'm kind of thinking now that I might want to try making that extension myself. I would have to do some more research into the places schema first, though.

[shoofy]

I'm starting to wonder whether the database itself is just corrupt. Although I can open it and modify it from Sqlite Browser, if I use Sqlite Browser to dump it to an sql file and then import that into a new sqlite database the import fails, and when I try to run

Code: Select all

sqlite places.sqlite

from the command line it says "Unable to open database "places.sqlite": file is encrypted or is not a database." Also, the dumped SQL code is only 85MB, and I can't think of any reason why the SQL code should be smaller than the database itself.

[alterna]

There was an excellent add-on by the excellent White Alice 0775 at AMO, Backup Restore History 2.0, that saved and imported history as a .json file. I've used it in 3.0, and it works fine.

I found a mac version hosted at Softpedia, but cannot find it for Windows.

Perhaps you could contact the author.
http://space.geocities.jp/alice0775/

[shoofy]

The version at Softpedia seems to actually be platform independent, so I was able to install it and it seems to do what I need. I'll try rebuilding my places.sqlite from scratch tomorrow and report back on what happens.

[shoofy]

Well the Backup Restore History add-on always seems to die after restoring 805 history items. My next plan of attack was to try exporting each table from the places.sqlite database and importing them into a fresh places.sqlite. When I did this, I noticed that the exported moz_places table was 75MB and everything else was a pretty reasonable size. I decided to take a look at what was inside it and I noticed that an awful lot of the entries were from Google News. A quick grep tells me that 260171 out of 305786 entries are Google News. Now, granted I do look at Google News at least once a day, but this is a little ridiculous, and most of the articles I have no memory of reading whatsoever. I'm wondering whether something has gone haywire with Google News prefetching. I'm going to try removing all the Google News entries and see where that leaves me.

[shoofy]

After removing all the Google News entries and then compacting my places.sqlite is now down to 19MB and Firefox is feeling much zippier. I'm not sure what's still taking up 19MB, but I think I'll leave well enough alone for now, unless the problem comes back. For anyone who cares, the SQL code I used to remove the Google News entries was:

Code: Select all

DELETE FROM moz_places WHERE hidden=1 AND url LIKE 'http://news.google.com/news/url%'

This only removes entries that are hidden, which apparently means that they don't appear in either history or bookmarks. I'm not sure what reason there is to ever have these entries around, but it seems safe to remove them.

Thanks to everyone for all your suggestions and hopefully this problem is finished with.

[shoofy]

I think the problem may have actually arisen from my "live bookmarks" which I haven't actually used in a couple years since I started using Google Reader. My guess is that the Google News RSS feed updates very regularly and changes completely with every update, which causes a ton of sites to be added to places. I think I may file a bug on that so that the Mozilla folks might rethink how they handle Live Bookmarks.

EDIT: https://bugzilla.mozilla.org/show_bug.cgi?id=469723

[need2knw]

YAY!! something helpful

not quite at 170mb; at this rate, predictably i'll have problems in just under 5 years, and everyone else who started storing all their history

my bookmarks don't have favicons for some reason (i must've done something goodness knows how long ago).. which helps

i come from here:
viewtopic.php?f=38&t=974715&p=5228605&sid=5d058057f7bfc2d12ea435a8661863fe#p5228605

[shoofy]

Just an update on my situation: turns out that Firefox removes Live Bookmarks from Places immediately after they stop being listed in one of the feeds, which is why nobody has seen this problem. I tried running in Safe Mode and the file size didn't grow at all, which suggests that the problem is with one of my extensions. I haven't had a chance to figure out which one it is yet, but hopefully I'll have time to investigate this week.

[crumblin]

If you have prefetch active which loads links into memory for faster browsing don't they all get stored in places.sqlite as well. I disabled prefetch and my places.sqlite shrank and is now stable at 15MB, maybe its just a coincidence.

[totedati]

I'm starting to wonder whether the database itself is just corrupt. Although I can open it and modify it from Sqlite Browser, if I use Sqlite Browser to dump it to an sql file and then import that into a new sqlite database the import fails, and when I try to run

Code: Select all

sqlite places.sqlite

from the command line it says "Unable to open database "places.sqlite": file is encrypted or is not a database." Also, the dumped SQL code is only 85MB, and I can't think of any reason why the SQL code should be smaller than the database itself.

you command is wrong ...

Code: Select all

sqlite3 places.sqlite

is the correct command code ...

[RobertJ]

My places file is 12M.

I have 516 bookmarks, 945 favicons and 20,300 history entries. BTW favicons typically are about 300 to 600 bytes although I have seen a few as high as 1,100 bytes.

So in my case assuming an average of 500 bytes per favicon the storage for the favicons would be about 500K bytes.

[LoudNoise]

Over six months old. Locking