Reading "jsonlz4" bookmarkbackup files

[antofthy]

For years I have been automatically parsing the latest "json" bookmark backup file to generate a 'hotlist' of my bookmarks so I can access them from anywhere in the world, or refer people to them.

I used the bookmarkbackup for a number of reasons.
1/ it was statis and would not chnage halfway through
2/ it was a lot easier to parse json tahn trying to extract sqlite records.
3/ it was clear text, so it was easy to see what was going on and fix/tweek things as needed.

More importantally this can be done automatically without manual exporting, overnight, and without requiring firefox running! Generally because I am logged out. (It is a linux system, not windows).

Now firefox started to compress the json file to a jsonlz4 format, and of course my bookmark extraction scripts no longer work.

The new format is suposed to be based on the very fast lz4 compression, but I have been unable to find information on exactly how it is compressed, or better still how I can decode and extract the bookmarks (outside firefox).

Attempting to simple run the files thru 'lz4' does not work as the files have a different header (file magic, and what I believe is a checksum hash). Attempts to remove that header and then run the file through lz4 have also failed.

Google searches on jsonlz4 and also failed to turn up any useful information.

Any help in decoding the jsonlz4 bookmark file, or perhaps extracting bookmarks from the places.sqlite file would be appreciated.

NOTE: I am not a java or javascript programmer. My expertise (more than 30 years) is in UNIX, shell scripting and perl scripting. Text processing to me is simple, processing constantly changing binary formats is not my forte.

[jarri]

You probably already know this but Firefox can be told to make an html export frequently - I am not sure if it is once per day, or every time it closes. http://kb.mozillazine.org/Browser.bookm ... ExportHTML

[dfoulkes]

I see this extension over in AMO ... from it's title it seems to have something to do with those new files... the author did not elaborate at all ... so, I have no idea if it even relates... maybe you can figure it out.
ExtJS 4 :: Add-ons for Firefox

[L.A.R. Grizzly]

You probably already know this but Firefox can be told to make an html export frequently - I am not sure if it is once per day, or every time it closes. http://kb.mozillazine.org/Browser.bookm ... ExportHTML

Every time it closes.

[antofthy]

Thank you jarri, and L.A.R.Grizzly. I pulled out a old script that handled the original netscape bookmarks.html file and after a little dusting off of the code (Whcih believe it or not is a 'ed' script) worked fine. I can convert the bookmarks.html file that firefox now automatically generates into the format I need.

Thank you for the tip.. Wish I had known about the "browser.bookmarks.autoExportHTML" setting when firefox version 3 first came out, it would have saved me a lot of hassles.

NOTE: I cannot use that HTML file directly for my online version of my bookmarks as there are bookmarks I would not care to publish generally. Also I do want to reformat the HTML and remove a other information too.

Still it would be good to know of other posible solutions, such as actually decoding those .jsonlz4 files, or posibly ways of extracting information from the "places.sqlite" file. Examples or pointers would be appreciated.

[therube]

> the files have a different header

> Error 44 : Unrecognized header : file cannot be decoded

Heh. Like is that to be unexpected.
They use ZIP (.jar, .ja), and while it might follow the rules, its still not "kosher".

So would you expect anything different with lz4.
(Now that may not be the case, it could be something with the [Windows binary, in this case] LZ4, but given the track record...)


(Now morat or someone similar might be able to come up with a bit of code to plug in to JavaScript Debugger, or similar, to spit something out?)


(And then you have a 10 MB, or often greater, places.sqlite out there, & they're worried about compressing a (in most cases relatively) small .json, to gain what? [Perhaps some obscurity.])

[therube]

Note that manually exported .json will not be compressed.
So if you manually export, then work with that file...

"It's somewhat unfortunate that we ended up with non standard format which users can't examine without loading it into a compatible Firefox version (and even then not examine directly, but rather just as much as the details are exposed through the bookmarks system)."
https://bugzilla.mozilla.org/show_bug.cgi?id=818587

[antofthy]

> Error 44 : Unrecognized header : file cannot be decoded
So would you expect anything different with lz4.
(Now that may not be the case, it could be something with the [Windows binary, in this case] LZ4, but given the track record...)
...
(And then you have a 10 MB, or often greater, places.sqlite out there, & they're worried about compressing a (in most cases relatively) small .json, to gain what? [Perhaps some obscurity.])

I would have expected them to... keep it simple. Obfuscating bookmarks makes no sense, and making it more complex only makes it harder future programmers, and for the development of third party tools. If it isn't for security reasons, KISS. And even if it is, it should not rely on obscurity, which is proven to be a bad security practice.

Also I am a Linux user, not windows! I did mention it!

And I agree compressing these also does not make a lot of sense, though there is a lot of room for compression and they do save 15 copies of the file. But then until recently they had a 100 Mb urlclassifier3.sqlite file listing 'dodgy web sites'.

Now the addition of checksum hashs to the title does make some sense, though this could have easilly hvae been included inside the JSON structure, without too big an impact. It is a very versitile data format that allowas for 'extra elements' to be added without breaking existing usage.

Note that manually exported .json will not be compressed.
So if you manually export, then work with that file...

Re-read the OP!!!! This was for a totally automatic system, at a time when firefox was NOT running!
Manual Export was NOT an option.

However you do seem like a person in the know. So prehaps you would like to re-read the problem and think about some other posible solution.

Again... No JavaScript! Though a downloadable JavaScript tool (not a API library) is accaptible. Perhaps the sqlite command can be used somehow to extract bookmarks into a more readable textual form. I have no idea what is posible, which is why I originally posted the problem on the forum.

[TheVisitor]

I've come across this site that seems to be written by one of the dev's of the new file format. Perhaps something here will be of interest:

http://fastcompression.blogspot.fr/p/lz4.html I could not get the 'command line utility' to work, but perhaps since it was command line I was leaving out something.

some info here also:
http://fastcompression.blogspot.fr/2013 ... final.html

Should be noted that I think Mozilla took parts of this and are using their own tweaks to the new file format as noted by 'therube' above.

[antofthy]

I did a search on the lz4 forum for "jsonlz4" nothing. Looks like it is internal to Mozilla. Non-standard use of the lz4 algorithm! What a Pain! Give me plain text any time... that was after all one of the main design points of the 'json' data format, a well designed plain text data format for structured data.

Small update. Some comments on Bugzilla@Mozilla has appead on this issue.
https://bugzilla.mozilla.org/show_bug.cgi?id=818587

I am also sorry to say I may have gone a little 'off my rocker' about one of the comments made by a developer, though you can't really go back and edit your comments.

[therube]

You said nothing wrong.
Sometimes you have to bang them on the head to wake them up a bit .

I said similar here, https://bugzilla.mozilla.org/show_bug.cgi?id=674824#c2.

[ipohraj]

Dear Forum and Members,

Actually i am using windows 8.1 and suddenly system shutdown due to bluescreen problem.
with my wrong attempt, then totally my system fail to start and finally collect my firebox bookmarks from harddisk.
my firebox bookmarks found it harddisk from harddisk in the jsonlz4 format.
i am trying to open it, but every try is getting failure and also can not find any solution even in google search ideas.
please anyone help me to get back my bookmarks in the way of open and read my jsonlz4 format files?
rest all in best.. Raj Malaysia

[ipohraj]

KIND ATTENTION: URGENT- PLEASE ANYONE HELP ME TO SOLVE MY PROBLEM..

Dear Forum and Members,

Actually i am using windows 8.1 and suddenly system shutdown due to bluescreen problem.
with my wrong attempt, then totally my system fail to start and finally collect my firebox bookmarks from harddisk.
my firebox bookmarks found it from harddisk in jsonlz4 format.
i am trying to open it, but every try is getting failure and also can not find any solution even in google search ideas.
please anyone help me to get back my bookmarks in the way of open and read my jsonlz4 format files?
rest all in best.. Raj Malaysia

[malliz]

Don't yell we are volunteers and don't need the abuse a polite "bump" would have done just as well. Did you try and import the file into firefox using Bookmarks - Show all bookmarks - import and backup - restore - choose file

[L.A.R. Grizzly]

KIND ATTENTION: URGENT- PLEASE ANYONE HELP ME TO SOLVE MY PROBLEM..

Dear Forum and Members,

Actually i am using windows 8.1 and suddenly system shutdown due to bluescreen problem.
with my wrong attempt, then totally my system fail to start and finally collect my firebox bookmarks from harddisk.
my firebox bookmarks found it from harddisk in jsonlz4 format.
i am trying to open it, but every try is getting failure and also can not find any solution even in google search ideas.
please anyone help me to get back my bookmarks in the way of open and read my jsonlz4 format files?
rest all in best.. Raj Malaysia

If you have a backup of places.sqlite, just copy that backup file into your existing profile. This is the actual bookmark file.