Reading "jsonlz4" bookmarkbackup files

User Help for Mozilla Firefox
User avatar
dickvl
Posts: 54145
Joined: July 18th, 2005, 3:25 am

Re: Reading "jsonlz4" bookmarkbackup files

Post by dickvl »

You did paste the code in the Browser Console (Firefox/Tools > Web Developer) and not in the Web Console (Firefox/Tools > Web Developer)?
https://developer.mozilla.org/Tools/Browser_Console
equanym
Posts: 6
Joined: September 30th, 2015, 12:00 am

Re: Reading "jsonlz4" bookmarkbackup files

Post by equanym »

Hello there,

I suppose you all hate these non-standard files as much as I do.

I opened a bug report for Mozilla to fix their stuff:
https://bugzilla.mozilla.org/show_bug.cgi?id=1209390

Please come and vote for it!
Last edited by LIMPET235 on September 30th, 2015, 4:42 am, edited 1 time in total.
Reason: Slight rephrasing of the terminology.
ThomasWayne
Posts: 4
Joined: October 16th, 2015, 9:05 am

Re: Willing to pay for help with a .jsonlz4 file!

Post by ThomasWayne »

For what it's worth, we have a Firefox .jsonlz4 backup file that we would very much like to have "decompressed" into a regular .json file so we can use the extensive list of bookmarks. We are NOT interested in learning to use Firefox Web Developer, or run any scripts, or any of the other "solutions" we're finding out there, as none of that is related to what we do in our lives. If there is anyone out there who would be willing to successfully decompress this .jsonlz4 backup file we would be happy to pay a reasonable fee and then NEVER neglect using regular html backups in the future.

Thanks!
TW
kreemoweet
Posts: 778
Joined: December 30th, 2009, 11:25 pm

Re: Reading "jsonlz4" bookmarkbackup files

Post by kreemoweet »

Well, can't Firefox decompress that file? Just create a Firefox profile, put that .jsonlz4 file in the bookmarksbackup folder, and have Firefox "restore" the bookmarks from that file.
You can then export in .html format. Oh, of course you'll have to use a relatively modern version of Firefox. Using FF 3 is just plain weird.
ThomasWayne
Posts: 4
Joined: October 16th, 2015, 9:05 am

Re: Reading "jsonlz4" bookmarkbackup files

Post by ThomasWayne »

kreemoweet wrote:Well, can't Firefox decompress that file? Just create a Firefox profile, put that .jsonlz4 file in the bookmarksbackup folder, and have Firefox "restore" the bookmarks from that file.
You can then export in .html format. Oh, of course you'll have to use a relatively modern version of Firefox. Using FF 3 is just plain weird.


Have not been able to make that work.
User avatar
dickvl
Posts: 54145
Joined: July 18th, 2005, 3:25 am

Re: Reading "jsonlz4" bookmarkbackup files

Post by dickvl »

Only automatically created backups are compressed.
If you create a JSON backup manually then this is an uncompressed file.
Bookmarks > Show All Bookmarks > Import & Backup > Backup

It is not that difficult to open the Browser Console via Ctrl+J and paste code in its command line.
You only first need to toggle devtools.chrome.enabled to true on the about:config page to enable the command line in the Browser Console.
ThomasWayne
Posts: 4
Joined: October 16th, 2015, 9:05 am

Re: Reading "jsonlz4" bookmarkbackup files

Post by ThomasWayne »

dickvl wrote:Only automatically created backups are compressed.
If you create a JSON backup manually then this is an uncompressed file.
Bookmarks > Show All Bookmarks > Import & Backup > Backup

It is not that difficult to open the Browser Console via Ctrl+J and paste code in its command line.
You only first need to toggle devtools.chrome.enabled to true on the about:config page to enable the command line in the Browser Console.


Unfortunately it's not a .json file, it's a .jsonlz4 file. We realize it was an automatically generated backup file, but it is all we have to work with. We actually have several such .jsonlz4 files, all pre-dating the loss of our most current data, but we can live with the loss of a a few weeks of data. failing to decompress one of these files means years of data lost.

What we need, pure and simple, is either some extremely easy-to-understand step-by-step method of decompressing a .jsonlz4 file "for dummies", OR someone who is capable of doing the work whom we can lean upon.
User avatar
dickvl
Posts: 54145
Joined: July 18th, 2005, 3:25 am

Re: Reading "jsonlz4" bookmarkbackup files

Post by dickvl »

Do you know how to inspect and modify a pref on the about:config page?

See:
http://kb.mozillazine.org/about%3Aconfig

The first step is to set devtools.chrome.enabled to true to enable the command line in the Browser Console.
You can open the about:config page via the location/address bar like you open a web page.

You can find the Browser Console in the Web Developer menu (Firefox menu button or Tools > Web Developer).
See https://developer.mozilla.org/Tools/Browser_Console

You can find the code that you need to run in my post on page two of this thread.
viewtopic.php?p=14111285#p14111285

You need to copy the text in the first section of that post to the clipboard: Code: Select All and use Ctrl+C to copy the text to the clipboard.
Paste the code in the command line of the Browser Console and press the Enter key to run the code.
That should give you a file picker window where you can select the compressed .jsonlz4 file.
Open the file to decompress the backup.
You should see a message in the Browser Console that the file is decompressed and saved under a name with a .json file extension.

This should work provided that you use a Firefox version that is compatible with this JSON backup.
If the backup is created in a much older version then you may have to decompress the file in that older version if it fails in the current release.
In that case you can install a Firefox portable version if you do not have a suitable version.
http://portableapps.com/apps/internet/firefox_portable
midas67
New Member
Posts: 1
Joined: October 18th, 2015, 12:28 pm

Re: Reading "jsonlz4" bookmarkbackup files

Post by midas67 »

Any help in decoding the jsonlz4 bookmark file, or perhaps extracting bookmarks from the places.sqlite file would be appreciated. ... In that case you can install a Firefox portable version if you do not have a suitable version. http://portableapps.com/apps/internet/firefox_portable


I have installed a Firefox portable version (37.0), and then copied the "places.sqlite" from my crashing firefox (40.0) to the "profiles" directory of that portable version, where I could then make a manual "sjon"-backup and "html"-backup of all my bookmarks =D> . I also copied "key3.db" and "logins.json" to the portable version, and that also worked fine. Now I can do a totally new install of firefox, because those new updates where all crashing on startup.
ThomasWayne
Posts: 4
Joined: October 16th, 2015, 9:05 am

Re: Reading "jsonlz4" bookmarkbackup files

Post by ThomasWayne »

PROBLEM SOLVED!

Sometimes it's too easy to overlook a simple solution. In the version of Firefox that we have (41.0.2), when we would try to restore the bookmarks we were only given two choices of dated file to try. Both were .json files; one had only the most current list of bookmarks (that was essentially empty), while the second one simply would not open at all. In the "bookmarksbackup" folder we found several earlier .jsonlz4 files, but could not find a way to open them.

It turns out we had overlooked the option of bypassing the offered "backup" files (.json) and instead choosing a jsonlz4 file (we had actually tried a work-around version of this and it had failed). Well, as luck would have it, one of the .jsonlz4 files, when selected manually (rather than from the offered list) opened up and contained all but the last two weeks of data. We immediately backed that file up in several different location, including an external hard drive. We also have set a standard schedule for manually backing up our bookmarks on a weekly basis in the future.

Thanks to every one who tried to help. Those efforts alone gave us enough hope to persevere, allowing us to eventually stumble over the rather simple solution that worked.
RaceQ
Posts: 21
Joined: November 11th, 2009, 2:39 pm

Re: Reading "jsonlz4" bookmarkbackup files

Post by RaceQ »

I have had problem with missing bookmarks due to a power failure and sync services that create problems, so I want to be able to do some sort of snapshot compares to see what recent changes were made. I expect the following to be simple tasks to recover stuff from a backup:
  • Be able to look at a simple plain text file view (which I am willing to create if I know how) of bookmark backups files to see what is in it when I need to salvage.
  • Be able to compare changes between two bookmark backups
It seems those objectives are now much more difficult because now another layer of "home made" compression (jsonlz4) has been slapped on the automatic bookmark backups.

So far here are the steps I need to do:
  1. Convert bookmarkbackup1.jsonlz4 to bookmarkbackup1.json
    A:use browser console + code posted earlier in this thread
  2. Convert bookmarkbackup1.json to bookmarkbackup1.txt
    This file will have one bookmark per line (like a csv file), converted internal dates to human readable format (desirable)
    A: unknown
  3. Compare versions of bookmarkbackup1.txt files to see what changed
    A:This will use suitable windows tools that can read plain text files e.g. Windiff, Examdiff, Total Commander
I want to make a command script that will do step 1 and 2.

Can step 1 be converted to a routine I can call from a batch command?

Anyone know how to do step 2 or have any suggestions?

BTW backup/restore in Firefox - I tried to restore 8700 entries from a jsonlz4 file and it fails, so that is another reason I want to look at a plain text version to see if anything looks "corrupt". If it had worked I would have to dump to html to compare two versions of the data from jsonlz4 files. That would be tedious if you wanted to look at all 15 jsonlz4 backup files. This was much simpler when backups were made as html format but now I have 15 jsonlz4 backup files to deal with.
kreemoweet
Posts: 778
Joined: December 30th, 2009, 11:25 pm

Re: Reading "jsonlz4" bookmarkbackup files

Post by kreemoweet »

@RaceQ: Are you aware you can still set FF to backup your bookmarks in .html form? No need to mess with those wacky .jsonlz4 files!
The pref is "browser.bookmarks.autoExportHTML".
User avatar
dickvl
Posts: 54145
Joined: July 18th, 2005, 3:25 am

Re: Reading "jsonlz4" bookmarkbackup files

Post by dickvl »

Easiest is to restore a JSON backup and let Firefox do the first part.
You can do that in a separate profile if you like.
Then you can export the bookmarks to an HTML file in case you want to do the comparison with other software.
User avatar
therube
Posts: 21703
Joined: March 10th, 2004, 9:59 pm
Location: Maryland USA

Re: Reading "jsonlz4" bookmarkbackup files

Post by therube »

@RaceQ

2A.

You pretty much have to figure out the format of the json, to understand what is where & why or why not you might or might not want that data, much less knowing what "corrupt" data might look like.

Anyhow, with something like: %s/"http/\rhttp/g would give you something like this:
(Vim to do it manually, or you could set something up with sed or similar to use in a batch file.)

Code: Select all

{"guid":"root________","title":"","index":0,"dateAdded":1452445690196000,"lastModified":1453866096586000,"id":1,"type":"text/x-moz-place-container","root":"placesRoot","children":[{"guid":"menu________","title":"Bookmarks Menu","index":0,"dateAdded":1452445690196000,"lastModified":1452445693702000,"id":2,"type":"text/x-moz-place-container","root":"bookmarksMenuFolder","children":[{"guid":"2SSYPVUHZ7wt","title":"Recently Bookmarked","index":0,"dateAdded":1452445692433000,"lastModified":1452445693188000,"id":13,"annos":[{"name":"Places/SmartBookmark","flags":0,"expires":4,"value":"RecentlyBookmarked"}],"type":"text/x-moz-place","uri":"place:folder=BOOKMARKS_MENU&folder=UNFILED_BOOKMARKS&folder=TOOLBAR&queryType=1&sort=12&maxResults=10&excludeQueries=1"},{"guid":"KKr-YGZT1zP9","title":"Recent Tags","index":1,"dateAdded":1452445693189000,"lastModified":1452445693691000,"id":14,"annos":[{"name":"Places/SmartBookmark","flags":0,"expires":4,"value":"RecentTags"}],"type":"text/x-moz-place","uri":"place:type=6&sort=14&maxResults=10"},{"guid":"Zh4p0mSLDjqb","index":2,"dateAdded":1452445693702000,"lastModified":1452445693702000,"id":15,"type":"text/x-moz-place-separator"},{"guid":"nPEVC1lEAZC1","title":"Mozilla Firefox","index":3,"dateAdded":1452445691735000,"lastModified":1452445691737000,"id":7,"type":"text/x-moz-place-container","children":[{"guid":"4N1FM-lOXgUN","title":"Help and Tutorials","index":0,"dateAdded":1452445691735000,"lastModified":1452445691735000,"id":8,"iconuri":
http://www.mozilla.org/2005/made-up-favicon/1-1452445691736","type":"text/x-moz-place","uri":
https://www.mozilla.org/en-US/firefox/help/"},{"guid":"ImJVjeyY595o","title":"Customize Firefox","index":1,"dateAdded":1452445691736000,"lastModified":1452445691736000,"id":9,"iconuri":
http://www.mozilla.org/2005/made-up-favicon/2-1452445691737","type":"text/x-moz-place","uri":
https://www.mozilla.org/en-US/firefox/customize/"},{"guid":"SYNan28nsjsv","title":"Get Involved","index":2,"dateAdded":1452445691737000,"lastModified":1452445691737000,"id":10,"iconuri":
http://www.mozilla.org/2005/made-up-favicon/3-1452445691738","type":"text/x-moz-place","uri":
https://www.mozilla.org/en-US/contribute/"},{"guid":"bORpcs1VivtR","title":"About Us","index":3,"dateAdded":1452445691737000,"lastModified":1452445691738000,"id":11,"iconuri":
http://www.mozilla.org/2005/made-up-favicon/4-1452445691738","type":"text/x-moz-place","uri":
https://www.mozilla.org/en-US/about/"}]}]},{"guid":"toolbar_____","title":"Bookmarks Toolbar","index":1,"dateAdded":1452445690196000,"lastModified":1452445691875000,"id":3,"annos":[{"name":"bookmarkProperties/description","flags":0,"expires":4,"value":"Add bookmarks to this folder to see them displayed on the Bookmarks Toolbar"}],"type":"text/x-moz-place-container","root":"toolbarFolder","children":[{"guid":"_LRv3NwjtOvy","title":"Most Visited","index":0,"dateAdded":1452445691875000,"lastModified":1452445692406000,"id":12,"annos":[{"name":"Places/SmartBookmark","flags":0,"expires":4,"value":"MostVisited"}],"type":"text/x-moz-place","uri":"place:sort=8&maxResults=10"},{"guid":"LV8oDjIqPCtc","title":"Getting Started","index":1,"dateAdded":1452445691733000,"lastModified":1452445691734000,"id":6,"iconuri":
http://www.mozilla.org/2005/made-up-favicon/0-1452445691734","type":"text/x-moz-place","uri":
https://www.mozilla.org/en-US/firefox/central/"}]},{"guid":"unfiled_____","title":"Unsorted Bookmarks","index":3,"dateAdded":1452445690196000,"lastModified":1452445691731000,"id":5,"type":"text/x-moz-place-container","root":"unfiledBookmarksFolder"}]}
Not all that pretty, but at least you can feed that to your diff programs.

See this thread, http://forums.mozillazine.org/viewtopic ... p=12095933, for other thoughts.

Nirsoft's, FavoritesView might be of benefit?
Or maybe an extension on AMO?


(Above is going to look kind of ugly cause it wraps, so %s/".*//g leaves ...)

Code: Select all

{
http://www.mozilla.org/2005/made-up-favicon/1-1452445691736
https://www.mozilla.org/en-US/firefox/help/
http://www.mozilla.org/2005/made-up-favicon/2-1452445691737
https://www.mozilla.org/en-US/firefox/customize/
http://www.mozilla.org/2005/made-up-favicon/3-1452445691738
https://www.mozilla.org/en-US/contribute/
http://www.mozilla.org/2005/made-up-favicon/4-1452445691738
https://www.mozilla.org/en-US/about/
http://www.mozilla.org/2005/made-up-favicon/0-1452445691734
https://www.mozilla.org/en-US/firefox/central/
Fire 750, bring back 250.
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.19) Gecko/20110420 SeaMonkey/2.0.14 Pinball CopyURL+ FetchTextURL FlashGot NoScript
antofthy
Posts: 14
Joined: April 12th, 2012, 5:11 pm

Re: Reading "jsonlz4" bookmarkbackup files

Post by antofthy »

There are many solutions to json being all on one line.
Actually unless in quotes, white space is ment to be ignored in the json format so adding returns *should* be valid. The question is does FireFox thinks so?

one method is to use some type of json re-formating program such as yajl (Yet Another JSON Library)

Code: Select all

  json_reformat < file.json
or the old python method

Code: Select all

   python -c'from simplejson.tool import main; main()'
or perl JSON module

Code: Select all

   perl -0007 -MJSON -nE'say to_json(from_json($_), {pretty=>1})'
For more of this look at my RAW notes in
http://www.ict.griffith.edu.au/anthony/ ... a/json.txt

The problem with that however is each data record ends up on a line by itself, making it tricky to re-parse it
what you really want is to process it a record group at a time.

How tricky that gets depends on if you need to preserve the data 'tree'.

One technique I have used for decoding firefox json, is to just specify that '{' is the record separator in perl.

Code: Select all

    $/='{';    # Record separator (start character)
that produces mostly one record per read, allowing simple parsing of that record into various parts.
For example extracting: title, type, uri, and with some care even tracking tree depth and structure, without needing full JSON decoding.

Here is relevant parts of the perl code (its ugly but it worked)...

Code: Select all

my $level = 0;   # level at this point.

open(JSON, $json_file) || die;
$/='{';                       # Record separator (start character)

# Locate the start of my bookmarks
while( <JSON> ) {
  last if /"type":"text\/x-moz-place-separator"/;
}

# Process each book mark entry
while( <JSON> ) {
  if ( ! /"type"/  ) {  # the entry is incomplete - get more
    $_ .= <JSON>;  redo;
  }
  my ( $title ) = /"title":"(.*?[^\\])"/;          # Title of this entry
  my ( $type ) = /"type":"text\/x-moz-place-?([^"]*)"/; # type of entry
  my ( $uri ) = /"uri":"(.*?[^\\])"/;              # any URI present?

  next if $uri =~ /^place:/;  # ignore special ones

  $title = '' unless defined $title;
  $type = 'UNDEFINED' unless defined $type;
  $title = '-'x20 if $type eq "separator";

  # Process this bookmark
  print STDERR "  "x$level, "\"$title\"\n"   if $DEBUG;

  # do what you what with the record here...

  # handle container levels
  if ( $type eq "container" ) {
    $level++;  # following records are in a folder
  }
if ( my (@count) = /]}/g ) {
    for ( @count ) {
      $level--;   # end of a folder
      print STDERR "  "x$level, "<<< ($level)\n" if $DEBUG;
    }
    if ( $level < 0 ) {
      print STDERR "Level Became Negative -- ABORTING\n" if $DEBUG>=0;
      last;
    }
  }
}
close JSON;
$/="\n";

That did not work for Chrome Browser JSON, as unlike FireFox, its JSON code places relevant data AFTER child container (folders), making linear parsing (as per firefox) impossible! For that I had to use the JSON perl module with a recursive data structure to walk the tree to process the data. Arrrggghhhhh....

See the link to my JSON data handling above.


HOWEVER. this topic is actually ment to be on the Stupid move by Mozillia to use a non-standard lz4 compression, destroying simple external parsing of the JSON data structure! That move was one of the main reasons I moved from FireFox to Chrome!
Last edited by antofthy on January 27th, 2016, 6:20 pm, edited 1 time in total.
Post Reply