[Ext] XML Search Engines Exporter/Importer (+ Fx57 tools)

Announce and Discuss the Latest Theme and Extension Releases.
Post Reply
nohamelin
Posts: 96
Joined: September 3rd, 2013, 4:04 pm
Location: Chile

[Ext] XML Search Engines Exporter/Importer (+ Fx57 tools)

Post by nohamelin »

XML Search Engines Exporter/Importer (XSEEI) in: AMO | GitHub | Crowdin (translations)

Compatibility: Firefox 45-56
License: Mozilla Public License (MPL) 2.0

It lets you to import and export search engines from local XML files in the OpenSearch format.

Note it's a legacy add-on, with a WebExtensions-based build being currently not possible at all.


/NEW/ Tools for Fx57+ (and SeaMonkey, by the way): Export tool | Import tool


Thread open to general discussion and user support.
Last edited by nohamelin on November 11th, 2017, 7:26 am, edited 8 times in total.
User avatar
Foxinabox
Posts: 166
Joined: November 10th, 2014, 8:29 pm

Re: [Ext] XML Search Engines Exporter/Importer

Post by Foxinabox »

This looks like a great idea.

You might find it beneficial to put it on GitHub so there is an organized place for people to submit issues and review the code.
nohamelin
Posts: 96
Joined: September 3rd, 2013, 4:04 pm
Location: Chile

Re: [Ext] XML Search Engines Exporter/Importer

Post by nohamelin »

The GitHub repository is up:
https://github.com/nohamelin/xseei
User avatar
streetwolf
Posts: 2700
Joined: August 21st, 2011, 8:07 am
Location: NJ (USA)

Re: [Ext] XML Search Engines Exporter/Importer

Post by streetwolf »

Any plans to handle search.json.mozlz4 ?
Intel i9-13900K | ASUS ROG MAXIMUS Z790 HERO DDR5 | 64GB CORSAIR VENGEANCE DDR5 @ 6400 Mhz.
H100i ELITE CAPELLIX XT Liquid CPU Cooler | PNY 12GB GeForce RTX 3080 Ti | 2 CORSAIR 2TB MP600 PRO XT GEN 4
HX1200 PLATINUM PSU | XENEON 32" IPS UHD 144Hz | BenQ 32" UHD | MS Windows 11 Pro
nohamelin
Posts: 96
Joined: September 3rd, 2013, 4:04 pm
Location: Chile

Re: [Ext] XML Search Engines Exporter/Importer

Post by nohamelin »

streetwolf wrote:Any plans to handle search.json.mozlz4 ?
in what sense? Maybe offering an editor for the contents of that file, as other tools lets it? I have nothing planned as that, as I feel that it is, though directly related, a bit out of scope too.
nohamelin
Posts: 96
Joined: September 3rd, 2013, 4:04 pm
Location: Chile

Re: [Ext] XML Search Engines Exporter/Importer

Post by nohamelin »

I commented some things in the tracker about the feasibility to port this extension to the WebExtensions model. With the current APIs it's not possible.
nohamelin
Posts: 96
Joined: September 3rd, 2013, 4:04 pm
Location: Chile

Re: [Ext] XML Search Engines Exporter/Importer

Post by nohamelin »

Firefox 57 will hit the release channel in less than a month, and there isn't any visible upstream development about WebExtensions APIs to interact with the installed user search engines. Given that it recently has been confirmed that the next ESR release (59.0) will not let to run legacy add-ons either, I don't have more reasons to keep working in the add-on.

The extension still works with current Nightly and DeveloperEdition builds (if the option to run legacy add-ons is forced), and I think it will be so for a while it's not. I probably will do a last release with some very minor changes accumulated since the previous release. I will not do any port for still-XUL-based Fx-derived projects as Waterfox or Pale Moon, as I don't see myself using these browsers for my own personal consumption.

If in the future Mozilla manages to make available a WE API to support any of the two main functions of the add-on (export or import) in a satisfactory way, I will write a WE-based update for the add-on. Though, maybe by then legacy add-ons already were pushed out from AMO...

There is also the fact that with only-WE-supporting Fx builds there will not be easy way to take your engines (a part of your own personal local browsing data) to another browser, for example (well, downloading a older Fx build and running it with a new profile, this add-on, and a copy of your search.json.mozlz4 file from your other profile will work, at least while Moz doesn't change the internal format of that file). I'm thinking to make available a script with the minimal code to execute the "Export All Search Engines to a Zip file" feature, to be run via the JavaScript Scratchpad, compatible with Firefox 59 ESR, at least (EDIT: and whatever the next ESR-based SeaMonkey release will be).
Last edited by nohamelin on November 9th, 2017, 8:03 pm, edited 1 time in total.
nohamelin
Posts: 96
Joined: September 3rd, 2013, 4:04 pm
Location: Chile

Re: [Ext] XML Search Engines Exporter/Importer

Post by nohamelin »

nohamelin wrote:There is also the fact that with only-WE-supporting Fx builds there will not be easy way to take your engines (a part of your own personal local browsing data) to another browser [...]

I'm thinking to make available a script with the minimal code to execute the "Export All Search Engines to a Zip file" feature, to be run via the JavaScript Scratchpad, compatible with Firefox 59 ESR, at least.
OK, I have a first implementation for this:
https://gist.github.com/nohamelin/6af89 ... 9c396c9521

Further comments and instructions in the file itself.

I think I will do a single-script version for the "Import from file(s)" feature too. With both of them I can provide a minimally capable alternative for those people who still want to keep strict control over his engines. EDIT: It's up:
https://gist.github.com/nohamelin/8e2e1 ... 981487c6ec
mwalden
Posts: 23
Joined: August 20th, 2009, 8:01 am

Re: [Ext] XML Search Engines Exporter/Importer (+ Fx57 tools

Post by mwalden »

Hello,

As of Firefox 57.0.1, the searchplugins folder in the Profile Folder has become totally obsolete and useless.

In Firefox 57.0 you could still cause the browser to load in all of the search plugins in the searchplugins folder by deleting the search.json.mozlz4 file in the Profile Folder.

That method no longer works in Firefox 57.0.1. This change is not mentioned in the Firefox 57.0.1 release notes.

So now, the only *official* way to put search plugins into Firefox is by going to each plugin related web site and then click on a button to add it to the browser's search plugins database.

This is unacceptable to me since I have 143 search plugins that I created myself. I just can not imagine having to go to 143 web pages and click a button to add the search plugin to Firefox (on a fresh install of Firefox). Of course my personally created plugins do not exist on the sites they relate to, so that method will not work for me.

This leads me to your two xseei Java scripts.

Your xseei.export-all.js script exports all search plugins, including the 8 inbuilt (for an installation in the USA) search plugins even when they are hidden by un-check marking them on the about:preferences#search page under "One-Click Search Engines" or even when they have been removed by using the "Remove" button on the about:preferences#search page under the "One-Click Search Engines" section.

I would expect that plugins Removed or hidden would not be included in the exported zip file.

The two following search plugin names can not be exported from Firefox using xseei.export-all.js .
(Note that these two search plugins names imported just fine in Firefox 57.0 using the searchplugins folder)

<ShortName>Amazon.com.</ShortName>
<ShortName>Twitter.</ShortName>

I believe that those two file names will end up being the same as the two similar inbuilt search plugins (due to the removal of the trailing period characters by your script's file naming code) which causes the script to abort when exporting to the zip file where the names will already exist. You need to make the script change the file name of a search plugin if it already exists in the zip file. For example you could add a "(1)" to the end of the file name or a "(2)" if a "(1)" already exists, etc.. Also, you need to improve error checking and reporting.

When attempting to export one of those two search plugins mentioned above, your script crashes with a 0 byte exported zip file on disk. There is no error reported. If I try to delete the 0 byte zip file Windows reports that the file is open. If I close the scratchpad window and wait, the zip file will eventually go to a non 0 byte file size. On inspection in a zip viewer, I see all search plugins exported up to one of the bad file names mentioned above. If I delete the bad search plugins from the Firefox search plugin database, all of the remaining search plugins will export properly when attempting another export.

When importing a single new search plugin into an existing collection of search plugins using xseei.import.js, the newly imported one will end up at the bottom of the collection and not inserted in alphabetical order within the collection.

When importing a collection of new search plugins into an existing collection of search plugins using xseei.import.js, the newly imported collection will end up at the bottom of the existing collection and not inserted in alphabetical order within the existing collection. They will be in alphabetical order by file name and not alphabetical order by the internal short name.

If it is your intention to reproduce the functionality lost by the removal of the searchplugins folder from Firefox, you need to make your import function import the search plugins into the search plugin database in alphabetical order by the short name. The search plugins in Firefox were *never* listed in a non-alphabetical order.

When importing a single (or multiple) search plugin that already exists in the plugins database, no error is produced and the script just stops running. You need to improve error reporting and recovery.

Also, in both xseei.export-all.js and xseei.import.js you need to change the following line:

* You will need to have enabled "Enable chrome and add-on debugging" in the

to be as follows:

* You will need to have enabled "Enable browser chrome and add-on debugging toolboxes" in the

Thanks for all of your work in creating these scripts.

Michael
nohamelin
Posts: 96
Joined: September 3rd, 2013, 4:04 pm
Location: Chile

Re: [Ext] XML Search Engines Exporter/Importer (+ Fx57 tools

Post by nohamelin »

Thanks for your input, mwalden. I consider it fairly valuable, as it's the first time getting feedback from someone with more of 100+ engines, and some needs that arise from that. I myself have always installed no more of ~25 engines, and I never had the necessity, as user, to have my engines sorted by name, by example :-" .
mwalden wrote:As of Firefox 57.0.1, the searchplugins folder in the Profile Folder has become totally obsolete and useless.
It's the respective bug page: I am a bit surprised seeing how it was pushed all the way to the release channel days after landing in Nightly.
mwalden wrote:Your xseei.export-all.js script exports all search plugins, including the 8 inbuilt (for an installation in the USA) search plugins even when they are hidden by un-check marking them on the about:preferences#search page under "One-Click Search Engines"
Yeah, it's intended in that way, the add-on does the same (and it had the option to exclude the inbuilt engines from the zip). Sure, "Export all your One-Click Search Engines to a ZIP file" would be a nice addition, but from I mentioned in a previous post here, I'm not adding any new features to the add-on.
mwalden wrote:or even when they have been removed by using the "Remove" button on the about:preferences#search page under the "One-Click Search Engines" section.

I would expect that plugins Removed or hidden would not be included in the exported zip file.
Sorry, but I'm not seeing that behaviour in Nightly 59 Linux: it should work just as you expect. Can you try again, and mentioning the operative system where you are getting it?
mwalden wrote: The two following search plugin names can not be exported from Firefox using xseei.export-all.js
...
When attempting to export one of those two search plugins mentioned above, your script crashes with a 0 byte exported zip file on disk. There is no error reported. If I try to delete the 0 byte zip file Windows reports that the file is open. If I close the scratchpad window and wait, the zip file will eventually go to a non 0 byte file size. On inspection in a zip viewer, I see all search plugins exported up to one of the bad file names mentioned above. If I delete the bad search plugins from the Firefox search plugin database, all of the remaining search plugins will export properly when attempting another export.
Your guessings about the names are right, and that 0 byte zip file is a fugly bug ](*,). I will fix it in the script AND the add-on, for those using it in Firefox ESR 52. But it will take me some time.
mwalden wrote:When importing a single new search plugin into an existing collection of search plugins using xseei.import.js, the newly imported one will end up at the bottom of the collection and not inserted in alphabetical order within the collection.

When importing a collection of new search plugins into an existing collection of search plugins using xseei.import.js, the newly imported collection will end up at the bottom of the existing collection and not inserted in alphabetical order within the existing collection. They will be in alphabetical order by file name and not alphabetical order by the internal short name.
While it is the type of enhancements that I would have consider for the add-on, I'm not adding any of this to the scripts; see the next
mwalden wrote:If it is your intention to reproduce the functionality lost by the removal of the searchplugins folder from Firefox
That idea could apply for the add-on, sort of, but definitely not for the scripts. These must be seen as a sort of "basic survival kit" what rely perforce in a fraglle approach to work: the internal Firefox code of what the scripts depends can change with any new Fx release without any warning or care of the world: that is one of the reasons Firefox 57 dropped support for all type of extensions besides WebExtensions: a faster peace to throw improvements to the internals. That is why I decided to ship the scripts with little more than the minimum code to do the task they do. I expect I can keep the scripts working at least until Firefox 59 (the next ESR release), so with it there will be at least one officially supported environment during all of 2018 where you can keep managing your engines as before, giving you enough time to see what Mozilla does about this subject, in relation to new browser features or new APIs for add-on developers, and then decide about it.

If a WebExtensions-based build of the add-on can be written ever, I will consider again all your points mentioned.
mwalden wrote:When importing a single (or multiple) search plugin that already exists in the plugins database, no error is produced and the script just stops running. You need to improve error reporting and recovery.
Importing already existing engines is a thing properly reported via the Browser Console. I rely in users understanding that tool, more or less by the same reasons mentioned before.
mwalden
Posts: 23
Joined: August 20th, 2009, 8:01 am

Re: [Ext] XML Search Engines Exporter/Importer (+ Fx57 tools

Post by mwalden »

nohamelin wrote:Thanks for your input, mwalden. I consider it fairly valuable, as it's the first time getting feedback from someone with more of 100+ engines, and some needs that arise from that. I myself have always installed no more of ~25 engines, and I never had the necessity, as user, to have my engines sorted by name, by example :-" .
Yes, I understand that I am not the average user when it comes to search plugins. :wink: Also, thanks for your quick reply to my message.
nohamelin wrote:
mwalden wrote:Your xseei.export-all.js script exports all search plugins, including the 8 inbuilt (for an installation in the USA) search plugins even when they are hidden by un-check marking them on the about:preferences#search page under "One-Click Search Engines"
Yeah, it's intended in that way, the add-on does the same (and it had the option to exclude the inbuilt engines from the zip). Sure, "Export all your One-Click Search Engines to a ZIP file" would be a nice addition, but from I mentioned in a previous post here, I'm not adding any new features to the add-on.
Understood.
nohamelin wrote:
mwalden wrote:or even when they have been removed by using the "Remove" button on the about:preferences#search page under the "One-Click Search Engines" section.

I would expect that plugins Removed or hidden would not be included in the exported zip file.
Sorry, but I'm not seeing that behaviour in Nightly 59 Linux: it should work just as you expect. Can you try again, and mentioning the operative system where you are getting it?
Now, when exporting my search plugins where the inbuilt search plugins have been removed, they do not show up in the created zip file, as you would expect. I do not know why I had different results before. Maybe I was confused and opened the wrong .zip file (but I do not think so).

My system is Windows 7 with Firefox 57.0. I will be upgrading to 57.0.2 shortly.
nohamelin wrote:
mwalden wrote: The two following search plugin names can not be exported from Firefox using xseei.export-all.js
...
When attempting to export one of those two search plugins mentioned above, your script crashes with a 0 byte exported zip file on disk. There is no error reported. If I try to delete the 0 byte zip file Windows reports that the file is open. If I close the scratchpad window and wait, the zip file will eventually go to a non 0 byte file size. On inspection in a zip viewer, I see all search plugins exported up to one of the bad file names mentioned above. If I delete the bad search plugins from the Firefox search plugin database, all of the remaining search plugins will export properly when attempting another export.
Your guessings about the names are right, and that 0 byte zip file is a fugly bug ](*,). I will fix it in the script AND the add-on, for those using it in Firefox ESR 52. But it will take me some time.
I am happy to have contributed to making the script better. :D No problem with the time required. I reported all of my findings as soon as I discovered them with the hope that you would be able to fix them before the next time I need to update a plugin.
nohamelin wrote:
mwalden wrote:When importing a single new search plugin into an existing collection of search plugins using xseei.import.js, the newly imported one will end up at the bottom of the collection and not inserted in alphabetical order within the collection.

When importing a collection of new search plugins into an existing collection of search plugins using xseei.import.js, the newly imported collection will end up at the bottom of the existing collection and not inserted in alphabetical order within the existing collection. They will be in alphabetical order by file name and not alphabetical order by the internal short name.
While it is the type of enhancements that I would have consider for the add-on, I'm not adding any of this to the scripts; see the next
Understood.
nohamelin wrote:
mwalden wrote:If it is your intention to reproduce the functionality lost by the removal of the searchplugins folder from Firefox
That idea could apply for the add-on, sort of, but definitely not for the scripts. These must be seen as a sort of "basic survival kit" what rely perforce in a fraglle approach to work: the internal Firefox code of what the scripts depends can change with any new Fx release without any warning or care of the world: that is one of the reasons Firefox 57 dropped support for all type of extensions besides WebExtensions: a faster peace to throw improvements to the internals. That is why I decided to ship the scripts with little more than the minimum code to do the task they do. I expect I can keep the scripts working at least until Firefox 59 (the next ESR release), so with it there will be at least one officially supported environment during all of 2018 where you can keep managing your engines as before, giving you enough time to see what Mozilla does about this subject, in relation to new browser features or new APIs for add-on developers, and then decide about it.

If a WebExtensions-based build of the add-on can be written ever, I will consider again all your points mentioned.
Understood. I hope that your scripts will continue working without the Mozilla developers screwing things up.
nohamelin wrote:
mwalden wrote:When importing a single (or multiple) search plugin that already exists in the plugins database, no error is produced and the script just stops running. You need to improve error reporting and recovery.
Importing already existing engines is a thing properly reported via the Browser Console. I rely in users understanding that tool, more or less by the same reasons mentioned before.
Yes, I was aware of the Browser Console.
Now, trying again, when importing an already existing search plugin, I still get no error reported in the Browser Console.

Also, when exporting my search plugins that include the hidden inbuilt search plugins "Amazon.com" and "Twitter", the problem with "Amazon.com." and "Twitter." where the export crashes, there is no error message reported in the Browser Console.

Thanks,
Michael
nohamelin
Posts: 96
Joined: September 3rd, 2013, 4:04 pm
Location: Chile

Re: [Ext] XML Search Engines Exporter/Importer (+ Fx57 tools

Post by nohamelin »

mwalden wrote:Now, trying again, when importing an already existing search plugin, I still get no error reported in the Browser Console.

Also, when exporting my search plugins that include the hidden inbuilt search plugins "Amazon.com" and "Twitter", the problem with "Amazon.com." and "Twitter." where the export crashes, there is no error message reported in the Browser Console.

Thanks,
Michael
The export crash triggers the following message:
[Exception... "Component returned failure code: 0x80520008 (NS_ERROR_FILE_ALREADY_EXISTS) [nsIZipWriter.addEntryStream]" nsresult: "0x80520008 (NS_ERROR_FILE_ALREADY_EXISTS)"...
Please check that you are not using the "Web Console" (vs. the Browser Console), and the "Errors" checkbox is selected in the "JS" button of the Browser Console.
mwalden
Posts: 23
Joined: August 20th, 2009, 8:01 am

Re: [Ext] XML Search Engines Exporter/Importer (+ Fx57 tools

Post by mwalden »

nohamelin wrote:The export crash triggers the following message:
[Exception... "Component returned failure code: 0x80520008 (NS_ERROR_FILE_ALREADY_EXISTS) [nsIZipWriter.addEntryStream]" nsresult: "0x80520008 (NS_ERROR_FILE_ALREADY_EXISTS)"...
Please check that you are not using the "Web Console" (vs. the Browser Console), and the "Errors" checkbox is selected in the "JS" button of the Browser Console.
I am using the Browser Console, not the Web Console. I did not have the "Errors" checkbox selected in the "JS" button, so now that I do, I get the following:

[Exception... "Component returned failure code: 0x80520008 (NS_ERROR_FILE_ALREADY_EXISTS) [nsIZipWriter.addEntryStream]" nsresult: "0x80520008 (NS_ERROR_FILE_ALREADY_EXISTS)" location: "JS frame :: Scratchpad/1 :: saveEnginesToZipFile/</< :: line 124" data: no]

I had not used the Browser Console in a few years and forgot about that detail.

Thanks again,
Michael
nohamelin
Posts: 96
Joined: September 3rd, 2013, 4:04 pm
Location: Chile

Re: [Ext] XML Search Engines Exporter/Importer (+ Fx57 tools

Post by nohamelin »

mwalden wrote:The two following search plugin names can not be exported from Firefox using xseei.export-all.js .

<ShortName>Amazon.com.</ShortName>
<ShortName>Twitter.</ShortName>

I believe that those two file names will end up being the same as the two similar inbuilt search plugins (due to the removal of the trailing period characters by your script's file naming code) which causes the script to abort when exporting to the zip file where the names will already exist. You need to make the script change the file name of a search plugin if it already exists in the zip file. For example you could add a "(1)" to the end of the file name or a "(2)" if a "(1)" already exists, etc..
It has been fixed in the export-all script, and a new release of the add-on will be published this week. mwalden, can you try it?
mwalden
Posts: 23
Joined: August 20th, 2009, 8:01 am

Re: [Ext] XML Search Engines Exporter/Importer (+ Fx57 tools

Post by mwalden »

nohamelin wrote:
mwalden wrote:The two following search plugin names can not be exported from Firefox using xseei.export-all.js .

<ShortName>Amazon.com.</ShortName>
<ShortName>Twitter.</ShortName>

I believe that those two file names will end up being the same as the two similar inbuilt search plugins (due to the removal of the trailing period characters by your script's file naming code) which causes the script to abort when exporting to the zip file where the names will already exist. You need to make the script change the file name of a search plugin if it already exists in the zip file. For example you could add a "(1)" to the end of the file name or a "(2)" if a "(1)" already exists, etc..
It has been fixed in the export-all script, and a new release of the add-on will be published this week. mwalden, can you try it?
Yes, I can try the xseei.export-all.js code-revision 2. I am now running Firefox 57.0.2, so I am not able to test the add-on when available.

On running the script I got the "Successful export" alert message and no error messages in the Browser Console!

On inspecting the resulting .zip file, I find "amazoncom (1).xml" and "twitter (1).xml" in the list of exported search addons!

The .zip file contains a total of 151 search addons. (My 143 + 8 hidden inbuilt = 151 exported)

From reading your comment "Engines with very similar names can end with the same sanitized filename; we catch these to add a proper suffix." I can tell you fully understood the problem that I described.

Excelent work! In quick time too. :D

Thanks,
Michael
Post Reply