Linux Firefox updates STILL deleting custom search engines

[phkhgh]

In post http://forums.mozillazine.org/viewtopic ... &t=3031325 I thought I solved Fx (Linux) updates repeatedly deleting every custom search plugin.
It's not solved. It's been going on since at least Fx 52 - through v55. So I need some suggestions.
I thought it was because my profile was created under Windows, then migrated to Linux - causing some "glitch." It's now apparent that wasn't the problem.

After updating to Fx 54 & the same search engines problem, I let Fx create a brand new search.json.mozlz4 - under Linux. Didn't copy anything search related from the old profile to the new.
There was no old "searchplugins" folder in the new profile, to confuse anything. The profiles.ini has no references to old profiles used under Windows. This time, all custom engines were added to a clean profile, created under Linux.

Then I added custom search engines to new search.json.mozlz4, via Fx search manager. Fresh ones. Like custom DDG, dictionary.com, etc. Nothing crazy. The default & custom engines all worked fine.
Before updating to Fx 55.0.2, I duplicated search.json.mozlz4 (gave a diff name, & .bak extension) & left the original file in place - with the custom engines.

Again, after the 55.0.2 update, all custom engines were gone. And search.json.mozlz4 file's "date modified" was the exact day & hour of the update.
I exported the "updated" search.json.mozlz4 and sure enough, only the default engines were in it.

search.json.mozlz4's size before and after updating was 35kb vs 14kb. There's no doubt the updates wipe perfectly functioning files, replacing with only default engines & resetting Yahoo as default.

Other than the fact I'd installed custom search engines & Yahoo wasn't default, there was no difference. All the engines (even Yahoo, Google) worked fine before updating. But, I rarely use any of the default engines. I even exported some of the custom engines & inspected - to make sure they were same format as default engines. The only differences between default & custom engine xml files (exported from search.json.mozlz4) were, the custom engines didn't have any Mozilla or Firefox references in them, so Mozilla wouldn't get any compensation.

[Grumpus]

With the changes in Firefox code if those custom search engines depend on internal Firefox code it could be that.
On a similar note I dump some of the search engines which come with different locales but have never seen any of these changes.
There is also a possibility the search engines change could be from your number one default engine.
I changed my number one default search to a different engine than Yahoo and Google and nothing has been changed since before 52.
Your User Agent String shows Windows




Were you aware of this DuckDuckGo Outage

[nohamelin]

Are you using the build of Firefox provided by your Linux distribution or the binary package available from mozilla.org? If it's the former case, try again your tests with a clean profile running Firefox 54 downloaded from moziila.org and update it: if it is working here then it could be a specific problem with the build from your Linux system.

[phkhgh]

@ nohamelin - no, I use the version from Mozilla's server, installed to /opt/firefox. And, I've updated it in 2 ways. 1) by D/L the partial.mar update file & applying it (simple steps from help sites). 2) Just to update, starting Fx with sudo & it updates normally. Both ways wiped out custom search engines. [ In Linux (Mint or Ubuntu), in non-root, std user mode, Fx installed from tar packages won't show "update now."]

Thanks Grumpus.

With the changes in Firefox code if those custom search engines depend on internal Firefox code it could be that.

Depend on internal code - how? Don't all search engines depend on Fx in some manner? Could you / others expand on that comment? And wouldn't 1000's of users be complaining?
I've read Mozilla's requirements for Creating OpenSearch plugins https://developer.mozilla.org/en-US/Add ... or_Firefox - and other similar docs.
I exported both default & custom engines to xml - to examine that they followed the format / met requirements. I wasn't modifying any, just inspecting.

On a similar note I dump some of the search engines which come with different locales but have never seen any of these changes.

You mean, you've never seen Fx wipe out custom search engines, that you added? Neither had I - for yrs under Windows, until started updating Fx in Linux.

There is also a possibility the search engines change could be from your number one default engine.

How do you mean (maybe I could check other things)? FYI: all the default engines are / were always left intact & unedited, in search.json.mozlz4. I just added some custom ones.
Yes, I usually set a custom engine as the default. Re: your comment, "search engines change could be from your number one default engine" :
My chosen default engine wasn't always the same one, at each Fx update (i.e., not always DDG or Ixquick, etc.). But surely, 1000's or millions of users set custom engines as their default?

I changed my number one default search to a different engine than Yahoo and Google and nothing has been changed since before 52.

Was your "default" engine (not yahoo, google) one that Fx provides, or a custom one? If the latter, was it from AMO, or from the search site itself? Are copies of (custom) "Example-search-site.xml" saved anywhere?

On at least 1 or 2 Fx updates - that wiped custom engines, some engines came from Mozilla.org; some from the search sites.
It's feasible that ONE custom engine was misformed (?? causing Fx update to delete it ??), but not all custom engines.

I'm not sure the User Agent String showing Windows would affect it deleting custom engines. AFAIK, Win vs. Linux don't have diff search engines.
I find no references to any OS in decompressed search.json. But if you want, you could PM me - or me you, get my email & send a copy of your search.json.mozl4, to compare w/ mine. That would probably show if search.json.mozl4 is OK, then the problem lies elsewhere.

BTW, a script to decompress search.json.mozlz4 to "search.json" is here: http://forums.mozillazine.org/viewtopic ... #p14719497
How to use: Start Fx w/ the profile you want to examine search.json.mozlz4. Start the browser CONSOLE (Ctrl + Shift + J). Enter the script in command line, at bottom of Console. It places search.json in same profile, but leaves original file intact. Later, delete or move the search.json.

I found info on Mozilla's site for creating open search plugins. It may be of use when I reload custom engines (if any complaints):

In addition, the search plugin service provides a logging mechanism that may be of use to plugin developers. Use about:config to set the pref 'browser.search.log' to true. Logging information will appear in Firefox's Error Console (Tools 〉 Error Console) when search plugins are added.

** Thought: In yours or others Fx Preferences > Advanced > Update, is Automatically Update Search engines checked or not? Mine's UNchecked (always).
Expanding on that, in the old profile from Windows (no longer used), in search.json (decompressed) my custom engines showed a "load path" pointing to an unmounted Win partition, on another disk. I thought that's what caused the custom engines to disappear during updates - that location couldn't be found. But a new profile under 'nix didn't solve it. But...

*** In search.json - default engines all show a load path: {"_name":"Google","_shortName":"google-nocodes","_loadPath":"jar:[app]/omni.ja!browser/google-nocodes.xml"
Makes sense - it gets them from omni.ja!
*** All custom engines show load paths similar to: "_shortName":"duckduckgo-html","_loadPath":"[https]duckduckgo.com/duckduckgo-html.xml"

I don't know the implication. If during updates, Fx looks for copies of custom XML files - they don't exist. Installing custom engines directly from sites, doesn't save xml copies to disk (searched w/ Gnome Commander). Fx USED TO save custom xml files in "searchplugins" folder in profile, but no more. If it's supposed to be saving them (somewhere), it may explain why they disappear during updates.

Looking at all Fx update logs (on entire disk), none logged any errors, if they do that.

It's a fact Fx changed search.json.mozlz4 - it's modified time stamp changed, and size (wiped custom engines). What it's actually supposed to do, I don't know.
Maybe it (was supposed to) "check" if all default engines were present & replace missing ones. They were all there, but it (consistently) wipes custom engines.

[Grumpus]

In plain language, not having a particular Linux operating system installed version of Firefox it is very likely some of the necessary connections are not made.
There is more than a noticeable difference between the operating systems version and the Mozilla tar ball's local connections made.
Having the automatic update of the search engines turned off seems a good idea and better to see what happens manually,.
Placing some search engines as bookmarks also seems to help.
You note your search of the system does not show search engines, one of the reasons for using a particular Linux systems version is the placement of those search engines.
In some they are easy to find but in others a little more difficult.
You can't always consider things for Microsoft platforms as working with Linux platforms, it is misguided to think so.
To perform a complete search you have to open the file manager as root and then it helps to look for words similar along with .xml and .exe.
Errors should be in the various logs in the systems logfiles folder.

[nohamelin]

...starting Fx with sudo...

If you did it a single time, you have added possible permission issues with your Fx profile files/folders to the pool.

[Grumpus]

Sorry I missed the starting Firefox with Sudo.
You have two levels of your system - User and root in Ubuntu.
nohamelin is right in some of the changes in permissions but also opening Firefox using sudo opens the root version and not the user version.
If you never altered/changed your root level version it does not have the changes (your search engines) made in the User version.
This same thing can happen if you open nautilus using sudo and then when asked what package to use to open a root level html file with Firefox it wipes the user level setup.
Do not open Firefox using sudo.
You can check your permissions in the properties panels of the folders and modify them accordingly.

[Reflective]

Why not just install Add To Search Bar

Next step is to go to the home page of your favorite search engines and then right click in the search field and click "Add to Search bar".

Provided you've also installed Context Search you can right click and use any of the search engines you've added. No plugins necessary.

Here's a pix of what my search engines look like in the context menu: https://i.imgur.com/bBwDASk.png

Oh, and Waterfox 55.0.1 for Linux is available to download which will offer support for legacy extensions beyond FF57.

[Grumpus]

@Reflective - The OPs circumstance was the search engines were being removed (not really removed but a different Firefox replaced it) when there was an update but the OP made the error of starting Firefox as root which changes all the user settings, not just the bookmarks. The addition of any extension would make little or no difference if opened as user and then later opened as root.
It's technically two different users with different permissions and settings in the Ubuntu system.
Installing Waterfox wouldn't help as long as the OP continues to open Firefox differently from time to time.

[phkhgh]

Again, thanks for all replies.
I know this thread has a lot of info & can be hard to follow. I apologize.
@ Grumpus - where did you get the idea I thought Windows & Linux worked the same? They both have -different- problems. Plenty of bugs in every OS & most apps.
Starting an app in elevated privilege doesn't change permissions of the files in user mode. If it does, I'd appreciate official documentation. There are hundreds of examples that it doesn't. If that were true, most Linux users would be constantly reinstalling apps.

Alas, there's no useful info that I hadn't already thought about. Which is OK - I don't have the answer, either.

1st & foremost, the way I installed and, at least the 1st method I used to update Fx (installed from tar package), isn't any different than what 1000's of users have done.
Mozilla recommended it as the way to update 'Nix Firefox - installed from tar packages. It didn't require starting Fx in sudo, but still wiped custom search engines.

Millions of 'Nix users install Mozilla's full version tarball packages. Then AFAIK, Linux / Firefox won't allow automatic or one click updating in user mode. It doesn't show an "update" button in Help> about:firefox, if in user mode.
That leaves only a couple of update options. Which update method do those using Fx from tarballs use? Download the full installer tarball & install to the existing Fx location?

* I said, there are no <custom-search-engine.XML> files stored for the search engines, for Fx tarball installations, that search.json.mozl4 refers to.
Thanks for the tip, but I'm familiar w/ finding hidden or system files. Nemo, btw, currently won't find -any- hidden files. The backend it uses has the ability (I think), but Nemo devs disabled / don't use it.

My guess is (for Fx 52 or 53+), no one else has these xml files, either. Unless they were manually saved.
It's probable that the original "load path" for each engine is recorded when 1st installed. Then I doubt Fx looks for those BU xml files, during each update. Each engine's open search style data is saved in search.json.mozl4.
Mozilla purposely removed the searchplugins folder, where they used to be stored. No one commented, if they see custom search xml files being automatically saved in Fx 54, 55+ (like older Fx versions did); if so, please post info.

@reflective = thanks, but as you noted, the addon will soon be broken. In your case, I guess it stores your search engines in a diff file than Firefox's search.json.mozl4?
So, I wouldn't expect Fx to mess w/ that during updates, even if you had previously seen the same problem as me. Yes - thanks, I'm considering other forks. At least for a while, there won't be web extension replacements for addons I consider essential.

1) We're overlooking that several update methods changed or replaced ONLY the search.json.mozl4 file, its "modified on" date & resets it to default search engines.
It started long before I ever started Fx with sudo. Thus, sudo wouldn't be the cause.

Either something's odd about this one file (for me). No other user settings or customizations changed.
Or maybe, for Fx installed from tar package, updating it by any method other than installing the full tar package over the existing version, has some bugs. I don't know.

I feel like we're getting off on tangents about permissions. A change in permissions isn't causing replacement of search.json.mozl4. That file's permissions are the same in Mint's Fx default profile as all other profiles. Unless others also see their custom search engines disappearing, I'd assume this is rare.

I may D/L the next to most recent Fx, install it in a new location, create new profile, add some custom searches (incl. from AMO), then install the next version over the top.
If it still wipes out custom engines, I'll report a bug, and just BU my profile before updates. Firefox & most apps have bugs 5, 10, 15 yrs old - & new ones every day.

[Grumpus]

I read or maybe misread one of the comments as the apps for Linux versions being the same as the apps (search engines) for Windows, not the systems themselves.
My remark was more for doing things like cross platforming the profile and apps between Linux and MS, is not a good thing to do.
Secondly the errors in Firefox permissions appear to be more important and not really in a tangent.
Having performed the dumb move myself (twice now due to being engrossed in reading or a process edit) of opening an html with Firefox while at admin level of nautilus and having the settings removed from Firefox it seems to me the ownerships and permissions play more of a part in the problems you're having with the search engines. There are a number of apps which are available which do not support Linux properly and even with some of the bash commands available on the Linux system can not be made to work.

I may D/L the next to most recent Fx, install it in a new location, create new profile, add some custom searches (incl. from AMO), then install the next version over the top. If it still wipes out custom engines, I'll report a bug, and just BU my profile before updates. Firefox & most apps have bugs 5, 10, 15 yrs old - & new ones every day.

It can't really be a bug issue if you do not follow a recommended procedure for the installation. Particularly with third party versions of Firefox. This is the main reason for using the Ubuntu repositories instead of the Mozilla supplied tar ball. Also the locations of the various shared and library files can and are in some cases different with the various third party installs. If Mozilla had a Debian install or a package which was using the newer Snap setup it might be different but so far it's the same old tar ball.
Downloading a Linux version from Mozilla is wrong minded unless you are a developer due to the connections which the local install might not but needs to make depending on the OS.
It would be interesting to see if the search engines disappeared with the Ubuntu Version of Firefox from the repository.