[Ext] Cookie Controller, now with DOM storage

Announce and Discuss the Latest Theme and Extension Releases.
Locked
Creideiki
Posts: 2
Joined: December 21st, 2015, 3:05 am

Re: [Ext] Cookie Controller, now with DOM storage

Post by Creideiki »

Hi!

I'm having problems with Cookie Controller when upgrading Firefox above 38ESR. I've tried a couple of different versions, including 41 and 43 (all running on Gentoo Linux), and all give the same result.

I have a profile which has been upgraded through a couple of versions of Firefox, currently running on 38.5.0 ESR. It has cookies denied by default (Preferences -> Privacy -> Accept cookies from sites is off) and a few domains whitelisted by Cookie Controller. These domains show up in Preferences -> Privacy -> Accept cookies from sites -> Exceptions and in about:permissions as "Allow" or "Allow first party only".

If I try to use this profile with a newer Firefox, it still has cookies denied by default, but it also blocks cookies on domains I've whitelisted. The Preferences -> Privacy -> Accept cookies from sites -> Exceptions dialog box is empty. about:permissions lists the domains that were whitelisted in version 38, but they are all set to "Block". If I change cookie policy for a domain using Cookie Controller or about:permissions, that setting stays during the current browsing session, but as soon as I restart Firefox all domains are set back to blocking cookies.

I've read that this could happen if Preferences -> Privacy -> Clear history when Firefox closes -> Settings -> Site preferences is activated, but it is not. In fact, Preferences -> Privacy -> Clear history when Firefox closes is off.

Looking around in the profile directory I found permissions.sqlite, which seems to contain the whitelisting data. However, if I change it, or even delete the file, Firefox >38 still has the exact same behaviour: domains show up in about:permissions, but are always set to "Block" after restarting Firefox.

A fresh profile created on Firefox 43 does seem to remember cookie whitelist settings between restarts (and seems to store them in permissions.sqlite), but I'd prefer not to have to recreate my entire browser environment from scratch. Do you think you could help me upgrade?
lithopsian
Posts: 3664
Joined: September 15th, 2010, 9:03 am

Re: [Ext] Cookie Controller, now with DOM storage

Post by lithopsian »

Firefox has changed its whitelisting method for cookies, although this shouldn't kick in at version 39. Version 41 perhaps, certainly 42.

They are now whitelisted by "origin" instead of by host. So instead of whitelisting google.com you would now need to whitelist http://google.com:8080, and more importantly https://google.com:8080. Or both! Clearly a recipe for trouble. When upgrading, there is supposed to be a mechanism to convert your existing whitelist, but it isn't perfect. For example this bug.

I don't know if this is what you're seeing. I haven't had a case where nothing gets migrated, but perhaps if your history is cleared it might happen. Probably there is something odd in your profile that is killing the migration. Or something strange with ports. Perhaps knowing what is going on with the cookie exceptions list will help you diagnose the problem. Perhaps there is an addon that will export the whitelist and re-import it? I'm not aware of one. CookieSafe used to do it, but it doesn't work any more, so definitely no good for this case. Most Cookie Managers are obsessed with cookies I'm afraid, not cookie permissions.
Creideiki
Posts: 2
Joined: December 21st, 2015, 3:05 am

Re: [Ext] Cookie Controller, now with DOM storage

Post by Creideiki »

Sorry for now responding sooner; holidays.

I got one Firefox package from each major version from my distro and upgraded in sequence. This confirmed what you said about versions, since the problem occurred when going from 40.0.2 to 41.0-r1.

I continued upgrading all the way up to 43.0, started a new profile, and looked at the differences in permissions.sqlite between the upgraded and the new profile. The upgraded profile did not include the protocol in the hostname strings in the database (so there's probably something wrong with the migration mechanism anyway), but the freshly created one did. I edited the data in the upgraded profile to match, but still got the same buggy behaviour.

At this point, I gave up trying to upgrade, made a fresh profile and started manually porting settings. While doing so, I found a setting in the old profile that looked suspicious: permissions.memory_only = true. Sure enough, toggling that back to false restored cookie permissions to working order.

I have no idea how that ended up in my configuration (searching the web for it mostly gives hits from the Tor Browser), or why Firefox worked as expected with permissions.memory_only = true before version 41. Anyway, with permissions.memory_only = false Firefox 43 works!
mangus
Posts: 3
Joined: October 15th, 2012, 5:18 am

Re: [Ext] Cookie Controller, now with DOM storage

Post by mangus »

Hi!

I have a problem with Cookie Controller 4.5 in Firefox 43: when cookies for the site are denied (default setting) or allowed only as a 1st party, the right click on the back button doesn't work (it should show the list of previous visited pages in the same tab).

Please, try this to reproduce the problem, I have tried it in a fresh profile with just Cookie Controller added:
- Visit, for example, addons.mozilla.org with Default setting: Cookies denied and navigate through the site, for example searching the Cookie Controller addon. Right click on the back button doesn't work.
- Set Cookies allowed for addons.mozilla.org only as a 1st party. Right click on the back button doesn't work.
- Set Cookies allowed for addons.mozilla.org (or Cookies allowed for this session). Now right click on the back button works as it should.

I have found out this behaviour on all the sites I visited. As a further information, I tried the previous versions of Cookie Controller and found out that this problem doesn't occur with v3.12.
GossamerGremlin
Posts: 15
Joined: May 9th, 2007, 8:27 pm

Re: [Ext] Cookie Controller, now with DOM storage

Post by GossamerGremlin »

I am experiencing the same bug just described by mangus. The Back button right-click feature does not work, just as he describes, but with one small difference. Although the Back button right-click does not work for a tab that has cookies denied, and allowing cookies for that tab will cause the Back button right-click to start working, reverting that same tab back to denied cookies does not cause the Back button right-click to stop working again.

I have no idea why Cookie Controller should have anything to do with the Back button, but a fix would be appreciated.
GossamerGremlin
Posts: 15
Joined: May 9th, 2007, 8:27 pm

Re: [Ext] Cookie Controller, now with DOM storage

Post by GossamerGremlin »

I am currently trying out Cookie Controller (CC) because Cookie Monster (CM) is perpetually in a preliminarily reviewed state and Mozilla is taking far too long to sign the latest release. Having used CM for many years, I have two feature requests for CC that would allow me to leave CM in the rear view mirror.

1. CM has explicit permissions for temporarily allowing cookies. CC seems to have similar functionality achieved by middle-clicking a cookie permission, but there is no visual confirmation for whether a permission is temporary or permanent. I would like to see something in the toolbar icon indicating the temporary status and for the sake of novice users I would prefer that middle-clicking permissions be replaced with explicit menu entries for temporarily allowing cookies. Novice users simply won't use the temporary feature otherwise, even though it is more often the right option for privacy best practises.

2. The upcoming CM release will make it visually explicit that permissions apply to HTTP or HTTPS. I would like to see CC make the entire "origin" visually explicit in the pull down menu (ie. protocol, domain, and port). While it is true that the user can deduce this by looking at the address bar, the origin concept is not at all obvious to novice users.

https://addons.mozilla.org/en-US/firefo ... on-1.3.0.0

Some thought needs to be given about mixed content websites. It is really frustrating to users when they use tools like CM and CC to enable cookies for a website and then find that it still doesn't work correctly because they don't realize that they need to enable cookies for both HTTP and HTTPS separately. I recognize that true mixed content webpages are a bad thing, but lousy web developers are all too common and such sites exist. However, there's also the problem of pseudo-mixed content websites. On such sites the user is transitioned from HTTP to HTTPS (or vice versus) without realizing it. Having enabled cookies on the first page, he doesn't notice his permission changes no longer apply after the transition. Throw in some forms, a redirection or two, perhaps some iframes, and it can require a debugger to discover that the user was being transitioned from HTTP to HTTPS and back to HTTP without ever once having "https" appear in the address bar. Thanks to Mozilla's change from domain-based to origin-based cookie permissions, cookie managers have now become rather incompatible with non-expert users.
lithopsian
Posts: 3664
Joined: September 15th, 2010, 9:03 am

Re: [Ext] Cookie Controller, now with DOM storage

Post by lithopsian »

I've reproduced the back button bug in Firefox 43, but I don't know why it is happening. It doesn't appear to happen in nightly, so presumably it is a bug in Firefox itself. At this point it is unclear whether I will be able to create a fix.

Edit: I've tracked this down to a bug in SessionStorage.jsm, the module that transfers session storage values up from content for the session store. There is an existing bug although there has been no activity since it was raised. Perhaps now that it causes a problem other than just a console message it might get more attention, but I'll look for a workaround so Cookie Controller doesn't trigger it.
Last edited by lithopsian on January 10th, 2016, 4:42 pm, edited 1 time in total.
lithopsian
Posts: 3664
Joined: September 15th, 2010, 9:03 am

Re: [Ext] Cookie Controller, now with DOM storage

Post by lithopsian »

The whole origin permission thing is a pain, not least for the handling mixed content. Port permissions don't even bear thinking about.

I don't have immediate plans to change the user interface to reflect this. For now, Cookie Controller creates permissions both both http and https versions of a domain, which is what most people want and need (in the absence of being psychic). It would break legacy support anyway. When the flow of Firefox bugs related to this slows down, I will look at it again. Any failure to migrate existing permissions properly is a bug in Firefox, although there is little chance of it being fixed at this stage.

Temporary cookie permissions were implemented by request for a small number of users. They are not going to become the default and I'm not going to make the interface more complicated to support them.
lithopsian
Posts: 3664
Joined: September 15th, 2010, 9:03 am

Re: [Ext] Cookie Controller, now with DOM storage

Post by lithopsian »

The permissions.memory_only pref was introduced in Firefox 41, primarily for TOR. The default should be false. Could you have been running a nightly or other pre-release version at some point which might have had this preference set to true?
GossamerGremlin
Posts: 15
Joined: May 9th, 2007, 8:27 pm

Re: [Ext] Cookie Controller, now with DOM storage

Post by GossamerGremlin »

@lithopsian: Please have a look at Firefox bug 1238178. It looks like it may be referring to the same Back button bug and be associated with a larger set of session-related symptoms.

https://bugzilla.mozilla.org/show_bug.cgi?id=1238178

You may be able to help by adding what you've learned to that bug report.
GossamerGremlin
Posts: 15
Joined: May 9th, 2007, 8:27 pm

Re: [Ext] Cookie Controller, now with DOM storage

Post by GossamerGremlin »

lithopsian wrote:Temporary cookie permissions were implemented by request for a small number of users. They are not going to become the default and I'm not going to make the interface more complicated to support them.
I respect your decision, but I hope you'll excuse me if I make this one last appeal to you to provide at least a little visual clue in the toolbar icon when permissions are temporary. All of my users have been taught to use CM's temporary permissions and have done so for many years. In fact, most of them never use session or permanent cookies. Having to middle-click for temporary permissions will not be something most of my more elderly users will be able to remember, but some of the younger ones can learn it, particularly if they get some kind of visual feedback.
lithopsian
Posts: 3664
Joined: September 15th, 2010, 9:03 am

Re: [Ext] Cookie Controller, now with DOM storage

Post by lithopsian »

I've linked the bug you list with another I've been following which is essentially the same thing. Perhaps the combination of multiple functionality problems will prompt a fix. The basic issue is that Firefox throws an exception for a perfectly legitimate situation: cookies blocked, which also blocks session storage. This was a fairly fundamental design decision taken after much discussion and I don't see it being reversed, so a wide range of code all across Firefox (and addons) needs to adapt.

Replacement icons are always welcome, but there's only so much you can cram into 16x16 pixels ;) It would be trivial to flip the code for setting temporary cookies. It is the UI for managing and visualising it that is more of a problem. I can make one very simple change in that direction, which would be to provide a hidden preference that would make temporary exceptions the default, while permanent ones would still be available through a middle click of ctrl click. Who knows where that might lead ...
GossamerGremlin
Posts: 15
Joined: May 9th, 2007, 8:27 pm

Re: [Ext] Cookie Controller, now with DOM storage

Post by GossamerGremlin »

lithopsian wrote:Replacement icons are always welcome, but there's only so much you can cram into 16x16 pixels ;)
Perhaps the letter 'T' in the upper left corner of your icons could indicate a temporary setting? Having never created an icon before I tried to edit yours to that effect, but they looked horrible. Easier said than done if one doesn't know how to achieve the nice aliasing :?
lithopsian
Posts: 3664
Joined: September 15th, 2010, 9:03 am

Re: [Ext] Cookie Controller, now with DOM storage

Post by lithopsian »

I think I'll have to put a final nail in the coffin of this one. Firefox does not have an API to tell the difference between temporary and permanent cookie permissions for a domain. The API simply says that whether a domain has cookie permissions or not, but there is no practical way to know whether those permissions will disappear at the end of a session or at some other time.
GossamerGremlin
Posts: 15
Joined: May 9th, 2007, 8:27 pm

Re: [Ext] Cookie Controller, now with DOM storage

Post by GossamerGremlin »

I suspect I'm just not understanding something here, but CC shows a blue checkmark in the icon to indicate "session" and CM has different icons for permanent, session, and temporary, which is why I'd thought it would be possible in CC too. Thanks for having considered it though.
Locked