Firefox 67 in Ubuntu 18.10 thinks it's an older version

[Maia Everett]

Today I installed the Firefox 67 update on Ubuntu 18.04, then updated to Ubuntu 18.10. After this, launching Firefox gave me a "Using an older version of Firefox" message with only two options: create a new profile or quit.

Apparently the Firefox 67 build for 18.10 thinks it's an older version than the Firefox 67 build for 18.04.

For now I worked around the issue by manually installing the Firefox 67 build for Ubuntu 18.04, which did load the existing user profile correctly.

[J-a-therrien+mozilla (j-a-therrien+mozilla)]

Created attachment 9066781
compatibility.ini

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0

Steps to reproduce:

On 2019-05-21 I updated my computer from Ubuntu 18.04 to 19.04. I'm not sure if Firefox was updated to Firefox 67 prior to my upgrade. I was not using any nightly or developer versions.

Actual results:

When loading Firefox an error window came up with the following text:
> Using an older version of Firefox can corrupt bookmarks and browsing history already saved to an existing Firefox profile. To protect your information, create a new profile for this installation of Firefox.

Expected results:

Firefox should have loaded my profile without issue.

[J-a-therrien+mozilla (j-a-therrien+mozilla)]

Just a comment that I was directed here from Reddit and told to upload the compatibility.ini file from the old profile.

[Maia Everett (linneris)]

From this thread https://www.reddit.com/r/firefox/comments/brjj82/firefox_67_thinks_its_an_old_version_and_wont/?ref=readnext

"From looking at the compatibility.ini for the old profile vs the new one it looks like the long version number for the newer Ubuntu is slightly smaller.

18.04 profile: 67.0_20190517140819/20190517140819

19.04 profile: 67.0_20190517140251/20190517140251"

[Dtownsend (dtownsend)]

Just to check, are you using the Ubuntu provided version of Firefox or are you using one downloaded directly from Mozilla?

[Olivier Tilloy (osomon)]

This was reported as an ubuntu bug: https://bugs.launchpad.net/firefox/+bug/1830096.

It seems unfortunate that firefox uses MOZ_BUILD_DATE as its build ID (if I followed correctly the code path, see https://hg.mozilla.org/mozilla-central/file/tip/build/variables.py#l15).

The official build from Mozilla might not be affected by that bug, but on Ubuntu we build firefox for each supported release (at present that's 16.04, 18.04, 18.10, 19.04 and the development series, 19.10), which means that the same package of firefox for two different Ubuntu releases will be getting different build IDs, and there's no way to guarantee that the build ID for release X will be < the build ID for release Y, where X < Y (other than building the packages sequentially, instead of in parallel).

[Olivier Tilloy (osomon)]

I wonder whether setting MOZ_BUILDID to the Ubuntu package version string (e.g. "67.0+build2-0ubuntu0.19.04.1") when building would fix the problem, and wouldn't introduce undesirable side effects?

[J-a-therrien+mozilla (j-a-therrien+mozilla)]

(In reply to Dave Townsend [:mossop] (he/him) from comment #2)
> Just to check, are you using the Ubuntu provided version of Firefox or are you using one downloaded directly from Mozilla?

I'm using the Ubuntu provided version, both before and after upgrading.

[Olivier Tilloy (osomon)]

An effective workaround would be to rebuild the packages for all supported series sequentially, to ensure the build IDs are in ascending order.

Currently, at least for amd64, they are reversed:

$ grep -r --include="application.ini" BuildID | sort
16.04/usr/lib/firefox/application.ini:BuildID=20190517141553
18.04/usr/lib/firefox/application.ini:BuildID=20190517140819
18.10/usr/lib/firefox/application.ini:BuildID=20190517140354
19.04/usr/lib/firefox/application.ini:BuildID=20190517140251
19.10/usr/lib/firefox/application.ini:BuildID=20190517114929

[Dtownsend (dtownsend)]

(In reply to Olivier Tilloy from comment #3)
> It seems unfortunate that firefox uses MOZ_BUILD_DATE as its build ID (if I followed correctly the code path, see https://hg.mozilla.org/mozilla-central/file/tip/build/variables.py#l15).

My apologies, I hadn't noticed that this was the case.

> The official build from Mozilla might not be affected by that bug, but on Ubuntu we build firefox for each supported release (at present that's 16.04, 18.04, 18.10, 19.04 and the development series, 19.10), which means that the same package of firefox for two different Ubuntu releases will be getting different build IDs, and there's no way to guarantee that the build ID for release X will be < the build ID for release Y, where X < Y (other than building the packages sequentially, instead of in parallel).

If the builds for each release are from the same source then I don't think it would be an issue for the Firefox build ID to match across those builds, though I also imagine that sharing profiles between different Ubuntu releases is going to be rare.

The chief thing that we care about is that if the Firefox code is newer then it should have a larger compatibility version than previously. The compatibility version is defined as `<version>_<appbuildid>/<platformbuildid>` (the latter two are the same in Mozilla builds). We compare as if the string is a valid toolkit version (https://developer.mozilla.org/en-US/docs/Mozilla/Toolkit_version_format), and to make that the case we do some special parsing in the case where the buildid is the expected number of characters that are generated from Mozilla builds, otherwise we just use the literal compatibility version for comparison.

I am not sure about other uses of the build ID in Firefox, but I think one viable option is just to set the build ID to your "build2". This would lead to a compatibility version like `67.0_build2/build2`. This would fall back to just interpreting that as a full version string which looks like it compares correctly:

```
"67.0_build1/build1" < "67.0_build2/build2"
"67.0_build3/build3" > "67.0_build2/build2"
"67.0_build3/build3" < "67.1_build2/build2"
"67.0.1_build1/build1" < "67.1_build2/build2"
"67.0.1_build1/build1" > "67.0_build2/build2"
```

What do you think of that?

[Olivier Tilloy (osomon)]

Packages are being rebuilt, sequentially.

In the meantime I have reproduced the issue as originally reported, by updating firefox to 67.0+build2-0ubuntu0.18.04.1 in a bionic VM, launching it, then dist-upgrading to cosmic and launching firefox again. I have also verified that deleting the file compatibility.ini in the profile folder "fixes" the issue.

[Olivier Tilloy (osomon)]

(In reply to Dave Townsend [:mossop] (he/him) from comment #6)

> If the builds for each release are from the same source then I don't think it would be an issue for the Firefox build ID to match across those builds, though I also imagine that sharing profiles between different Ubuntu releases is going to be rare.

Indeed this wouldn't be an issue, rather a logical thing: two binary packages of the same version of firefox (sharing a single source package) have the same build ID.

> The chief thing that we care about is that if the Firefox code is newer then it should have a larger compatibility version than previously. The compatibility version is defined as `<version>_<appbuildid>/<platformbuildid>` (the latter two are the same in Mozilla builds). We compare as if the string is a valid toolkit version (https://developer.mozilla.org/en-US/docs/Mozilla/Toolkit_version_format), and to make that the case we do some special parsing in the case where the buildid is the expected number of characters that are generated from Mozilla builds, otherwise we just use the literal compatibility version for comparison.
>
> I am not sure about other uses of the build ID in Firefox, but I think one viable option is just to set the build ID to your "build2". This would lead to a compatibility version like `67.0_build2/build2`.
>
> […]
>
> What do you think of that?

This makes sense to me, and sounds like it would work. Note that "build2" in that context comes from the upstream build candidate, it's not an Ubuntu-specific thing: https://ftp.mozilla.org/pub/firefox/candidates/67.0-candidates/.

[Launchpad Janitor (janitor)]

This bug was fixed in the package firefox - 67.0+build2-0ubuntu2

---------------
firefox (67.0+build2-0ubuntu2) eoan; urgency=medium

  * No-change rebuild to fix the BuildID ordering between the same version of
    firefox on all supported Ubuntu releases (LP: #1830096)

 -- Olivier Tilloy <email address hidden> Thu, 23 May 2019 13:00:20 +0200

[Olivier Tilloy (osomon)]

Reverting the status to "Fix Committed" because the updates to bionic, cosmic and disco haven't been released yet.

[Olivier Tilloy (osomon)]

I am testing this suggestion, and it appears to work as expected.
Would the following patch be acceptable?

    --- a/build/variables.py
    +++ b/build/variables.py
    @@ -13,10 +13,12 @@ SOURCESTAMP_FILENAME = 'sourcestamp.txt'

     def buildid_header(output):
    - buildid = os.environ.get('MOZ_BUILD_DATE')
    - if buildid and len(buildid) != 14:
    - print('Ignoring invalid MOZ_BUILD_DATE: %s' % buildid, file=sys.stderr)
    - buildid = None
    + buildid = os.environ.get('MOZ_BUILDID')
    + if buildid is None:
    + buildid = os.environ.get('MOZ_BUILD_DATE')
    + if buildid and len(buildid) != 14:
    + print('Ignoring invalid MOZ_BUILD_DATE: %s' % buildid, file=sys.stderr)
    + buildid = None
         if not buildid:
             buildid = datetime.now().strftime('%Y%m%d%H%M%S')
         output.write("#define MOZ_BUILDID %s\n" % buildid)

I.e. prioritize the `MOZ_BUILDID` environment variable over `MOZ_BUILD_DATE`, if it is set.

[Dtownsend (dtownsend)]

(In reply to Olivier Tilloy from comment #8)
> I am testing this suggestion, and it appears to work as expected.

Ok, so the patch looks ok to me, though it would really need a build config peer to look at it. But we've discovered another issue related to how we compare the compatibility versions in bug 1554029 so we are going to be making changes to that. I suspect that what we change will still work for the suggestion above but while I realise this is an important issue for you it might be worth holding off any changes on your end until there is an accepted patch there. I'm hoping to have something written today.

[Olivier Tilloy (osomon)]

Fair enough, I'll hold off changes until bug 1554029 is fixed. Thanks Dave!

[Dtownsend (dtownsend)]

Ok so it looks like bug 1554029 has landed and stuck and will be rolling out in a 67.0.1 soon. During testing of that I found that the suggestion I made in comment 6 would have actually only worked on some platforms due to a bug in the version comparator code. It probably would have worked on Linux but now the code is fixed it that changes things a little.

If you care about your users being able to upgrade from a version with the old-style build ID (14 digits based on the date) then either you need to:

1. Continue to use a purely numeric build ID that is larger each time you build (and the first must be larger than the last build ID your users have).

2. Switch to a build ID that can be parsed by the toolkit version code (https://developer.mozilla.org/en-US/docs/Mozilla/Toolkit_version_format) and must contain a character that is not a digit in it (or Firefox will just parse it as a number) and do the above.

For the second option in the first upgrade where users are switching from the old build ID to your new one the version comparator will consider the old build ID to be "0" because the number is too large and overflows. So for your new build ID to be larger than that then it has to be a valid version considered larger than or equal to 0. Mostly that means it must start with at least one digit so "1build1" would work (and then after "1build2" etc. would be considered larger.

It's a little uglier than I'd like unfortunately but it should work.

[Seth Arnold (seth-arnold)]

I saw a comment somewhere that suggested this was "rare".

I think this is actually pretty common -- it feels like this has been discussed more than any other single issue on irc lately.

The consequence for most users is losing their entire Firefox history, settings, addons, saved passwords, sessions, etc.

Can we please push the time-delayed-rebuilds, if not the actual fixes, sooner rather than later?

Thanks

[Mason Loring Bliss (y-mason)]

For what it's worth, my kids encountered this on two desktops today, moving from 16.04 to 18.04. As this would be a major issue for non-technical folk I'd urge getting the fix out the door sooner rather than later - especially on 18.04, to preserve LTS as being a safe choice.

The workaround I used here, prior to being shown this bug, was to let Firefox make a new profile and to follow these instructions to copy over all the critical files:

    https://support.mozilla.org/en-US/kb/recovering-important-data-from-an-old-profile

[Jonathan Johnsson (jjohnsson)]

I just upgraded from 18.10 to 19.04, since it looked like a good time given the release cycle, and I have this problem.

How do I "manually installing the Firefox 67 build for Ubuntu 18.04" (as mentioned in the description)? In my case I guess it would be "Firefox 67 build for Ubuntu 18.10".

[Jonathan Johnsson (jjohnsson)]

For those who come here and can't wait until this fix is released (thanks for your hard work, people fixing this!) you can find workarounds here (I can't say if the workarounds are safe or not):

https://old.reddit.com/r/firefox/comments/brjj82/firefox_67_thinks_its_an_old_version_and_wont/

Running
MOZ_ALLOW_DOWNGRADE=1 firefox --ProfileManager
from bash worked for me, since I accidentally created a new profile when exploring my options, and had to switch back to my original profile. I now have access to my session, bookmarks and containers again.

[Dtownsend (dtownsend)]

*** Bug 1557014 has been marked as a duplicate of this bug. ***

[Dtownsend (dtownsend)]

Based on this needing an ubuntu change at this point I'm going to close this out.

[Launchpad Janitor (janitor)]

This bug was fixed in the package firefox - 67.0.1+build1-0ubuntu0.18.10.1

---------------
firefox (67.0.1+build1-0ubuntu0.18.10.1) cosmic; urgency=medium

  * New upstream stable release (67.0.1build1)

 -- Olivier Tilloy <email address hidden> Thu, 30 May 2019 11:04:58 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package firefox - 67.0.1+build1-0ubuntu0.18.04.1

---------------
firefox (67.0.1+build1-0ubuntu0.18.04.1) bionic; urgency=medium

  * New upstream stable release (67.0.1build1)

 -- Olivier Tilloy <email address hidden> Thu, 30 May 2019 07:39:22 +0200

[Dtownsend (dtownsend)]

*** Bug 1555986 has been marked as a duplicate of this bug. ***

[risto3 (risto3)]

I also experienced similar, but in my case I have my usual aarch64 client with /home on a NFS mount using ff67.
What happens is upon occasion mozilla apps (amongst others) from the server direct (using, for example, ssh -Y).

Firefox seems to change the LastOSABI in compatibility to Linux_x64_86-gcc3 from Linux_aarch64-gcc3 whereupon
the next time I launch firefox from my habitual client I get the message (like above)
> Using an older version of Firefox can corrupt bookmarks and browsing history already saved to an existing Firefox profile. To protect your
> information, create a new profile for this installation of Firefox.

[Olivier Tilloy (osomon)]

Created attachment 9077386
Bug 1553554 - Allow overriding the default build ID with a MOZ_BUILDID environment variable.

[Olivier Tilloy (osomon)]

Dave, I have submitted https://phabricator.services.mozilla.com/D37722 for review, I hope you and other build config experts can comment.

I have tested this with Ubuntu packages, setting the build ID to e.g. "build3". Your suggestion of prefixing the build ID with a number (i.e. 1build3) doesn't seem necessary if we ensure that this change becomes effective only in a new version of firefox, because the build ID won't matter in the version comparison, e.g. from 68.0_20190701120911/20190701120911 to 68.0.1_build1/build1.

Or am I missing a use case?

[Dtownsend (dtownsend)]

(In reply to Olivier Tilloy from comment #17)
> Dave, I have submitted https://phabricator.services.mozilla.com/D37722 for review, I hope you and other build config experts can comment.
>
> I have tested this with Ubuntu packages, setting the build ID to e.g. "build3". Your suggestion of prefixing the build ID with a number (i.e. 1build3) doesn't seem necessary if we ensure that this change becomes effective only in a new version of firefox, because the build ID won't matter in the version comparison, e.g. from 68.0_20190701120911/20190701120911 to 68.0.1_build1/build1.
>
> Or am I missing a use case?

Ah yes, you are right. As long as for a single "version" you use the same build id style you should be fine.

[Mh+mozilla (mh+mozilla)]

I'm confused at to why a MOZ_BUILDID "override" is necessary for you, and why the build date is a problem in the first place.

[Olivier Tilloy (osomon)]

Created attachment 9078090
Bug 1553554 - Minor syntactic change (use "not" instead of "is None").

Depends on D37722

[Graham Smith (graham-u)]

THIS IS STUPID!
The problem is arising in Ubuntu because the builds against the current supported versions of Ubuntu have differing BuildIDs. This is because some stupid idiot chose to use the build time as the BuildID. Whoever that was needs teaching some lessons in basic software design and development. what makes it worse is the the BuildID is hard-coded into the specific install.

It is wrong to say that it would be rare for someone to have more than one version of Ubuntu and be sharing profiles across those installations. I know quite a few folk who try differing versions of Ubuntu and do just that. In my case, I have a current build of Xfce 19 amd64 and Cinnamon 18 amd64 on the same machine and also have Xfce 18 i386 on a very useful but old-ish netbook for work during travelling. I share profiles. The build of Firefox is EXACTLY the same version (68.0.1) on all three -- SO WHY THE F**K CAN'T I SHARE PROFILES!!???

There is a clear and obvious solution to this. Firefox HAS and EXISTING and PERFECTLY VALID Build ID.... SourceStamp. For what unearthly reason is that NOT being used as the means of identifying whether the profile is workable for the installed version of Firefox. Given that the rest of the profile coding is platform independent, this should be perfectly sufficient.

Go to work boys!!

[Ubu Foo (ubufoo)]

This hit me on 18.04 and did kill my saturday morning - thank you very much.

Also a BIG THANK YOU to Graham Smith for pointing out why this bug exists - it is very important to take the role of the one that tells the truth even when it hurts sometimes. It is of course possible to write more softly, but sometimes it is needed to point out that something is really a bad idea if it is a really bad idea.

It would be interesting to read why SourceStam was not used here and why was that BuildID implemented?

Also this shows how important it is to better test migration from old systems to newer releases - I have the impression that this area is lacking serious efforts.