All plugins disabled due to expired cert

Bug #1827727 reported by Jean-Louis Dupond on 2019-05-04
134
This bug affects 25 people
Affects Status Importance Assigned to Milestone
Mozilla Firefox
Fix Released
Critical
firefox (Ubuntu)
High
Olivier Tilloy
Xenial
High
Olivier Tilloy
Bionic
High
Olivier Tilloy
Cosmic
High
Olivier Tilloy
Disco
High
Olivier Tilloy

Bug Description

See https://bugzilla.mozilla.org/show_bug.cgi?id=1548973

Due to expiration of an intermediate cert, all plugins were disabled.
Firefox pushed a fix via 'Studies' option, but this seems to be disabled by default in Ubuntu builds?

When this is disabled, no update is getting pushed!

Think we need to get a fix into repositories asap :)

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0

Steps to reproduce:

Wait until it's past midnight on 2019-05-04 UTC.

Actual results:

All addons got disabled due not having valid signature.

Expected results:

If the signature was due to expire, it should have been renewed weeks ago. Not all extensions were disabled. Fakespot and Google Scholar Button were left in their disabled state.

Some reports on reddit [1] says that they had their clocks a day forward, but they may be just early canaries for the actual widespread issue.

Going backwards in time allows installation from AMO but do not remove the unsupported mark from the add ons already installed.

[1]: https://www.reddit.com/r/firefox/comments/bk54mu/addonsextensions_broken/

List of affected add-ons:

```
Activate Reader View 0.1.0 true @activatereaderview
Netflix 1080p 1.8 true {89d04aec-e93f-4f56-b77c-f2295051c13e}
Amazon Assistant for Firefox 10.1904.10.11834 false <email address hidden>
Amazon SMILE! 1.4.5 false {1417a6e0-be73-4358-912c-5dce719b5791}
CanvasBlocker 0.5.8 false <email address hidden>
Check4Change 2.2.3 false <email address hidden>
Facebook Container 1.6.5 false @contain-facebook
Fakespot - Analyze Fake Amazon Reviews 0.3.1 false <email address hidden>
Firefox Multi-Account Containers 6.1.0 false @testpilot-containers
Ghostery – Privacy Ad Blocker 8.3.3 false <email address hidden>
Google Scholar Button 2.0 false <email address hidden>
Greasemonkey 4.7 false {e4a8a97b-f2ed-450b-b12d-ee082ba24781}
Honey 11.1.0 false jid1-93CWPmRbVPjRQA@jetpack
HTTPS Everywhere 2019.5.2.1 false <email address hidden>
InvisibleHand 6.6 false <email address hidden>
Kee - Password Manager 3.1.21 false <email address hidden>
Laboratory 3.0.5 false <email address hidden>
MEGA 3.57.9 false <email address hidden>
NflxMultiSubs (Netflix Multi. Subtitles) 1.6.7 false {e7ca39ec-6668-455e-9768-db28c364e4d2}
NoScript 10.6.1 false {73a6fe31-595d-460b-a920-fcc0f8843232}
ReviewMeta.com Review Analyzer 2.5 false <email address hidden>
Substital 2.1.0 false jid1-Cn7LiNrWh4k6RA@jetpack
uBlock Origin 1.18.16 false <email address hidden>
User-Agent Switcher 1.2.11 false <email address hidden>
```

Note, only Activate Reader View and Netflix 1080p were tested to check possible workarounds. I would leave those disabled for now. Also Firefox own Multi-Account Containers was blocked.

I can confirm

TREES ARE CLOSED FOR THIS.

*** Bug 1548975 has been marked as a duplicate of this bug. ***

*** Bug 1548980 has been marked as a duplicate of this bug. ***

*** Bug 1548979 has been marked as a duplicate of this bug. ***

*** Bug 1548978 has been marked as a duplicate of this bug. ***

(In reply to Andreea Pavel [:apavel] from comment #4)

> TREES ARE CLOSED FOR THIS.

To clarify, XPCShell signing tests are failing because of the expired cert.

*** Bug 1548976 has been marked as a duplicate of this bug. ***

Should other bug reports be opened about the empty error message that the browser console shows and related symptoms to help people know what it's going on? Or should that be implemented in a post-morten?

In case it's not understood I'm seeing a rash of reports of this across mozilla and freenode IRC networks as well as reddit.

Many people are very angry and it seems to be growing.

We don't yet know how broadly affected the user base is.
This seems like an urgent matter we want to get fixed as quickly as possible, at a high cost if necessary.

(In reply to Caspy7 from comment #12)
> We don't yet know how broadly affected the user base is.

We do. All users with add-ons and remotely accurate system clocks are affected, with the possible exception of nightly/dev edition users with signing disabled.

*** Bug 1548983 has been marked as a duplicate of this bug. ***

CloudOps is taking a look at this

Can somehow signing be disabled?

Confirming that add-ons were also disabled here on 66.0.3 (Win 10) at approx 9pm ET.

(In reply to Milos from comment #16)

> Can somehow signing be disabled?

Only on dev, nightly versions as :kmag noted.

(In reply to Milos from comment #16)

> Can somehow signing be disabled?

I don't think so if you are using Firefox 48+ on PC: https://wiki.mozilla.org/Add-ons/Extension_Signing#Timeline

However, on Firefox for Android (at least up to 66.0.2), you can set xpinstall.signatures.required to false and bypass this problem.

Stable, Beta, Nightly and Dev got hit for me.

I took off from work early today but someone from work texted me earlier and said about 200+ dev stations (major media conglomerate most people have heard of at one point in their life or another) got fuzzed hard thanks to this expired certificate.

Really surprised Mozilla is this careless about renewals.

(In reply to Alex from comment #20)

> Stable, Beta, Nightly and Dev got hit for me.
>
> I took off from work early today but someone from work texted me earlier and said about 200+ dev stations (major media conglomerate most people have heard of at one point in their life or another) got fuzzed hard thanks to this expired certificate.
>
> Really surprised Mozilla is this careless about renewals.

The only difference between release and the others is that non-release versions allow you to disable the check. It comes enabled by default for obvious reasons, like how malicious actors convinced people to download Chrome Canary to sidestep the safeguards on their release versions.

We have confirmed this issue. Extra comments about this being broken will not advance this bug to being fixed.

*** Bug 1548986 has been marked as a duplicate of this bug. ***

*** Bug 1548993 has been marked as a duplicate of this bug. ***

*** Bug 1548981 has been marked as a duplicate of this bug. ***

*** Bug 1548984 has been marked as a duplicate of this bug. ***

*** Bug 1548996 has been marked as a duplicate of this bug. ***

*** Bug 1548998 has been marked as a duplicate of this bug. ***

*** Bug 1549000 has been marked as a duplicate of this bug. ***

*** Bug 1549002 has been marked as a duplicate of this bug. ***

*** Bug 1549003 has been marked as a duplicate of this bug. ***

*** Bug 1549004 has been marked as a duplicate of this bug. ***

*** Bug 1549006 has been marked as a duplicate of this bug. ***

*** Bug 1549008 has been marked as a duplicate of this bug. ***

*** Bug 1549009 has been marked as a duplicate of this bug. ***

If you want to watch somewhere for user-facing updates on this issue, it looks like https://twitter.com/mozamo is the place to watch.

caitmuenster has also said this page will receive official updates/statuses on the issue:
https://discourse.mozilla.org/t/certificate-issue-causing-add-ons-to-be-disabled-or-fail-to-install/39047

38 comments hidden view all 112 comments
Jean-Louis Dupond (dupondje) wrote :

An option is to enable 'Studies' and wait 6 hours until you receive the fix.

Another option seems to download & install the fix xpi manually.
Studies seem to get fetched from https://normandy.cdn.mozilla.net/api/v1/recipe/

And there you have a link to the fix:
https://storage.googleapis.com/moz-fx<email address hidden>

Download & install. Everything fixed!

description: updated
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in firefox (Ubuntu):
status: New → Confirmed
Changed in firefox:
importance: Unknown → Critical
status: Unknown → Confirmed
70 comments hidden view all 112 comments

*** Bug 1549090 has been marked as a duplicate of this bug. ***

*** Bug 1549094 has been marked as a duplicate of this bug. ***

*** Bug 1549105 has been marked as a duplicate of this bug. ***

*** Bug 1549110 has been marked as a duplicate of this bug. ***

*** Bug 1549120 has been marked as a duplicate of this bug. ***

*** Bug 1549124 has been marked as a duplicate of this bug. ***

*** Bug 1549137 has been marked as a duplicate of this bug. ***

124 failures in 56 pushes (2.214 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 124

Platform breakdown:
* linux64-qr: 16
* linux64: 36
* android-em-4-3-armv7-api16: 12
* linux32: 12
* osx-10-10: 1
* windows7-32: 2
* linux64-ccov: 5
* windows10-64: 2
* windows10-aarch64: 3
* osx-10-10-shippable: 1
* windows10-64-ccov: 8
* windows7-32-shippable: 1
* linux32-shippable: 8
* linux64-shippable-qr: 8
* linux64-shippable: 8
* windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1548973&startday=2019-05-04&endday=2019-05-04&tree=all

*** Bug 1549138 has been marked as a duplicate of this bug. ***

*** Bug 1549143 has been marked as a duplicate of this bug. ***

*** Bug 1549173 has been marked as a duplicate of this bug. ***

*** Bug 1549118 has been marked as a duplicate of this bug. ***

*** Bug 1549180 has been marked as a duplicate of this bug. ***

*** Bug 1549198 has been marked as a duplicate of this bug. ***

*** Bug 1549203 has been marked as a duplicate of this bug. ***

*** Bug 1549248 has been marked as a duplicate of this bug. ***

*** Bug 1549218 has been marked as a duplicate of this bug. ***

124 failures in 4131 pushes (0.03 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:
* mozilla-central: 124

Platform breakdown:
* linux64-qr: 16
* linux64: 36
* android-em-4-3-armv7-api16: 12
* linux32: 12
* osx-10-10: 1
* windows7-32: 2
* linux64-ccov: 5
* windows10-64: 2
* windows10-aarch64: 3
* osx-10-10-shippable: 1
* windows10-64-ccov: 8
* windows7-32-shippable: 1
* linux32-shippable: 8
* linux64-shippable-qr: 8
* linux64-shippable: 8
* windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1548973&startday=2019-04-29&endday=2019-05-05&tree=all

*** Bug 1549352 has been marked as a duplicate of this bug. ***

Olivier Tilloy (osomon) on 2019-05-06
Changed in firefox (Ubuntu):
assignee: nobody → Olivier Tilloy (osomon)
importance: Undecided → High
status: Confirmed → In Progress

Please document here the root cause of this issue and what is being done to ensure that it does not happen again, or if that has already been documented elsewhere, please post a link to that documentation here. I've reviewed all the comments on this bug as well as the other information sources to which it links, and I don't see an explanation of the root cause or of steps being taken to prevent recurrence.

Olivier Tilloy (osomon) wrote :

Fixed in 66.0.4+build3-0ubuntu1. Backports to stable releases are coming.

Changed in firefox (Ubuntu):
status: In Progress → Fix Released
10 comments hidden view all 112 comments

*** Bug 1549679 has been marked as a duplicate of this bug. ***

9 comments hidden view all 112 comments
arQon (pf.arqon) wrote :

This shouldn't be marked as "Fix Released" when the LTS's still don't have an update available. I realise the "real" work is done, but we can't have LTS's being treated as second-class citizens to the extent that this crippling defect doesn't even show up as an active issue to their users in Launchpad.

/opinion :)

mp (m-p) wrote :

arQon wrote:

"....This shouldn't be marked as "Fix Released" when the LTS's still don't have an update available. I realise the "real" work is done, but we can't have LTS's being treated as second-class citizens to the extent that this crippling defect doesn't even show up as an active issue to their users in Launchpad...."

Yes, it would be great to know when Update Manager will offer that fix/upgrade to 66.0.4.

It is somewhat (a little) ironic that Ubuntu and derivatives users will be the last to get this fixed: Windozers et al. are already sorted.

Janne Snabb (snabb) wrote :

I agree. At least an ETA would be nice instead of vague "releases are coming" for such a critical bug.

It has been long two working days waiting for a fix and still no idea when to expect it to be working again.

Olivier Tilloy (osomon) wrote :

The bug is marked "Fix Released" because a bug against a source package always targets the latest development series (in this case 19.10, where the bug is indeed fixed).

This doesn't mean the supported releases are being treated as second-class citizens. The updates will be available to everyone very soon.

While this is indeed a critical bug, it has largely been mitigated by Mozilla issuing a new certificate through Normandy, which is enabled by default in Ubuntu packages. So unless users have forcefully disabled Normandy, they shouldn't be affected by the bug anymore.

eo (eo1000) wrote :

While it may be true the "studies" feature which enabled the temporary fix is turned on by default it is a highly contentious feature which many security minded users (isn't that what most firefox users are) have disabled. Yes, it can be turned back on to get the "fix" but I choose not to compromise one form of security to regain another that was lost due to an unfortunate mozilla error. Just the same as I and many others choose to use LTS and the standard release system to minimize risk and maximize stability. I hope the maintainers can understand this.

6 comments hidden view all 112 comments

*** Bug 1549200 has been marked as a duplicate of this bug. ***

5 comments hidden view all 112 comments
Manfred Hampl (m-hampl) wrote :

Shouldn't there be sub-tasks for "Firefox on Ubuntu xenial" "Firefox on Ubuntu bionic" "Firefox on Ubuntu cosmic" and "Firefox on Ubuntu disco" to allow tracking the status for the older supported Ubuntu Releases?

Olivier Tilloy (osomon) on 2019-05-08
Changed in firefox (Ubuntu Xenial):
assignee: nobody → Olivier Tilloy (osomon)
Changed in firefox (Ubuntu Bionic):
assignee: nobody → Olivier Tilloy (osomon)
Changed in firefox (Ubuntu Cosmic):
assignee: nobody → Olivier Tilloy (osomon)
Changed in firefox (Ubuntu Disco):
assignee: nobody → Olivier Tilloy (osomon)
Changed in firefox (Ubuntu Xenial):
status: New → In Progress
Changed in firefox (Ubuntu Bionic):
status: New → In Progress
Changed in firefox (Ubuntu Cosmic):
status: New → In Progress
Changed in firefox (Ubuntu Disco):
status: New → In Progress
Changed in firefox (Ubuntu Xenial):
importance: Undecided → High
Changed in firefox (Ubuntu Bionic):
importance: Undecided → High
Changed in firefox (Ubuntu Cosmic):
importance: Undecided → High
Changed in firefox (Ubuntu Disco):
importance: Undecided → High
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package firefox - 66.0.4+build3-0ubuntu0.16.04.1

---------------
firefox (66.0.4+build3-0ubuntu0.16.04.1) xenial; urgency=medium

  * New upstream stable release (66.0.4build3) (LP: #1827727)

 -- Olivier Tilloy <email address hidden> Mon, 06 May 2019 12:07:09 +0200

Changed in firefox (Ubuntu Xenial):
status: In Progress → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package firefox - 66.0.4+build3-0ubuntu0.18.04.1

---------------
firefox (66.0.4+build3-0ubuntu0.18.04.1) bionic; urgency=medium

  * New upstream stable release (66.0.4build3) (LP: #1827727)

 -- Olivier Tilloy <email address hidden> Mon, 06 May 2019 11:54:39 +0200

Changed in firefox (Ubuntu Bionic):
status: In Progress → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package firefox - 66.0.4+build3-0ubuntu0.19.04.1

---------------
firefox (66.0.4+build3-0ubuntu0.19.04.1) disco; urgency=medium

  * New upstream stable release (66.0.4build3) (LP: #1827727)

 -- Olivier Tilloy <email address hidden> Mon, 06 May 2019 11:36:12 +0200

Changed in firefox (Ubuntu Disco):
status: In Progress → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package firefox - 66.0.4+build3-0ubuntu0.18.10.1

---------------
firefox (66.0.4+build3-0ubuntu0.18.10.1) cosmic; urgency=medium

  * New upstream stable release (66.0.4build3) (LP: #1827727)

 -- Olivier Tilloy <email address hidden> Mon, 06 May 2019 11:42:56 +0200

Changed in firefox (Ubuntu Cosmic):
status: In Progress → Fix Released
2 comments hidden view all 112 comments

is the part
"[first mitigation completed, working on a second one]"
in the bug title meanigful in any way?

Changed in firefox:
status: Confirmed → Fix Released
Displaying first 40 and last 40 comments. View all 112 comments or add a comment.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers