All plugins disabled due to expired cert

[Jean-Louis Dupond]

See https://bugzilla.mozilla.org/show_bug.cgi?id=1548973

Due to expiration of an intermediate cert, all plugins were disabled.
Firefox pushed a fix via 'Studies' option, but this seems to be disabled by default in Ubuntu builds?

When this is disabled, no update is getting pushed!

Think we need to get a fix into repositories asap :)

[Braiam Peguero (braiampe)]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0

Steps to reproduce:

Wait until it's past midnight on 2019-05-04 UTC.

Actual results:

All addons got disabled due not having valid signature.

Expected results:

If the signature was due to expire, it should have been renewed weeks ago. Not all extensions were disabled. Fakespot and Google Scholar Button were left in their disabled state.

Some reports on reddit [1] says that they had their clocks a day forward, but they may be just early canaries for the actual widespread issue.

Going backwards in time allows installation from AMO but do not remove the unsupported mark from the add ons already installed.

[1]: https://www.reddit.com/r/firefox/comments/bk54mu/addonsextensions_broken/

[Braiam Peguero (braiampe)]

List of affected add-ons:

```
Activate Reader View 0.1.0 true @activatereaderview
Netflix 1080p 1.8 true {89d04aec-e93f-4f56-b77c-f2295051c13e}
Amazon Assistant for Firefox 10.1904.10.11834 false <email address hidden>
Amazon SMILE! 1.4.5 false {1417a6e0-be73-4358-912c-5dce719b5791}
CanvasBlocker 0.5.8 false <email address hidden>
Check4Change 2.2.3 false <email address hidden>
Facebook Container 1.6.5 false @contain-facebook
Fakespot - Analyze Fake Amazon Reviews 0.3.1 false <email address hidden>
Firefox Multi-Account Containers 6.1.0 false @testpilot-containers
Ghostery – Privacy Ad Blocker 8.3.3 false <email address hidden>
Google Scholar Button 2.0 false <email address hidden>
Greasemonkey 4.7 false {e4a8a97b-f2ed-450b-b12d-ee082ba24781}
Honey 11.1.0 false jid1-93CWPmRbVPjRQA@jetpack
HTTPS Everywhere 2019.5.2.1 false <email address hidden>
InvisibleHand 6.6 false <email address hidden>
Kee - Password Manager 3.1.21 false <email address hidden>
Laboratory 3.0.5 false <email address hidden>
MEGA 3.57.9 false <email address hidden>
NflxMultiSubs (Netflix Multi. Subtitles) 1.6.7 false {e7ca39ec-6668-455e-9768-db28c364e4d2}
NoScript 10.6.1 false {73a6fe31-595d-460b-a920-fcc0f8843232}
ReviewMeta.com Review Analyzer 2.5 false <email address hidden>
Substital 2.1.0 false jid1-Cn7LiNrWh4k6RA@jetpack
uBlock Origin 1.18.16 false <email address hidden>
User-Agent Switcher 1.2.11 false <email address hidden>
```

Note, only Activate Reader View and Netflix 1080p were tested to check possible workarounds. I would leave those disabled for now. Also Firefox own Multi-Account Containers was blocked.

[Asif Youssuff (yoasif)]

See also https://github.com/mozilla/addons/issues/978

[Laurent Dinclaux (laurent-gecka)]

I can confirm

[Apavel-2 (apavel-2)]

TREES ARE CLOSED FOR THIS.

[Haftandilian (haftandilian)]

*** Bug 1548975 has been marked as a duplicate of this bug. ***

[Gingerbread-man-2 (gingerbread-man-2)]

*** Bug 1548980 has been marked as a duplicate of this bug. ***

[Gingerbread-man-2 (gingerbread-man-2)]

*** Bug 1548979 has been marked as a duplicate of this bug. ***

[Gingerbread-man-2 (gingerbread-man-2)]

*** Bug 1548978 has been marked as a duplicate of this bug. ***

[Wkocher (wkocher)]

(In reply to Andreea Pavel [:apavel] from comment #4)

> TREES ARE CLOSED FOR THIS.

To clarify, XPCShell signing tests are failing because of the expired cert.

[Gingerbread-man-2 (gingerbread-man-2)]

*** Bug 1548976 has been marked as a duplicate of this bug. ***

[Braiam Peguero (braiampe)]

Should other bug reports be opened about the empty error message that the browser console shows and related symptoms to help people know what it's going on? Or should that be implemented in a post-morten?

[Caspy77 (caspy77)]

In case it's not understood I'm seeing a rash of reports of this across mozilla and freenode IRC networks as well as reddit.

Many people are very angry and it seems to be growing.

We don't yet know how broadly affected the user base is.
This seems like an urgent matter we want to get fixed as quickly as possible, at a high cost if necessary.

[Kmaglione+bmo (kmaglione+bmo)]

(In reply to Caspy7 from comment #12)
> We don't yet know how broadly affected the user base is.

We do. All users with add-ons and remotely accurate system clocks are affected, with the possible exception of nightly/dev edition users with signing disabled.

[Sancus (sancus)]

*** Bug 1548983 has been marked as a duplicate of this bug. ***

[Habib-j (habib-j)]

CloudOps is taking a look at this

[Diabolique (vozdovac)]

Can somehow signing be disabled?

[Crea8ive49 (crea8ive49)]

Confirming that add-ons were also disabled here on 66.0.3 (Win 10) at approx 9pm ET.

[Braiam Peguero (braiampe)]

(In reply to Milos from comment #16)

> Can somehow signing be disabled?

Only on dev, nightly versions as :kmag noted.

[Outvi V (outloudvi)]

(In reply to Milos from comment #16)

> Can somehow signing be disabled?

I don't think so if you are using Firefox 48+ on PC: https://wiki.mozilla.org/Add-ons/Extension_Signing#Timeline

However, on Firefox for Android (at least up to 66.0.2), you can set xpinstall.signatures.required to false and bypass this problem.

[Atamas090 (atamas090)]

Stable, Beta, Nightly and Dev got hit for me.

I took off from work early today but someone from work texted me earlier and said about 200+ dev stations (major media conglomerate most people have heard of at one point in their life or another) got fuzzed hard thanks to this expired certificate.

Really surprised Mozilla is this careless about renewals.

[Braiam Peguero (braiampe)]

(In reply to Alex from comment #20)

> Stable, Beta, Nightly and Dev got hit for me.
>
> I took off from work early today but someone from work texted me earlier and said about 200+ dev stations (major media conglomerate most people have heard of at one point in their life or another) got fuzzed hard thanks to this expired certificate.
>
> Really surprised Mozilla is this careless about renewals.

The only difference between release and the others is that non-release versions allow you to disable the check. It comes enabled by default for obvious reasons, like how malicious actors convinced people to download Chrome Canary to sidestep the safeguards on their release versions.

[Kbrosnan-mozilla (kbrosnan-mozilla)]

We have confirmed this issue. Extra comments about this being broken will not advance this bug to being fixed.

[Ehumphries (ehumphries)]

*** Bug 1548986 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1548993 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1548981 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1548984 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1548996 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1548998 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1549000 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1549002 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1549003 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1549004 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1549006 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1549008 has been marked as a duplicate of this bug. ***

[R-jorge (r-jorge)]

*** Bug 1549009 has been marked as a duplicate of this bug. ***

[Sphink (sphink)]

If you want to watch somewhere for user-facing updates on this issue, it looks like https://twitter.com/mozamo is the place to watch.

[Caspy77 (caspy77)]

caitmuenster has also said this page will receive official updates/statuses on the issue:
https://discourse.mozilla.org/t/certificate-issue-causing-add-ons-to-be-disabled-or-fail-to-install/39047

[Mauricio Navarro Miranda (mautematico)]

*** Bug 1548988 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1549016 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1549012 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1549019 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1549023 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1549024 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1549026 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

*** Bug 1549029 has been marked as a duplicate of this bug. ***

[Virtual-q (virtual-q)]

*** Bug 1549017 has been marked as a duplicate of this bug. ***

[L. David Baron (dbaron)]

The dates on the cert in question were:

Not Before: May 4 00:09:46 2017 GMT
Not After : May 4 00:09:46 2019 GMT

[Virtual-q (virtual-q)]

*** Bug 1549031 has been marked as a duplicate of this bug. ***

[Virtual-q (virtual-q)]

*** Bug 1549030 has been marked as a duplicate of this bug. ***

[Sledru (sledru)]

*** Bug 1549005 has been marked as a duplicate of this bug. ***

[Sledru (sledru)]

*** Bug 1549032 has been marked as a duplicate of this bug. ***

[Sledru (sledru)]

*** Bug 1549036 has been marked as a duplicate of this bug. ***

[Sledru (sledru)]

For all the CC folks: we are making progress

[Gijskruitbosch+bugs (gijskruitbosch+bugs)]

*** Bug 1549038 has been marked as a duplicate of this bug. ***

[longsonr (longsonr)]

*** Bug 1549039 has been marked as a duplicate of this bug. ***

[yfdyh000 (yfdyh000)]

*** Bug 1549040 has been marked as a duplicate of this bug. ***

[Ekr-0 (ekr-0)]

Update: We have rolled out a partial fix for this issue. We generated a new intermediate certificate with the same name/key but an updated validity window and pushed it out to users via Normandy (this should be most users). Users who have Normandy on should see their add-ons start working over the next few hours. We are continuing to work on packaging up the new certificate for users who have Normandy disabled.

[Guillaume-demesy (guillaume-demesy)]

See Also: -> https://webcompat.com/issues/30516, https://webcompat.com/issues/30483, https://webcompat.com/issues/30515, https://webcompat.com/issues/30511, https://webcompat.com/issues/30510, https://webcompat.com/issues/30507, https://webcompat.com/issues/30506, https://webcompat.com/issues/30500, https://webcompat.com/issues/30499, https://webcompat.com/issues/30487

[Russell Odom (russ-gloomytrousers)]

(In reply to Eric Rescorla (:ekr) from comment #57)

> [...] Users who have Normandy on should see their add-ons start working over the next few hours. [...]

Will we need to take any action, e.g. quit and restart, or check for add-on updates, or will it just start working?

[Russell Odom (russ-gloomytrousers)]

For everyone's info: we don't need to so anything *if* ""studies" is enabled (Firefox Preferences -> Privacy & Security -> Allow Firefox to install and run studies).

See https://discourse.mozilla.org/t/certificate-issue-causing-add-ons-to-be-disabled-or-fail-to-install/39047/12

Thanks Eddi for the tip.

[longsonr (longsonr)]

*** Bug 1549062 has been marked as a duplicate of this bug. ***

[longsonr (longsonr)]

*** Bug 1549065 has been marked as a duplicate of this bug. ***

[Gingerbread-man-2 (gingerbread-man-2)]

*** Bug 1549067 has been marked as a duplicate of this bug. ***

[longsonr (longsonr)]

*** Bug 1549071 has been marked as a duplicate of this bug. ***

[longsonr (longsonr)]

*** Bug 1549072 has been marked as a duplicate of this bug. ***

[longsonr (longsonr)]

*** Bug 1549077 has been marked as a duplicate of this bug. ***

[Jean-Louis Dupond (dupondje)]

An option is to enable 'Studies' and wait 6 hours until you receive the fix.

Another option seems to download & install the fix xpi manually.
Studies seem to get fetched from https://normandy.cdn.mozilla.net/api/v1/recipe/

And there you have a link to the fix:
https://storage.googleapis.com/moz-fx<email address hidden>

Download & install. Everything fixed!

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Virtual-q (virtual-q)]

*** Bug 1549087 has been marked as a duplicate of this bug. ***

[longsonr (longsonr)]

*** Bug 1549088 has been marked as a duplicate of this bug. ***

[longsonr (longsonr)]

*** Bug 1549090 has been marked as a duplicate of this bug. ***

[Guillaume-demesy (guillaume-demesy)]

See Also: → https://webcompat.com/issues/30522, https://webcompat.com/issues/30529, https://webcompat.com/issues/30530, https://webcompat.com/issues/30534, https://webcompat.com/issues/30536, https://webcompat.com/issues/30539

[Andre Klapper (a9016009)]

*** Bug 1549094 has been marked as a duplicate of this bug. ***

[Ehumphries (ehumphries)]

See https://blog.mozilla.org/addons/2019/05/04/update-regarding-add-ons-in-firefox/

[Ehumphries (ehumphries)]

*** Bug 1549105 has been marked as a duplicate of this bug. ***

[longsonr (longsonr)]

*** Bug 1549110 has been marked as a duplicate of this bug. ***

[Andre Klapper (a9016009)]

*** Bug 1549120 has been marked as a duplicate of this bug. ***

[Andre Klapper (a9016009)]

*** Bug 1549124 has been marked as a duplicate of this bug. ***

[longsonr (longsonr)]

*** Bug 1549137 has been marked as a duplicate of this bug. ***

[Guillaume-demesy (guillaume-demesy)]

See Also: → https://webcompat.com/issues/30556, https://webcompat.com/issues/30561, https://webcompat.com/issues/30562, https://webcompat.com/issues/30563, https://webcompat.com/issues/30564, https://webcompat.com/issues/30565, https://webcompat.com/issues/30566, https://webcompat.com/issues/30571, https://webcompat.com/issues/30572

[Orangefactor (orangefactor)]

124 failures in 56 pushes (2.214 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 124

Platform breakdown:
* linux64-qr: 16
* linux64: 36
* android-em-4-3-armv7-api16: 12
* linux32: 12
* osx-10-10: 1
* windows7-32: 2
* linux64-ccov: 5
* windows10-64: 2
* windows10-aarch64: 3
* osx-10-10-shippable: 1
* windows10-64-ccov: 8
* windows7-32-shippable: 1
* linux32-shippable: 8
* linux64-shippable-qr: 8
* linux64-shippable: 8
* windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1548973&startday=2019-05-04&endday=2019-05-04&tree=all

[Francesco-lodolo (francesco-lodolo)]

*** Bug 1549138 has been marked as a duplicate of this bug. ***

[Francesco-lodolo (francesco-lodolo)]

*** Bug 1549143 has been marked as a duplicate of this bug. ***

[Francesco-lodolo (francesco-lodolo)]

*** Bug 1549173 has been marked as a duplicate of this bug. ***

[Beanow (beanow)]

As per https://bugzilla.mozilla.org/show_bug.cgi?id=1549061 patches have been created as minor version releases, are these ready to be packaged?

[Virtual-q (virtual-q)]

*** Bug 1549118 has been marked as a duplicate of this bug. ***

[Gingerbread-man-2 (gingerbread-man-2)]

*** Bug 1549180 has been marked as a duplicate of this bug. ***

[Andre Klapper (a9016009)]

*** Bug 1549198 has been marked as a duplicate of this bug. ***

[Francesco-lodolo (francesco-lodolo)]

*** Bug 1549203 has been marked as a duplicate of this bug. ***

[Francesco-lodolo (francesco-lodolo)]

*** Bug 1549248 has been marked as a duplicate of this bug. ***

[Francesco-lodolo (francesco-lodolo)]

*** Bug 1549218 has been marked as a duplicate of this bug. ***

[Guillaume-demesy (guillaume-demesy)]

See Also: → https://webcompat.com/issues/30576, https://webcompat.com/issues/30578, https://webcompat.com/issues/30584, https://webcompat.com/issues/30588, https://webcompat.com/issues/30600, https://webcompat.com/issues/30603, https://webcompat.com/issues/30637, https://webcompat.com/issues/30638, https://webcompat.com/issues/30643, https://webcompat.com/issues/30645

[Orangefactor (orangefactor)]

124 failures in 4131 pushes (0.03 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:
* mozilla-central: 124

Platform breakdown:
* linux64-qr: 16
* linux64: 36
* android-em-4-3-armv7-api16: 12
* linux32: 12
* osx-10-10: 1
* windows7-32: 2
* linux64-ccov: 5
* windows10-64: 2
* windows10-aarch64: 3
* osx-10-10-shippable: 1
* windows10-64-ccov: 8
* windows7-32-shippable: 1
* linux32-shippable: 8
* linux64-shippable-qr: 8
* linux64-shippable: 8
* windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1548973&startday=2019-04-29&endday=2019-05-05&tree=all

[Francesco-lodolo (francesco-lodolo)]

*** Bug 1549352 has been marked as a duplicate of this bug. ***

[Markus-popp (markus-popp)]

(When) can http://ftp.mozilla.org/pub/firefox/candidates/ be opened again?

[Jonathan Kamens (jik)]

Please document here the root cause of this issue and what is being done to ensure that it does not happen again, or if that has already been documented elsewhere, please post a link to that documentation here. I've reviewed all the comments on this bug as well as the other information sources to which it links, and I don't see an explanation of the root cause or of steps being taken to prevent recurrence.

[Sjw (sjw)]

Please see https://support.mozilla.org/en-US/kb/add-ons-disabled-or-fail-to-install-firefox for current solutions and workarounds.
Status updates are posted here: https://blog.mozilla.org/addons/2019/05/04/update-regarding-add-ons-in-firefox/

[Olivier Tilloy (osomon)]

Fixed in 66.0.4+build3-0ubuntu1. Backports to stable releases are coming.

[madperson (madperson)]

*** Bug 1549679 has been marked as a duplicate of this bug. ***

[arQon (pf.arqon)]

This shouldn't be marked as "Fix Released" when the LTS's still don't have an update available. I realise the "real" work is done, but we can't have LTS's being treated as second-class citizens to the extent that this crippling defect doesn't even show up as an active issue to their users in Launchpad.

/opinion :)

[mp (m-p)]

arQon wrote:

"....This shouldn't be marked as "Fix Released" when the LTS's still don't have an update available. I realise the "real" work is done, but we can't have LTS's being treated as second-class citizens to the extent that this crippling defect doesn't even show up as an active issue to their users in Launchpad...."

Yes, it would be great to know when Update Manager will offer that fix/upgrade to 66.0.4.

It is somewhat (a little) ironic that Ubuntu and derivatives users will be the last to get this fixed: Windozers et al. are already sorted.

[Janne Snabb (snabb)]

I agree. At least an ETA would be nice instead of vague "releases are coming" for such a critical bug.

It has been long two working days waiting for a fix and still no idea when to expect it to be working again.

[Olivier Tilloy (osomon)]

The bug is marked "Fix Released" because a bug against a source package always targets the latest development series (in this case 19.10, where the bug is indeed fixed).

This doesn't mean the supported releases are being treated as second-class citizens. The updates will be available to everyone very soon.

While this is indeed a critical bug, it has largely been mitigated by Mozilla issuing a new certificate through Normandy, which is enabled by default in Ubuntu packages. So unless users have forcefully disabled Normandy, they shouldn't be affected by the bug anymore.

[eo (eo1000)]

While it may be true the "studies" feature which enabled the temporary fix is turned on by default it is a highly contentious feature which many security minded users (isn't that what most firefox users are) have disabled. Yes, it can be turned back on to get the "fix" but I choose not to compromise one form of security to regain another that was lost due to an unfortunate mozilla error. Just the same as I and many others choose to use LTS and the standard release system to minimize risk and maximize stability. I hope the maintainers can understand this.

[Kmaglione+bmo (kmaglione+bmo)]

*** Bug 1549200 has been marked as a duplicate of this bug. ***

[Manfred Hampl (m-hampl)]

Shouldn't there be sub-tasks for "Firefox on Ubuntu xenial" "Firefox on Ubuntu bionic" "Firefox on Ubuntu cosmic" and "Firefox on Ubuntu disco" to allow tracking the status for the older supported Ubuntu Releases?

[Launchpad Janitor (janitor)]

This bug was fixed in the package firefox - 66.0.4+build3-0ubuntu0.16.04.1

---------------
firefox (66.0.4+build3-0ubuntu0.16.04.1) xenial; urgency=medium

  * New upstream stable release (66.0.4build3) (LP: #1827727)

 -- Olivier Tilloy <email address hidden> Mon, 06 May 2019 12:07:09 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package firefox - 66.0.4+build3-0ubuntu0.18.04.1

---------------
firefox (66.0.4+build3-0ubuntu0.18.04.1) bionic; urgency=medium

  * New upstream stable release (66.0.4build3) (LP: #1827727)

 -- Olivier Tilloy <email address hidden> Mon, 06 May 2019 11:54:39 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package firefox - 66.0.4+build3-0ubuntu0.19.04.1

---------------
firefox (66.0.4+build3-0ubuntu0.19.04.1) disco; urgency=medium

  * New upstream stable release (66.0.4build3) (LP: #1827727)

 -- Olivier Tilloy <email address hidden> Mon, 06 May 2019 11:36:12 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package firefox - 66.0.4+build3-0ubuntu0.18.10.1

---------------
firefox (66.0.4+build3-0ubuntu0.18.10.1) cosmic; urgency=medium

  * New upstream stable release (66.0.4build3) (LP: #1827727)

 -- Olivier Tilloy <email address hidden> Mon, 06 May 2019 11:42:56 +0200

[Moz-jeka (moz-jeka)]

is the part
"[first mitigation completed, working on a second one]"
in the bug title meanigful in any way?

[Andrew-swan (andrew-swan)]

*** Bug 1569470 has been marked as a duplicate of this bug. ***