Firefox 89 won't load pages due to many fonts

[shemgp]

Upgraded Firefox to 89 and it shows a crash tab every time because I have 22000+ fonts. In safe-mode it works, but refreshed firefox still has the bug.

When starting firefox from the terminal it shows messages like:

[Parent 42669, Main Thread] WARNING: Too many file descriptors for one message!: file /build/firefox-Sx7Sor/firefox-89.0+build2/ipc/chromium/src/chrome/common/ipc_message_utils.h:414

This only happens since Firefox 89 and not in 88.

ProblemType: Bug
DistroRelease: Ubuntu 21.04
Package: firefox 89.0+build2-0ubuntu0.21.04.1
ProcVersionSignature: Ubuntu 5.11.0-18.19-generic 5.11.17
Uname: Linux 5.11.0-18-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
AddonCompatCheckDisabled: False
ApportVersion: 2.20.11-0ubuntu65.1
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC2: shemgp 5850 F.... pulseaudio
 /dev/snd/controlC0: shemgp 5850 F.... pulseaudio
 /dev/snd/controlC1: shemgp 5850 F.... pulseaudio
BuildID: 20210527174632
CasperMD5CheckResult: unknown
Channel: Unavailable
CurrentDesktop: GNOME
Date: Thu Jun 3 19:36:30 2021
EcryptfsInUse: Yes
Extensions: extensions.sqlite corrupt or missing
ForcedLayersAccel: False
IncompatibleExtensions: Unavailable (corrupt or non-existant compatibility.ini or extensions.sqlite)
InstallationDate: Installed on 2015-09-28 (2074 days ago)
InstallationMedia: Ubuntu 15.04 "Vivid Vervet" - Release amd64 (20150422)
Locales: extensions.sqlite corrupt or missing
PrefErrors: Unexpected character ',' before close parenthesis @ /usr/lib/firefox/omni.ja:greprefs.js:355
PrefSources: prefs.js
Profiles: Profile0 (Default) - LastVersion=89.0/20210527174632
RunningIncompatibleAddons: False
SourcePackage: firefox
Themes: extensions.sqlite corrupt or missing
UpgradeStatus: Upgraded to hirsute on 2021-02-16 (106 days ago)
dmi.bios.date: 09/25/2019
dmi.bios.release: 2.75
dmi.bios.vendor: LENOVO
dmi.bios.version: GCETB5WW (2.75 )
dmi.board.asset.tag: Not Available
dmi.board.name: 34361A0
dmi.board.vendor: LENOVO
dmi.board.version: Not Defined
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.ec.firmware.release: 1.15
dmi.modalias: dmi:bvnLENOVO:bvrGCETB5WW(2.75):bd09/25/2019:br2.75:efr1.15:svnLENOVO:pn34361A0:pvrThinkPadX220:rvnLENOVO:rn34361A0:rvrNotDefined:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.family: ThinkPad X220
dmi.product.name: 34361A0
dmi.product.sku: LENOVO_MT_3436
dmi.product.version: ThinkPad X220
dmi.sys.vendor: LENOVO

[Jfkthame (jfkthame)]

Currently gfx.e10s.font-list.shared is enabled by default on Nightly/early Beta, but disabled for release.

To reduce content-process memory footprint and startup time, we should get this enabled all the way to the Release channel.

[Cpeterson-3 (cpeterson-3)]

How much memory does the shared font-list save per content process?

Since Fission will have launch many more content processes than e10s, the shared font-list might be nice to have for Fission MVP.

[Cpeterson-3 (cpeterson-3)]

@ Jonathan, SearchAllFontsForChar crash bug 1667977 and src:local() bug 1694123 are blocking this bug. I see you already have a patch up to fix bug 1694123. Does bug 1667977 also need to be fixed before we can let the shared font-list ride the trains to late Beta and Release?

Fission's current perf measurements are all from Nightly, so already include any perf and memory benefits from the shared font-list. Tracking for Fission M7 Beta milestone because we'd like to have the shared font-list in late Beta and when we ship to Release.

kmag estimates the shared font-list might save about 500 KB in each content process.

[Jfkthame (jfkthame)]

Bug 1667977 currently means that on Windows there's a possibility of content processes crashing if the user installs/removes a bunch of fonts while the browser is running (so we get a flurry of notifications that trigger repeated, overlapping updates to our font list). This is probably not a common situation -- most users aren't installing and removing fonts frequently in the midst of a browser session -- but still, I'd be reluctant to ship something that is known to be crashy in such a scenario.

So I'd definitely prefer to have bug 1667977 fixed before it rides to Release. I've been attempting to debug but it's proving difficult to track down the root cause at the moment. (Not having a tool equivalent to rr or pernosco on Windows hurts...) I'll try to spend some more time digging into it this week.

[Cpeterson-3 (cpeterson-3)]

(In reply to Jonathan Kew (:jfkthame) from comment #3)
> So I'd definitely prefer to have bug 1667977 fixed before it rides to Release. I've been attempting to debug but it's proving difficult to track down the root cause at the moment. (Not having a tool equivalent to rr or pernosco on Windows hurts...) I'll try to spend some more time digging into it this week.

A fix for SearchAllFontsForChar crash bug 1667977 probably won't be ready for 88 Beta, so I will move this bug from Fission milestone M7 to M8.

btw, the reporter of bug 1667977 says the crash is also reproducible on Linux, so you may be able to catch the crash in rr on Linux.

[Jfkthame (jfkthame)]

This has been enabled on Nightly since 82, and also early Beta since 83. Now that a fix for bug 1667977 has landed, I think we should remove the early-beta condition and let this go all the way to release.

[Jfkthame (jfkthame)]

Created attachment 9214874
Bug 1694174 - Pref-on the shared font-list by default for all channels. r=#layout-reviewers

[Pulsebot (pulsebot)]

Pushed by <email address hidden>:
https://hg.mozilla.org/integration/autoland/rev/622e8fe7e539
Pref-on the shared font-list by default for all channels. r=emilio

[Cbrindusan (cbrindusan)]

https://hg.mozilla.org/mozilla-central/rev/622e8fe7e539

[shemgp (shemgp)]

Just to add:

fc-list | awk 'BEGIN{FS=":"}; {print $2}' | sort | uniq | wc -l
4276

fc-list | wc -l
12881

because of snaps that remounts /run/user/1000/doc/3285b9f/.fonts/

[shemgp (shemgp)]

More info. So I lowered down my fonts to 12881, it was 22k something by removing duplicate fonts in the ~/.fonts/google-fonts folder and Firefox loads now.

[Pascalc (pascalc)]

Jonathan, we have already 4 regressions reported in 4 days since we shipped this feature in 89, among them 2 marked as S2 + a medium crasher. Should we consider deactivate this feature in a 89 dot release?

[Jfkthame (jfkthame)]

If we're going to do a dot release, we might want to consider deactivating on Linux due to bug 1714282, which I think is actually the most serious issue we've seen. A patch has landed there (which I hope to uplift to 90), but for 89, I'd consider disabling it.

The crash on Windows (bug 1689379) is probably not too big of a concern; it's specifically at shutdown, and it's not really a "bug" that's crashing us, it's a timeout because directwrite is being slow, which we then detect as a hang and deliberately crash. Again, a patch has just landed, which I hope to uplift to 90. I suppose for 89, though, deactivating via the pref would be a better user experience.

I think the other issues were categorized as S2 simply due to being "regressions", but may not be so serious - there are several examples where users have odd font configurations, and have experienced a change because our behavior arguably became *more* correct; I don't think these type of edge cases would justify reverting the feature.

So - yes, I think deactivating in an 89.x dot-release, while we wait for bug 1689379 and bug 1714282 to be fixed in 90, may be a good option here.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Kaneŝina Rinna (rinna)]

I am affected by this bug too, with 18290 fonts.
`Web Content` processes segfault as soon as they are spawned.

Attached is a log of all the messages when this happens.

[Kaneŝina Rinna (rinna)]

Setting the Firefox preference `gfx.e10s.font-list.shared` to `false` works around this bug for me.

[Jfkthame (jfkthame)]

Created attachment 9226420
[patch for 89.x dot-release only] Disable on Linux/Windows

I'm not sure how to manage a patch intended for dot-release-only in phabricator, so simply attaching it here.

Approval Request Comment
[Feature/Bug causing the regression]: 1694174
[User impact if declined]: see regressions: bitmap font issues on linux; crashes for linux users with huge font directories; possible windows shutdown hang
[Is this code covered by automated tests?]: no
[Has the fix been verified in Nightly?]: simply reverts to earlier behavior
[Needs manual test from QE? If yes, steps to reproduce]: -
[List of other uplifts needed for the feature/fix]: -
[Is the change risky?]: no
[Why is the change risky/not risky?]: revert to previous code via pref flip
[String changes made/needed]: -

[Pascalc (pascalc)]

Disabled on 89
https://hg.mozilla.org/releases/mozilla-release/rev/a764fa6c10f9

[madbiologist (me-again)]

The workaround from comment #6 has been implemented by default in Firefox 89.0.1.

Can you retest and confirm?

[shemgp (shemgp)]

I can confirm that in 89.0.1, gfx.e10s.font-list.shared is set to false and the bug does not appear. When I set it to true again, it appears.