Firefox break when returning from console with wayland (nvidia) on ubuntu 21.10 beta

[MV]

(again snap firefox) Returning from console break all firefox tabs which have to be refreshed

mv@i56400:~$ firefox
[GFX1-]: glxtest: libEGL missing
[GFX1-]: glxtest: EGL test failed
xkbcommon: ERROR: failed to add default include path /usr/share/X11/xkb

###!!! [Parent][MessageChannel] Error: (msgtype=0x39008C,name=PContent::Msg_FlushTabState) Channel error: cannot send/recv

###!!! [Parent][MessageChannel] Error: (msgtype=0x390143,name=PContent::Msg_CommitBrowsingContextTransaction) Channel error: cannot send/recv

xkbcommon: ERROR: failed to add default include path /usr/share/X11/xkb
xkbcommon: ERROR: failed to add default include path /usr/share/X11/xkb
xkbcommon: ERROR: failed to add default include path /usr/share/X11/xkb
xkbcommon: ERROR: failed to add default include path /usr/share/X11/xkb
xkbcommon: ERROR: failed to add default include path /usr/share/X11/xkb
xkbcommon: ERROR: failed to add default include path /usr/share/X11/xkb

###!!! [Parent][MessageChannel] Error: (msgtype=0x39008C,name=PContent::Msg_FlushTabState) Channel error: cannot send/recv

###!!! [Parent][MessageChannel] Error: (msgtype=0x390143,name=PContent::Msg_CommitBrowsingContextTransaction) Channel error: cannot send/recv

###!!! [Parent][MessageChannel] Error: (msgtype=0x390144,name=PContent::Msg_AsyncMessage) Channel error: cannot send/recv

###!!! [Parent][MessageChannel] Error: (msgtype=0x39008C,name=PContent::Msg_FlushTabState) Channel error: cannot send/recv

###!!! [Parent][MessageChannel] Error: (msgtype=0x390143,name=PContent::Msg_CommitBrowsingContextTransaction) Channel error: cannot send/recv

###!!! [Parent][MessageChannel] Error: (msgtype=0x39008C,name=PContent::Msg_FlushTabState) Channel error: cannot send/recv

[Parent 6310, IPC I/O Parent] WARNING: FileDescriptorSet destroyed with unconsumed descriptors: file /build/firefox/parts/firefox/build/ipc/chromium/src/chrome/common/file_descriptor_set_posix.cc:19

After that, I logged under x11 (for test) but wasn't able to launch firefox:
 Error: cannot open display: :0

I had to reboot and the problem occurs only with wayland.

[W-jan-k (w-jan-k)]

Created attachment 9242948
aboutsupport.txt

Gnome Wayland, Debian Testing, Intel

(Darkspirit from bug 1726510 comment 36)
> $ sudo snap remove firefox
> $ sudo snap install firefox
> $ snap run firefox

> WebGL and WebRender work according to about:support, but https://webglsamples.org/aquarium/aquarium.html does not work.
> > It does not appear your computer supports WebGL.
> > Click here for more information.
> > Status: WebGL creation failed: * tryNativeGL (FEATURE_FAILURE_NO_DISPLAY) * Exhausted GL driver options. (FEATURE_FAILURE_WEBGL_EXHAUSTED_DRIVERS)
>
> No word on about:support about this FEATURE_FAILURE_NO_DISPLAY thing.
>
> $ sudo cat /var/log/syslog | grep denied
> > Sep 26 23:36:58 darkspirit-laptop kernel: [41105.417113] audit: type=1400 audit(1632692218.808:14756): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/proc/sys/dev/i915/perf_stream_paranoid" pid=49327 comm="firefox" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
> > Sep 26 23:36:59 darkspirit-laptop kernel: [41105.850276] audit: type=1400 audit(1632692219.240:14757): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/proc/sys/dev/i915/perf_stream_paranoid" pid=49258 comm="Renderer" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

> ---
> Manually installed firefox - without dangerous, but with acking - works the same as official firefox:
> $ sudo snap remove firefox
> $ sudo snap ack firefox_595.assert
> $ sudo snap install firefox_595.snap
> $ snap run firefox

> https://webglsamples.org/aquarium/aquarium.html
> > It does not appear your computer supports WebGL.
> Click here for more information.
> Status: WebGL creation failed: * tryNativeGL (FEATURE_FAILURE_NO_DISPLAY) * Exhausted GL driver options. (FEATURE_FAILURE_WEBGL_EXHAUSTED_DRIVERS)
>
> /var/log/syslog:
> Same two errors as above and this one is new:
> > Sep 26 23:21:31 darkspirit-laptop kernel: [40177.949121] audit: type=1400 audit(1632691291.299:14602): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/proc/46373/environ" pid=46215 comm=427265616B70616420536572766572 requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000

[W-jan-k (w-jan-k)]

$ snap info firefox
> channels:
> latest/stable: 92.0-3 2021-09-09 (595) 159MB -
> latest/candidate: 92.0.1-1 2021-09-23 (625) 159MB -
> latest/beta: 93.0b9-1 2021-09-24 (628) 155MB -
> latest/edge: ↑
> esr/stable: 78.14.0esr-1 2021-09-07 (591) 148MB -
> esr/candidate: 91.1.0esr-1 2021-09-10 (603) 158MB -
> esr/beta: ↑
> esr/edge: ↑

$ sudo snap remove firefox --purge; sudo snap install firefox --channel=esr/stable; snap run firefox https://webglsamples.org/aquarium/aquarium.html
> firefox (esr/stable) 78.14.0esr-1 from Mozilla✓ installed

works
Edit: because it uses the official Mozilla config (GLX/Xwayland)

$ sudo snap remove firefox; sudo snap install firefox --channel=esr/candidate; snap run firefox https://webglsamples.org/aquarium/aquarium.html
> firefox (esr/candidate) 91.1.0esr-1 from Mozilla✓ installed

broken

$ sudo snap remove firefox; sudo snap install firefox --channel=latest/candidate; snap run firefox https://webglsamples.org/aquarium/aquarium.html
> firefox (candidate) 92.0.1-1 from Mozilla✓ installed

broken

$ sudo snap remove firefox; sudo snap install firefox --channel=latest/beta; snap run firefox https://webglsamples.org/aquarium/aquarium.html
> firefox (beta) 93.0b9-1 from Mozilla✓ installed

broken

[W-jan-k (w-jan-k)]

93.0b9: Problem does not occur with GLX/Xwayland. Problem also occurs with EGL/Xwayland.
$ DISABLE_WAYLAND=1 MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html

[W-jan-k (w-jan-k)]

Tested:
* GLX is used on Gnome X11/Nvidia: fine
* EGL is used on Wayland because MOZ_ENABLE_WAYLAND is used (either Snap sets it as default or xwayland is not available)
* As of 94, EGL will also be used on X11 (bug 1695933).

This bug is also reproducible with Firefox Snap 92 when setting gfx.x11-egl.force-enabled to true on Gnome X11/Nvidia, but I don't see any DENIED in syslog.

[Robert Mader (robert.posteo)]

Hm, this reminds me of bug 1700601 and bug 1696691.

For the record: I'm going to land bug 1732002 to give us some more time to figure out stuff like this. Nvidia does not yet default to Wayland sessions on Ubuntu so setups with default configuration remain unaffected.

[W-jan-k (w-jan-k)]

Created attachment 9243285
about:support from Firefox snap 92, Gnome X11, Intel

Also reproducible with Firefox Snap 92 and gfx.x11-egl.force-enabled=true on Gnome X11/Intel. No crash report.
Edit: Attachment is XFCE X11, but Gnome X11 looks the same.

glxtest works without problem, WebGL info on about:support is correct. It's just that WebGL doesn't work on websites.

[Robert Mader (robert.posteo)]

Ah I see. Sounds like it needs to get fixed by whoever maintains the Snap sandbox - likely little we can do in FF.

[Olivier Tilloy (osomon)]

Pasting a comment of mine from bug 1726510:

« I instrumented the corresponding code and rebuilt the snap: in GetAndInitDisplay() (gfx/gl/GLLibraryEGL.cpp), egl.fGetDisplay() (eglGetDisplay()) always returns null. This requires further investigation. »

[Olivier Tilloy (osomon)]

The apparmor denials on "/proc/sys/dev/i915/perf_stream_paranoid" mentioned in the description are red herrings: I can see them too, but if I edit the snap's generated apparmor profile (`/var/lib/snapd/apparmor/profiles/snap.firefox.firefox`) to allow read access to that file and reload it, the denials go away but the problem with WebGL persists.

[Olivier Tilloy (osomon)]

I instrumented further the code, and here are my observations so far:

When launching firefox, the very first call to `GLLibraryEGL::CreateDisplay()` returns a valid EGL display. This is being called from `RenderThread::CreateGLContextEGL()`, from the app's main process.

Subsequent calls to `GLLibraryEGL::CreateDisplay()` follow a different code path (from `GLContextProviderEGL::CreateHeadless()`, from child (content) processes), and those invariably return `null`, which results in WebGL not working.

[Olivier Tilloy (osomon)]

Running the app with `MOZ_DISABLE_CONTENT_SANDBOX=1` "fixes" WebGL, so there is something in the content sandbox that prevents access to an EGL display.

[W-jan-k (w-jan-k)]

Disabling widget.dmabuf-webgl.enabled doesn't make a difference.

[W-jan-k (w-jan-k)]

Summary:
STR on Debian Testing:
$ sudo apt install snapd
$ sudo systemctl start snapd

$ sudo snap install firefox

When using a Wayland session:
* EGL/Wayland backend:
  broken: `$ snap run firefox https://webglsamples.org/aquarium/aquarium.html`
  works: `$ MOZ_DISABLE_CONTENT_SANDBOX=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html`
* EGL/Xwayland:
  broken: `$ DISABLE_WAYLAND=1 MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html`
  works: `$ MOZ_DISABLE_CONTENT_SANDBOX=1 DISABLE_WAYLAND=1 MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html`
* fine with glx/xwayland: `$ DISABLE_WAYLAND=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html`

When using an X11 session:
* EGL/X11:
  broken: `$ MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html`
  works: `$ MOZ_DISABLE_CONTENT_SANDBOX=1 MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html`
* fine with GLX/X11: `$ snap run firefox https://webglsamples.org/aquarium/aquarium.html`

[Olivier Tilloy (osomon)]

Also, reducing `security.sandbox.content.level` to `2` (from its default value of `4`) in `about:config` makes the problem go away.

[Olivier Tilloy (osomon)]

And setting `security.sandbox.content.read_path_whitelist` to `$SNAP/` (e.g. `/snap/firefox/595/`, the trailing slash is important), also makes the problem go away, confirming that the content process sandboxing is what prevents the WebGL code from accessing the EGL library.

[Olivier Tilloy (osomon)]

I'm tempted to patching `SandboxBrokerPolicyFactory::InitContentPolicy()` to call `policy->AddPath(rdonly, "$SNAP/")` when firefox is running as a snap. This will obviously require a thorough security review.

[Gpascutto (gpascutto)]

I would expect https://searchfox.org/mozilla-central/source/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp#499 to pick up libraries shipped alongside Firefox. I guess what's happening here is that the library is shipped in the snap (but not in the default system), not next to the binary, and then "something" is done to make the dynamic linker pick it up?

Our sandbox knows about LD_LIBRARY_PATH and such https://searchfox.org/mozilla-central/rev/c3d7964c593e0bedabea2fea0b35ba243cf9e696/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp#258 but I guess this is using something different?

In general readonly permission to trusted system library dirs should not be a security concern.

[Olivier Tilloy (osomon)]

Indeed, the library is shipped by the snap. To be exact, it is shipped by the platform snap that the firefox snap uses as its base (gnome-3-38-2004), and the snap sees it at `$SNAP/gnome-platform/usr/lib/x86_64-linux-gnu/libEGL.so`. The snap's launcher modifies `LD_LIBRARY_PATH` accordingly. This is the value for a webcontent (child) process (where `x21` is the snap's revision, because I manually installed an instrumented build):

    LD_LIBRARY_PATH=/snap/firefox/x21/usr/lib/firefox:/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/firefox/x21/usr/lib:/snap/firefox/x21/usr/lib/x86_64-linux-gnu:/snap/firefox/x21/gnome-platform/lib/x86_64-linux-gnu:/snap/firefox/x21/gnome-platform/usr/lib/x86_64-linux-gnu:/snap/firefox/x21/gnome-platform/usr/lib:/snap/firefox/x21/gnome-platform/lib:/snap/firefox/x21/gnome-platform/usr/lib/x86_64-linux-gnu/dri:/var/lib/snapd/lib/gl:/snap/firefox/x21/gnome-platform/usr/lib/x86_64-linux-gnu/libunity:/snap/firefox/x21/gnome-platform/usr/lib/x86_64-linux-gnu/pulseaudio

The path in question is there, so it's not immediately clear to me why it's not being added to the policy's list of readonly paths. Maybe the call to `realpath(…)` doesn't work well with the snap's confinement?

[Olivier Tilloy (osomon)]

I was able to narrow down the one missing path (required to make WebGL work) to `$SNAP/gnome-platform/usr/share/glvnd/egl_vendor.d/`.

That path contains one single file (`50_mesa.json`):

    {
        "file_format_version" : "1.0.0",
        "ICD" : {
            "library_path" : "libEGL_mesa.so.0"
        }
    }

[Olivier Tilloy (osomon)]

So the problem is not with `LD_LIBRARY_PATH` or with how the sandbox parses its value to add readonly permissions, it is with a configuration file that's located in a place that the policy doesn't allow read access to by default, and which contains information for EGL to locate the right implementation to load.

In that light, and considering that we might see other similar bugs (caused by configuration files for certain libs not being readable) in the future, my suggestion in comment 15 seems to be a valid and future-proof solution.

[Olivier Tilloy (osomon)]

Created attachment 9244377
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

[Pulsebot (pulsebot)]

Pushed by <email address hidden>:
https://hg.mozilla.org/integration/autoland/rev/f52ebe8f3b52
Allow read access to files under $SNAP/ in the webcontent sandbox. r=gcp

[Mlaza (mlaza)]

https://hg.mozilla.org/mozilla-central/rev/f52ebe8f3b52

[Release-mgmt-account-bot (release-mgmt-account-bot)]

The patch landed in nightly and beta is affected.
:olivier, is this bug important enough to require an uplift?
If not please set `status_beta` to `wontfix`.

For more information, please visit [auto_nag documentation](https://wiki.mozilla.org/Release_Management/autonag#uplift_beta.py).

[Olivier Tilloy (osomon)]

Yes, this definitely needs to be uplifted to beta.

[Olivier Tilloy (osomon)]

Comment on attachment 9244377
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

### Beta/Release Uplift Approval Request
* **User impact if declined**: WebGL doesn't work for users of the firefox snap in Wayland sessions.
* **Is this code covered by automated tests?**: Yes
* **Has the fix been verified in Nightly?**: Yes
* **Needs manual test from QE?**: No
* **If yes, steps to reproduce**:
* **List of other uplifts needed**: None
* **Risk to taking this patch**: Low
* **Why is the change risky/not risky? (and alternatives if risky)**: Not risky because only the snap package is affected (the sandbox allows one additional folder for readonly access if the $SNAP environment variable is set).
* **String changes made/needed**:

[Olivier Tilloy (osomon)]

I can confirm the problem:
 - Ubuntu 21.10 (to be released this week), Wayland session (the default), firefox snap (the default)
 - Open tabs and browse to any website in firefox (e.g. http://example.org)
 - Press Ctrl+Alt+F3 to switch to a VT
 - Press Ctrl+Alt+F2 to switch back to your desktop session

Expected result: firefox behaves normally

Actual result: the current tab in firefox crashes and offers to report to crash to Mozilla

[Olivier Tilloy (osomon)]

And interestingly, the patch for https://bugzilla.mozilla.org/show_bug.cgi?id=1732580 appears to fix the problem (actually, I tested the workaround, which is to set `security.sandbox.content.read_path_whitelist` in `about:config` to the value of $SNAP for that revision of firefox, e.g. "/snap/firefox/631/" (the trailing slash matters)).

[Olivier Tilloy (osomon)]

This patch also appears to fix another problem that happens only in Wayland sessions, and was reported in Ubuntu (against the upcoming 21.10 release which ships the Firefox snap by default): https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/1946599.

[Ryanvm (ryanvm)]

Comment on attachment 9244377
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

Approved for 94.0b5.

[Ryanvm (ryanvm)]

https://hg.mozilla.org/releases/mozilla-beta/rev/111441e21e18

[Robie Basak (racb)]

I can reproduce this issue with this snap:

installed: 93.0-1 (631) 157MB -

In my case, it's standby/resume that triggers it. However Ctrl+Alt+F3 then Ctrl+Alt+F2 does also. I suggest that the broken standby/resume use case makes this issue much more significant from a user impact perspective.

Olivier's security.sandbox.content.read_path_whitelist workaround works perfectly. Thank you!

[Ryanvm (ryanvm)]

Does this need an ESR91 approval request also?

[Olivier Tilloy (osomon)]

Yes indeed, this is clearly a candidate for an ESR91 uplift. Thanks for the suggestion Ryan.

[Olivier Tilloy (osomon)]

Comment on attachment 9244377
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

### ESR Uplift Approval Request
* **If this is not a sec:{high,crit} bug, please state case for ESR consideration**: Snap packaging has changed significantly between ESR78 and ESR91. In ESR78 WebGL works in Wayland sessions, whereas it doesn't in ESR91, making it a very visible functional regression.
* **User impact if declined**: WebGL doesn't work for users of the firefox snap in Wayland sessions.
* **Fix Landed on Version**: 94.0b5
* **Risk to taking this patch**: Low
* **Why is the change risky/not risky? (and alternatives if risky)**: Not risky because only the snap package is affected (the sandbox allows one additional folder for readonly access if the $SNAP environment variable is set).
* **String or UUID changes made by this patch**:

[Olivier Tilloy (osomon)]

Thanks for the feedback Robie. I agree this makes the problem even worse from a user perspective.

Fortunately the fix has been approved for the upcoming firefox 94.0, and uplifting to the ESR91 branch has been requested too.

[Ryanvm (ryanvm)]

Comment on attachment 9244377
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

Approved for 91.3esr.

[Ryanvm (ryanvm)]

Looks like this depends on bug 1718084, though?

[Olivier Tilloy (osomon)]

Yes, I hadn't realized this, but it depends on bug 1718084 indeed.

[Olivier Tilloy (osomon)]

If pulling in the patches from bug 1718084 is deemed too risky for ESR91, I can rework this patch to not depend on them.
That would imply no unit tests, but on the upside the patch would be trivial.

[Ryanvm (ryanvm)]

I think we should take it unless there's a strong reason I'm missing. It's been baking for a couple months now, we're early in the ESR91 lifespan, and I won't be surprised if we eventually need to backport other patches that depend on it.

[Ryanvm (ryanvm)]

https://hg.mozilla.org/releases/mozilla-esr91/rev/13c638762db7

[MV (mvidal)]

snap firefox 94 beta solve the problem when coming back from console but nor the webgl problem neither the extensions problem.

[Olivier Tilloy (osomon)]

I assume by "extensions problem" you mean https://bugzilla.mozilla.org/1661935 ? That's an entirely separate issue.

Regarding WebGL, that's unexpected. Can you share the contents of about:support, and the error message you get when you browse to e.g. http://webglsamples.org/aquarium/aquarium.html ?

[MV (mvidal)]

Yes for the extension problem.

Same error when launching firefox from terminal :
mv@i56400:~$ firefox
[GFX1-]: glxtest: libEGL missing
[GFX1-]: glxtest: EGL test failed

Aquarium page write :
It does not appear your computer supports WebGL.
Click here for more information.

[W-jan-k (w-jan-k)]

(In reply to Darkspirit from comment #1)
> $ sudo snap remove firefox; sudo snap install firefox --channel=latest/beta; snap run firefox https://webglsamples.org/aquarium/aquarium.html
> > firefox (beta) 93.0b9-1 from Mozilla✓ installed
>
> broken

Debian Testing, Gnome Wayland, Intel

sudo snap remove firefox; sudo snap install firefox --channel=latest/beta

EGL/Wayland works:
snap run firefox https://webglsamples.org/aquarium/aquarium.html

EGL/Xwayland works:
DISABLE_WAYLAND=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html

[W-jan-k (w-jan-k)]

*** Bug 1732582 has been marked as a duplicate of this bug. ***

[Olivier Tilloy (osomon)]

Can you please browse to about:support, copy the raw contents of the page and attach them here as a text file? Thanks!

[MV (mvidal)]

here the ugly raw about

[MV (mvidal)]

I tried to remove and then reinstall firefox beta. "snap remove firefox" removed all my firefox data. The "Snap run firefox" command did not allow me to retrieve this data from the ~/.mozilla directory.

I deleted the firefox snap and installed the deb version 93.

[Olivier Tilloy (osomon)]

This is relevant:

  Pilote WebGL 1 - Rendu WebGL creation failed:
  * WebglAllowWindowsNativeGl:false restricts context creation on this system. ()
  * Exhausted GL driver options. (FEATURE_FAILURE_WEBGL_EXHAUSTED_DRIVERS)

What graphics card do you have, and which drivers? Does WebGL work in the firefox deb?

You need to make sure that ~/snap/firefox/common/.mozilla doesn't exist before running the snap, otherwise it won't attempt to do the profile import.

[MV (mvidal)]

Every thing works fine with the deb firefox : webgl and gnome extensions

NVIDIA Corporation -- NVIDIA GeForce GTX 960/PCIe/SSE2
Pilote WebGL 1 - Version 4.6.0 NVIDIA 470.74

[W-jan-k (w-jan-k)]

*** Bug 1737737 has been marked as a duplicate of this bug. ***

[Mathew Hodson (mhodson)]

Fixed in Firefox 94

[Andreas (andreas-rabus)]

Not sure if related, but googling led me here ...
Slightly different on a laptop with intel/nvidia dGpu (optimus...) and property nvidia 470 drivers:

I assume its an issue with finding or selecting the right egl library

```
MOZ_DISABLE_CONTENT_SANDBOX=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html
[GFX1-]: glxtest: libEGL missing
[GFX1-]: glxtest: EGL test failed
[GFX1-]: More than 1 GPU vendor detected via PCI, cannot deduce vendor
[GFX1-]: PCI candidate 0x8086/0x0a16
[GFX1-]: PCI candidate 0x10de/0x1290
glxtest
^C
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.

```

[Olivier Tilloy (osomon)]

Andreas, please file a new bug to track this separately. Thanks!

[Lissyx+mozillians (lissyx+mozillians)]

FTR, the `AddFutureDir` we had to land on that patch was finally removed in bug 1750539.