Xwayland crash in gdm as screen off and touchscreen detach.

[Yuan-Chen Cheng]

[Impact]

 * Gdm crashed and user need to be an expert to know how to work around,
   or need to reboot to fix.

 * OEM projects got certain SKU that can reliably reproduce this issue.

 * A merged patch in upstream fix the race condition.

[Test Plan]

 * Use the SKU that can reliably reproduce this issue.
 * Do a cold boot, and wait for 5 mins. The monitor should turned-black
   but showing the Machine Vendor logo that was previously
   displayed by plymouth.
 * Try to dog-food the deb from the proposed channel and make sure
   all input events still properly propagate.

[Where problems could occur]

 * If the patch is not properly working as expected, per the file this
   patch touch, the user desktop session could stop responding to
   mouse, keyboard, and touch screen input.

[Other Info]

 * Can't think of any for now.

----

This is a reproducible issue. Step to reproduce
1. cold boot into gdm login screen.
2. wait for 5 mins, and the screen will be turned off. (also observed that the touchscreen USB device drop of a USB bus on the target machine)

Note: this bug seems to be timing sensitive. I got an OEM project machine that can reproduce this issue (reproducible on several machines with the same HW design). The machine is using ADL CPU, which needs a patched 5.14 kernel. So I can't reproduce this on impish. With the backported mutter, I verified an upstream patch seems to fix this issue.

Expected result:
The screen just turns black.

Actual result:
Saw Vendor boot logo on the monitor (previous draw by plymouth during booting)

Analysis as it happens:
1. Saw "LGDisplay Incell Touch" drop off the USB bus.
   Kernel message like: "usb 1-9: USB disconnect, device number 6"
   as the screen turned black.
2. There is no switch on virtual tty.
3. journal log from Xwayland "wl_display@1: error 0: invalid object 24"
4. Xwayland and gnome-shell process both die as this happens.
5. Try to add sleep(30) to OsVendorFatalError(), then both Xwayland and
   gnome-shell stop there for 30 more seconds.
6. backtrace as Xwayland crashed:

```
#0 0x00007f447b8e818b in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f447b8c7859 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x0000564374403f30 in OsAbort () at ../../../../os/utils.c:1351
#3 0x0000564374409369 in AbortServer () at ../../../../os/log.c:872
#4 0x000056437440a1ca in FatalError (f=f@entry=0x564374416266 "%s") at ../../../../os/log.c:1010
#5 0x000056437429c8b0 in xwl_log_handler (format=<optimized out>, args=<optimized out>) at ../../../../../hw/xwayland/xwayland.c:1312
#6 0x00007f447c3d027e in wl_log (fmt=fmt@entry=0x7f447c3d121a "%s@%u: error %d: %s\n") at ../src/wayland-util.c:404
#7 0x00007f447c3cb78b in display_handle_error (data=<optimized out>, display=0x56437639f000, object=0x56437639f000, code=0, message=<optimized out>)
    at ../src/wayland-client.c:911
#8 0x00007f447b7c0ff5 in ?? () from /lib/x86_64-linux-gnu/libffi.so.7
#9 0x00007f447b7c040a in ?? () from /lib/x86_64-linux-gnu/libffi.so.7
#10 0x00007f447c3cf3a8 in wl_closure_invoke (closure=closure@entry=0x564376e57350, flags=flags@entry=1, target=<optimized out>,
    target@entry=0x56437639f000, opcode=opcode@entry=0, data=<optimized out>) at ../src/connection.c:1018
#11 0x00007f447c3cbc48 in dispatch_event (display=display@entry=0x56437639f000, queue=<optimized out>) at ../src/wayland-client.c:1445
#12 0x00007f447c3cd1ef in dispatch_queue (queue=0x56437639f0d0, display=0x56437639f000) at ../src/wayland-client.c:1584
#13 wl_display_dispatch_queue_pending (display=0x56437639f000, queue=0x56437639f0d0) at ../src/wayland-client.c:1833
#14 0x00007f447c3cd280 in wl_display_dispatch_pending (display=<optimized out>) at ../src/wayland-client.c:1896
#15 0x000056437429c9bb in xwl_read_events (xwl_screen=0x564376399dd0) at ../../../../../hw/xwayland/xwayland.c:957
#16 0x0000564374401c61 in ospoll_wait (ospoll=0x56437638f4a0, timeout=<optimized out>) at ../../../../os/ospoll.c:657
#17 0x00005643743faa23 in WaitForSomething (are_ready=0) at ../../../../os/WaitFor.c:208
#18 0x00005643743ca4a7 in Dispatch () at ../../../../include/list.h:220
#19 0x00005643743ce794 in dix_main (argc=16, argv=0x7ffc44d49298, envp=<optimized out>) at ../../../../dix/main.c:276
#20 0x00007f447b8c90b3 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#21 0x000056437429c60e in _start ()```

[Yuan-Chen Cheng (ycheng-twn)]

upstream bug: https://gitlab.freedesktop.org/xorg/xserver/-/issues/1239

[Daniel van Vugt (vanvugt)]

This was from focal so Xwayland comes from the 'xorg-server' source package.

In hirsute and later it changed to the 'xwayland' source package.

[Daniel van Vugt (vanvugt)]

Oops. That link suggests it's a Wayland server bug in mutter. Fixed in tag 41.0

[Yuan-Chen Cheng (ycheng-twn)]

Currently we are doing a little bit more test to make sure we this patch does work.

For now, the versions of mutter:
 jammy: 40.5-1ubuntu2
 impish: 40.5-1ubuntu2
Debian:
 sid: 41.0-5

[Daniel van Vugt (vanvugt)]

Expect it to still fail in 40.5-1ubuntu2 as the expected fix (mutter!77) is only in version 41.0.

[Yuan-Chen Cheng (ycheng-twn)]

debdiff for jammy.

also uploaded to ppa: https://launchpad.net/~ycheng-twn/+archive/ubuntu/mutter-f3

After installing jammy, do some quick trial use of desktop environment with the deb in the ppa works fine so far.

Note: we'd need to SRU back to focal to fulfill the OEM project requirement.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "mutter_40.5-1ubuntu3.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Yuan-Chen Cheng (ycheng-twn)]

Also create MP in https://salsa.debian.org/gnome-team/mutter/-/merge_requests/76

[Daniel van Vugt (vanvugt)]

In future debdiffs, don't forget to add:

Bug-Ubuntu: https://launchpad.net/bugs/1948894

[Yuan-Chen Cheng (ycheng-twn)]

debdiff for hirsute. Note I back-port a chunk from the patch.

[Yuan-Chen Cheng (ycheng-twn)]

add Bug-Ubuntu in salsa git, and re-upload jammy debdiff.

[Yuan-Chen Cheng (ycheng-twn)]

debdiff for impish

[Yuan-Chen Cheng (ycheng-twn)]

debdiff for focal. Note I back-port a chunk from the patch as hirsute.

[Yuan-Chen Cheng (ycheng-twn)]

MP for focal: https://salsa.debian.org/gnome-team/mutter/-/merge_requests/78

[Yuan-Chen Cheng (ycheng-twn)]

Given ubuntu/impish is not exist in salsa, so I create it in my salsa git and another patch to include the change.

https://salsa.debian.org/ycheng-guest/mutter/-/tree/salsa_wayland_race_fix_impish

[Łukasz Zemczak (sil2100)]

Hello Yuan-Chen, or anyone else affected,

Accepted mutter into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mutter/40.5-1ubuntu3~21.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Yuan-Chen, or anyone else affected,

Accepted mutter into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mutter/3.36.9-0ubuntu0.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Accepted both focal and impish, but I don't see anyone sponsoring the hirsute version - could we get that uploaded as well? I know it's not the latest series, but hirsute will still be supported for a while and we shouldn't forget about users of 21.04, at least not yet. Not everyone upgraded yet.

Can someone sponsor hirsute before we go live with these updates? Thanks.

[Marco Trevisan (Treviño) (3v1n0)]

Ok, I've uploaded hirsute too, we normally don't have much resources to handle also such versions, but this was easy enough.

[Łukasz Zemczak (sil2100)]

Hello Yuan-Chen, or anyone else affected,

Accepted mutter into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mutter/3.38.4-1ubuntu2.21.04.01 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Daniel van Vugt (vanvugt)]

From bug 1947739 comment #53:

> test mutter 3.36.0-0ubuntu0.20.04.2 from the focal-proposed channel on
> the target machine for 10 times, can't reproduce this issue.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mutter - 40.5-1ubuntu3

---------------
mutter (40.5-1ubuntu3) jammy; urgency=medium

  [ Yuan-Chen Cheng ]
  * d/p/mutter-wayland-seat-race-fix.patch:
    - Fix Xwayland crash Xwayland crash in gdm as screen off and touchscreen
      detach (cherry picked commits — LP: #1948894)

  [ Marco Trevisan (Treviño) ]
  * debian/patches: Refresh

 -- Marco Trevisan (Treviño) <email address hidden> Wed, 03 Nov 2021 19:37:09 +0100

[Yuan-Chen Cheng (ycheng-twn)]

I try to install impish and hirsute on the target machine, with the kernel from focal (so that it support the new ADL GPU, and then the monitor can be turned off so that the touchscreen will be detached, that's the way we reproduce this issue in focal)

With that, with or without mutter from the proposed channel, I can't reproduce this issue. (We do see the backlight turned off and the touchscreen detached from kernel message.)

Given this is a timing-sensitive issue, I think as long as there is no regression, then we can call that as verified done.

I've dog food those, and can't find regression so far. So I'll tag verify done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mutter - 40.5-1ubuntu3~21.10.1

---------------
mutter (40.5-1ubuntu3~21.10.1) impish; urgency=medium

  * Backport to impish
  * debian: Update vcs references for impish branching

mutter (40.5-1ubuntu3) jammy; urgency=medium

  [ Yuan-Chen Cheng ]
  * d/p/mutter-wayland-seat-race-fix.patch:
    - Fix Xwayland crash Xwayland crash in gdm as screen off and touchscreen
      detach (cherry picked commits — LP: #1948894)

  [ Marco Trevisan (Treviño) ]
  * debian/patches: Refresh

 -- Marco Trevisan (Treviño) <email address hidden> Wed, 03 Nov 2021 19:57:15 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for mutter has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mutter - 3.38.4-1ubuntu2.21.04.01

---------------
mutter (3.38.4-1ubuntu2.21.04.01) hirsute; urgency=medium

  [ Yuan-Chen Cheng ]
  * d/p/mutter-wayland-seat-race-fix.patch:
    - Fix Xwayland crash Xwayland crash in gdm as screen off and touchscreen
      detach (cherry picked commits — LP: #1948894)
    - backported one chunk of above patch.

  [ Marco Trevisan (Treviño) ]
  * debian: Update VCS references for hirsute branching

 -- Marco Trevisan (Treviño) <email address hidden> Thu, 04 Nov 2021 16:15:12 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package mutter - 3.36.9-0ubuntu0.20.04.2

---------------
mutter (3.36.9-0ubuntu0.20.04.2) focal; urgency=medium

  [ Yuan-Chen Cheng ]
  * d/p/mutter-wayland-seat-race-fix.patch:
    - Fix Xwayland crash Xwayland crash in gdm as screen off and touchscreen
      detach (cherry picked commits — LP: #1948894)
    - backported one chunk of above patch.

 -- Marco Trevisan (Treviño) <email address hidden> Wed, 03 Nov 2021 20:10:34 +0100