gnome-shell on non-active VT busy-loops through libwayland-server.so (sometimes)

[Alan Jenkins]

I run several gnome sessions at once - different users on different Virtual Terminals.

Sometimes, gnome-shell for a non-active VT starts using 100% CPU. Or, actually 99% CPU because it's doing IPC as well. And the corresponding XWayland uses 40% CPU.

FWIW, sending SIGSTOP to either gnome-shell or XWayland stops the other one from busy-looping as well.

If I switch back to the affected VT, it appears to function fine, but it is still using 100% + 40% CPU.

Version numbers:

Ubuntu 22.04.3 LTS
gnome-shell 42.9-0ubuntu2
xwayland 2:22.1.1-1ubuntu0.7

$ ps -aux | grep 'alan .*gnome-shell$'
alan 9248 26.1 3.4 5492888 269136 ? Rsl 19:54 30:26 /usr/bin/gnome-shell

$ sudo strace -p 9248 2>&1|head
strace: Process 9248 attached
epoll_wait(37, [{events=EPOLLIN, data={u32=403234816, u64=139797293752320}}], 32, 0) = 1
recvmsg(21, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\0\0\0\0\f\0\3\0\0\0", iov_len=752}, {iov_base="", iov_len=3344}], msg_iovlen=2, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 12
write(4, "\1\0\0\0\0\0\0\0", 8) = 8
sendmsg(21, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\3\0\0\0\0\0\f\0[\2\0\0\1\0\0\0\1\0\f\0\3\0\0\0", iov_len=24}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 24
poll([{fd=4, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLIN}, {fd=19, events=POLLIN}, {fd=20, events=POLLIN}, {fd=23, events=POLLIN}, {fd=24, events=POLLIN}, {fd=25, events=POLLIN}, {fd=34, events=POLLIN}, {fd=36, events=POLLIN}, {fd=37, events=POLLIN}, {fd=54, events=POLLIN}, {fd=63, events=POLLIN}, {fd=66, events=POLLIN}], 14, 79809702) = 2 ([{fd=4, revents=POLLIN}, {fd=37, revents=POLLIN}])
read(4, "\2\0\0\0\0\0\0\0", 16) = 8
write(4, "\1\0\0\0\0\0\0\0", 8) = 8
epoll_wait(37, [{events=EPOLLIN, data={u32=403234816, u64=139797293752320}}], 32, 0) = 1

The recvmsg() call is here:

#0 __libc_recvmsg (fd=21, msg=0x7fff1de64a50, flags=1073741888) at ../sysdeps/unix/sysv/linux/recvmsg.c:35
#1 0x00007f2531ea5ad6 in () at /lib/x86_64-linux-gnu/libwayland-server.so.0
#2 0x00007f2531ea4eea in wl_event_loop_dispatch () at /lib/x86_64-linux-gnu/libwayland-server.so.0
#3 0x00007f2531d2b45b in wayland_event_source_dispatch (base=<optimised out>, callback=<optimised out>, data=<optimised out>)
    at ../src/wayland/meta-wayland.c:110
#4 0x00007f2532b42d3b in g_main_dispatch (context=0x55c0046cc8f0) at ../../../glib/gmain.c:3419
#5 g_main_context_dispatch (context=0x55c0046cc8f0) at ../../../glib/gmain.c:4137
#6 0x00007f2532b98258 in g_main_context_iterate.constprop.0
    (context=0x55c0046cc8f0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimised out>) at ../../../glib/gmain.c:4213
#7 0x00007f2532b422b3 in g_main_loop_run (loop=0x55c00698ae30) at ../../../glib/gmain.c:4413
#8 0x00007f2531cd0b79 in meta_context_run_main_loop (context=context@entry=0x55c0046c70c0, error=error@entry=0x7fff1de64f08)
    at ../src/core/meta-context.c:437
#9 0x000055c003cf4f12 in main (argc=<optimised out>, argv=<optimised out>) at ../src/main.c:650

The sendmsg() call is here:

Thread 1 "gnome-shell" hit Breakpoint 1, __libc_sendmsg (fd=21, msg=0x7fff1de64c50, flags=16448) at ../sysdeps/unix/sysv/linux/sendmsg.c:26
26 ../sysdeps/unix/sysv/linux/sendmsg.c: No such file or directory.
(gdb) where
#0 __libc_sendmsg (fd=21, msg=0x7fff1de64c50, flags=16448) at ../sysdeps/unix/sysv/linux/sendmsg.c:26
#1 0x00007f2531ea1c9d in () at /lib/x86_64-linux-gnu/libwayland-server.so.0
#2 0x00007f2531ea565e in wl_display_flush_clients () at /lib/x86_64-linux-gnu/libwayland-server.so.0
#3 0x00007f2531d2b437 in wayland_event_source_prepare (base=base@entry=0x55c004a374d0, timeout=timeout@entry=0x7fff1de64dc4)
    at ../src/wayland/meta-wayland.c:97
#4 0x00007f2532b428e7 in g_main_context_prepare (context=0x55c0046cc8f0, priority=0x7fff1de64e50) at ../../../glib/gmain.c:3739
#5 0x00007f2532b98123 in g_main_context_iterate.constprop.0
    (context=0x55c0046cc8f0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimised out>) at ../../../glib/gmain.c:4193
#6 0x00007f2532b422b3 in g_main_loop_run (loop=0x55c00698ae30) at ../../../glib/gmain.c:4413
#7 0x00007f2531cd0b79 in meta_context_run_main_loop (context=context@entry=0x55c0046c70c0, error=error@entry=0x7fff1de64f08)
    at ../src/core/meta-context.c:437
#8 0x000055c003cf4f12 in main (argc=<optimised out>, argv=<optimised out>) at ../src/main.c:650

[Alan Jenkins (aj504)]

I tried installing dbgsym packages or using debuginfod to get a full backtrace for libwayland-server.so, but I failed.

$ sudo apt install -qq libwayland-server0-dbgsym
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies.
 libwayland-server0-dbgsym : Depends: libwayland-server0 (= 1.20.0-1) but 1.20.0-1ubuntu0.1 is to be installed
E: Unable to correct problems, you have held broken packages.

$ export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com"
$ sudo gdb
...
(gdb) set debuginfod enabled on
(gdb) attach 9248
(gdb) break recvmsg
Breakpoint 1 at 0x7f25319279d0: file ../sysdeps/unix/sysv/linux/recvmsg.c, line 35.
(gdb) cont
Continuing.

Thread 1 "gnome-shell" hit Breakpoint 1, __libc_recvmsg (fd=21, msg=0x7fff1de64a50, flags=1073741888) at ../sysdeps/unix/sysv/linux/recvmsg.c:35
35 ../sysdeps/unix/sysv/linux/recvmsg.c: No such file or directory.
(gdb) bt full
#0 __libc_recvmsg (fd=21, msg=0x7fff1de64a50, flags=1073741888) at ../sysdeps/unix/sysv/linux/recvmsg.c:35
#1 0x00007f2531ea5ad6 in () at /lib/x86_64-linux-gnu/libwayland-server.so.0
#2 0x00007f2531ea4eea in wl_event_loop_dispatch () at /lib/x86_64-linux-gnu/libwayland-server.so.0
#3 0x00007f2531d2b45b in wayland_event_source_dispatch (base=<optimised out>, callback=<optimised out>, data=<optimised out>)
    at ../src/wayland/meta-wayland.c:110
        source = <optimised out>
        loop = <optimised out>
...
(gdb) finish
Run till exit from #0 __libc_recvmsg (fd=21, msg=0x7fff1de64a50, flags=1073741888) at ../sysdeps/unix/sysv/linux/recvmsg.c:35
0x00007f2531ea5ad6 in ?? () from /lib/x86_64-linux-gnu/libwayland-server.so.0
Value returned is $2 = 12
(gdb) finish
Run till exit from #0 0x00007f2531ea5ad6 in ?? () from /lib/x86_64-linux-gnu/libwayland-server.so.0
0x00007f2531ea4eea in wl_event_loop_dispatch () from /lib/x86_64-linux-gnu/libwayland-server.so.0

(gdb) s
Single stepping until exit from function wl_event_loop_dispatch,
which has no line number information.
wayland_event_source_dispatch (base=<optimised out>, callback=<optimised out>, data=<optimised out>) at ../src/wayland/meta-wayland.c:112
112 ../src/wayland/meta-wayland.c: No such file or directory.

[Daniel van Vugt (vanvugt)]

Thanks for the bug report. Please also report the issue upstream at https://gitlab.gnome.org/GNOME/mutter/-/issues

[Alan Jenkins (aj504)]

The bug re-occurred, after manually building libwayland with debugging symbols.

So I could do more debugging, and I think this Ubuntu bug needs to be reassigned to Xwayland.

I have filed more details in the upstream Xwayland tracker.

https://gitlab.freedesktop.org/xorg/xserver/-/issues/1648