wayland session hard freezes on login from GDM (X11 session works fine)

[John Jacob]

When I login to my laptop running Ubuntu 20.04, after entering my username, when I choose the "gear" button and select Ubuntu on Wayland as the session type, my laptop freezes hard. Mouse and keyboard do not respond, and the first three times I hit this issue I had to forcibly power off. The normal (X11) Ubuntu session works fine.

However, I realized that I could still ping and ssh in, and doing that gave me some clues to the issue.

I have keychain installed for password-less SSH, and my bash_profile has:
---
#!/bin/bash
# Get the aliases and functions
if [ -f $HOME/.bashrc ] ; then
  source $HOME/.bashrc
fi

# Setup password-less SSH
[ -z "$HOSTNAME" ] && HOSTNAME=`uname -n`
keychain $HOME/.ssh/id_rsa
source $HOME/.keychain/${HOSTNAME}-sh
---

When I login via SSH from another machine after hitting the issue, I get:
[johnjaco@johnjaco-m-v02f: ~/Downloads] ssh john@192.168.1.10
Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-29-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

0 updates can be installed immediately.
0 of these updates are security updates.

Your Hardware Enablement Stack (HWE) is supported until April 2025.
Last login: Mon Apr 27 19:41:17 2020 from 192.168.1.3

 * keychain 2.8.5 ~ http://www.funtoo.org
 * Waiting 5 seconds for lock...
 * Found existing ssh-agent: 2086
 * Adding 1 ssh key(s): /home/john/.ssh/id_rsa
Enter passphrase for /home/john/.ssh/id_rsa:
 * ssh-add: Identities added: /home/john/.ssh/id_rsa
[john@andromeda: ~] pstree
systemd─┬─NetworkManager───2*[{NetworkManager}]
        ├─accounts-daemon───2*[{accounts-daemon}]
        ├─acpid
        ├─avahi-daemon───avahi-daemon
        ├─bluetoothd
        ├─boltd───2*[{boltd}]
        ├─colord───2*[{colord}]
        ├─containerd───13*[{containerd}]
        ├─cron
        ├─cups-browsed───2*[{cups-browsed}]
        ├─cupsd
        ├─dbus-daemon
        ├─dnsmasq───dnsmasq
        ├─gdm3─┬─gdm-session-wor─┬─gdm-wayland-ses─┬─gnome-session-b───3*[{gnom+
        │ │ │ └─2*[{gdm-wayland-ses}]
        │ │ └─2*[{gdm-session-wor}]
        │ ├─gdm-session-wor─┬─gdm-wayland-ses─┬─bash───keychain───ssh-add
        │ │ │ └─2*[{gdm-wayland-ses}]
        │ │ └─2*[{gdm-session-wor}]
        │ └─2*[{gdm3}]
        ├─gnome-keyring-d───2*[{gnome-keyring-d}]
        ├─iio-sensor-prox───2*[{iio-sensor-prox}]
        ├─irqbalance───{irqbalance}
        ├─2*[kerneloops]
        ├─libvirtd───16*[{libvirtd}]
        ├─networkd-dispat
        ├─packagekitd───2*[{packagekitd}]
        ├─polkitd───2*[{polkitd}]
        ├─rsyslogd───3*[{rsyslogd}]
        ├─rtkit-daemon───2*[{rtkit-daemon}]
        ├─ssh-agent
        ├─sshd───sshd───sshd───bash───pstree
        ├─switcheroo-cont───2*[{switcheroo-cont}]
        ├─systemd─┬─(sd-pam)
        │ ├─at-spi-bus-laun─┬─dbus-daemon
        │ │ └─3*[{at-spi-bus-laun}]
        │ ├─at-spi2-registr───2*[{at-spi2-registr}]
        │ ├─dbus-daemon
        │ ├─dconf-service───2*[{dconf-service}]
        │ ├─gjs───10*[{gjs}]
        │ ├─gnome-keyring-d───3*[{gnome-keyring-d}]
        │ ├─gnome-session-b───3*[{gnome-session-b}]
        │ ├─gnome-session-c───{gnome-session-c}
        │ ├─gnome-shell─┬─Xwayland───20*[{Xwayland}]
        │ │ ├─ibus-daemon─┬─ibus-dconf───3*[{ibus-dconf}]
        │ │ │ ├─ibus-engine-sim───2*[{ibus-engi+
        │ │ │ └─2*[{ibus-daemon}]
        │ │ └─16*[{gnome-shell}]
        │ ├─goa-daemon───3*[{goa-daemon}]
        │ ├─goa-identity-se───2*[{goa-identity-se}]
        │ ├─gsd-a11y-settin───3*[{gsd-a11y-settin}]
        │ ├─gsd-color───3*[{gsd-color}]
        │ ├─gsd-keyboard───3*[{gsd-keyboard}]
        │ ├─gsd-media-keys───3*[{gsd-media-keys}]
        │ ├─gsd-power───3*[{gsd-power}]
        │ ├─gsd-print-notif───2*[{gsd-print-notif}]
        │ ├─gsd-printer───2*[{gsd-printer}]
        │ ├─gsd-rfkill───2*[{gsd-rfkill}]
        │ ├─gsd-smartcard───4*[{gsd-smartcard}]
        │ ├─gsd-sound───3*[{gsd-sound}]
        │ ├─gsd-usb-protect───3*[{gsd-usb-protect}]
        │ ├─gsd-wacom───3*[{gsd-wacom}]
        │ ├─gsd-wwan───3*[{gsd-wwan}]
        │ ├─gsd-xsettings───7*[{gsd-xsettings}]
        │ ├─gvfs-afc-volume───3*[{gvfs-afc-volume}]
        │ ├─gvfs-goa-volume───2*[{gvfs-goa-volume}]
        │ ├─gvfs-gphoto2-vo───2*[{gvfs-gphoto2-vo}]
        │ ├─gvfs-mtp-volume───2*[{gvfs-mtp-volume}]
        │ ├─gvfs-udisks2-vo───3*[{gvfs-udisks2-vo}]
        │ ├─gvfsd───2*[{gvfsd}]
        │ ├─gvfsd-fuse───5*[{gvfsd-fuse}]
        │ ├─ibus-portal───2*[{ibus-portal}]
        │ ├─ibus-x11───6*[{ibus-x11}]
        │ ├─pulseaudio───3*[{pulseaudio}]
        │ ├─tracker-miner-f───4*[{tracker-miner-f}]
        │ └─xdg-permission-───2*[{xdg-permission-}]
        ├─systemd─┬─(sd-pam)
        │ ├─dbus-daemon
        │ ├─goa-daemon───3*[{goa-daemon}]
        │ ├─goa-identity-se───2*[{goa-identity-se}]
        │ ├─gvfs-afc-volume───3*[{gvfs-afc-volume}]
        │ ├─gvfs-goa-volume───2*[{gvfs-goa-volume}]
        │ ├─gvfs-gphoto2-vo───2*[{gvfs-gphoto2-vo}]
        │ ├─gvfs-mtp-volume───2*[{gvfs-mtp-volume}]
        │ ├─gvfs-udisks2-vo───3*[{gvfs-udisks2-vo}]
        │ ├─gvfsd───2*[{gvfsd}]
        │ ├─gvfsd-fuse───5*[{gvfsd-fuse}]
        │ ├─pulseaudio───3*[{pulseaudio}]
        │ └─tracker-miner-f───4*[{tracker-miner-f}]
        ├─systemd-journal
        ├─systemd-logind
        ├─systemd-machine
        ├─systemd-resolve
        ├─systemd-timesyn───{systemd-timesyn}
        ├─systemd-udevd
        ├─thermald───{thermald}
        ├─udisksd───4*[{udisksd}]
        ├─unattended-upgr───{unattended-upgr}
        ├─upowerd───2*[{upowerd}]
        └─wpa_supplicant

This makes me think that gdm-wayland-session is somehow stuck on trying to add the SSH key.

This issue is consistently reproducible, so let me know what additional information is needed and I shall provide it.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: gdm3 3.34.1-1ubuntu1 [modified: usr/share/gdm/greeter.dconf-defaults]
ProcVersionSignature: Ubuntu 5.4.0-29.33-generic 5.4.30
Uname: Linux 5.4.0-29-generic x86_64
ApportVersion: 2.20.11-0ubuntu27
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
Date: Tue May 12 15:14:35 2020
ExecutablePath: /usr/lib/gdm3/gdm-wayland-session
InstallationDate: Installed on 2020-04-23 (19 days ago)
InstallationMedia: Ubuntu 20.04 LTS "Focal Fossa" - Release amd64 (20200423)
ProcEnviron:
 SHELL=/bin/bash
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 XDG_RUNTIME_DIR=<set>
SourcePackage: gdm3
UpgradeStatus: No upgrade log present (probably fresh install)

[Daniel van Vugt (vanvugt)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It sounds like some part of the system has crashed. To help us find the cause of the crash please follow these steps:

1. Look in /var/crash for crash files and if found run:
    ubuntu-bug YOURFILE.crash
Then tell us the ID of the newly-created bug.

2. If step 1 failed then look at https://errors.ubuntu.com/user/ID where ID is the content of file /var/lib/whoopsie/whoopsie-id on the machine. Do you find any links to recent problems on that page? If so then please send the links to us.

3. If step 2 also failed then apply the workaround from bug 994921, reboot, reproduce the crash, and retry step 1.

Please take care to avoid attaching .crash files to bugs as we are unable to process them as file attachments. It would also be a security risk for yourself.

[John Jacob (johnjacob10)]

Hi Daniel,

Thank you for taking a look at my bug report.
1. There are no crash files in /var/crash.
2. https://errors.ubuntu.com/user/27232d1c0322b0e12fbced3d5cc680848662bdab3723e8c5a2ef4f4994f0f37d64c8663e929ae708ddbfce7b00cf9529860901cc62597a8b0b414d4b29fc9792 has three problems reported, but none of them correspond to this issue.

I actually do not think there is a crash, rather that some part of the session is waiting for me to enter the keychain passphrase in a way that is hanging the GUI.

I verified that if I SSH in from another machine, and enter the passphrase in the SSH session *before* I login from the GDM login screen, everything works fine.

Let me know what other steps you would like me to take.

[John Jacob (johnjacob10)]

Here are steps to reproduce the issue. Confirmed on a fresh Ubuntu 20.04 install in a VM.

sudo apt install keychain
mkdir ~/.ssh
cd ~/.ssh
ssh-keygen -t rsa -b 4096 -C "user@domain"
 # -- Enter a passphrase when asked.
vi ~/.bash_profile
 # -- Add the following text. Replace "user" with the actual user name.
     #!/bin/bash
     set autologout=0
     # Setup password-less SSH
     [ -z "$HOSTNAME" ] && HOSTNAME=`uname -n`
     keychain /home/user/.ssh/id_rsa
     source /home/user/.keychain/${HOSTNAME}-sh

Reboot. On the login screen, click on the user, then select "Ubuntu on Wayland" in the Gear icon.
Enter password and login.

Result: login screen hangs indefinitely.

[John Jacob (johnjacob10)]

Provided additional information, therefore resetting to New as per https://wiki.ubuntu.com/Bugs/Bug%20statuses
"Should the original reporter reset the report status to "New" if providing more information to an "Incomplete" report?
Yes."

[Daniel van Vugt (vanvugt)]

Next please:

1. Reproduce another Wayland freeze.

2. Reboot once only.

3. Log in to an X11 session and run:

   journalctl -b-1 > prevboot.txt
   lspci -k > lspcik.txt

   and attach the resulting two text files here.

[John Jacob (johnjacob10)]

After reproducing the freeze, rebooted by Ctrl-Alt-Del (which works in the VM, does not work on my laptop).

[Daniel van Vugt (vanvugt)]

You originally reported the problem against your laptop, but those attachments are from a VM.

Please confirm the VM experiences the same freeze.

[Daniel van Vugt (vanvugt)]

Actually, you can't be sure...

Please open a new bug about this issue from the VM by running:

  ubuntu-bug mutter

and keep that new bug about the VM only.

[John Jacob (johnjacob10)]

Daniel,

I reproduced the same issue in a VM in post #4, and I gave you the steps to reproduce it. I strongly suggest you *try* it. It is trivially simple, takes less than five minutes to reproduce, or 15 minutes tops if you want to spin up a new VM. This is exactly the same issue that I saw on my laptop.

The reason I reproduced this on a VM was to confirm that the issue could be seen on other machines, and to give you an easily reproducible test case, so that you can confirm the bug and move it the right team to investigate.

Otherwise, we can keep repeating this dance of asking for more information and moving the bug back to incomplete, but this does nothing to advance our mutual goal of helping make Ubuntu better.

[Daniel van Vugt (vanvugt)]

Following the steps in comment #4 I do not experience any freezing. It logs in successfully.

Next I suggest you ssh into the frozen machine and check:

1. If any processes are using high CPU.

2. How many gnome-shell processes are running.

3. Try to force the newest gnome-shell process (either running as you or as 'gdm') to dump core by sending it a fatal signal:

   kill -ABRT <PID-of-gnome-shell>

If that works then you will have a crash file in /var/crash that we can analyse per comment #2.

[John Jacob (johnjacob10)]

You can see the core at https://errors.ubuntu.com/user/e35124d979e3b918d0a8f821141d16026a6862cda0707840355369e69b328df9d3a372a704b7af732e746e1600eb565efebfb0f79a83711873cfbe43fe12a08c

Killing the gnome-shell process resolves the freeze and the login continues normally after that.

[Daniel van Vugt (vanvugt)]

Thanks. That core is of the login screen itself. We haven't got a retrace yet but maybe wait a few hours and the bots will get to it...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Daniel van Vugt (vanvugt)]

Please repeat step 3 from comment #14. It seems the link in comment #15 still isn't usable.

[John Jacob (johnjacob10)]

I repeated the steps on my laptop. The previous report was from the VM.

Look at the gnome-shell crash received on 2020-06-05 20:06 UTC at:
https://errors.ubuntu.com/user/27232d1c0322b0e12fbced3d5cc680848662bdab3723e8c5a2ef4f4994f0f37d64c8663e929ae708ddbfce7b00cf9529860901cc62597a8b0b414d4b29fc9792

[Daniel van Vugt (vanvugt)]

Thanks but that also is failing to retrace. Try a different signal like:

  kill -SEGV ...

[Daniel van Vugt (vanvugt)]

And if all else fails, please attach 'gdb' to the process:

  sudo gdb /usr/bin/gnome-shell PID

and retrieve a backtrace of all threads using:

  thread apply all bt

[John Jacob (johnjacob10)]

I used apport-unpack to extract the coredump from the crash file, and then ran gdb on it. The two attachments gdb_laptop.log and gdb_vm.log are from gdb in the laptop and in the VM, respectively. Without debug symbols, the backtraces are not complete.

I tried to install debug symbols in the VM, as detailed at https://wiki.ubuntu.com/Debug%20Symbol%20Packages , but it failed due to:

john@ubuntu-test:~/Downloads/core$ sudo apt install gnome-shell-dbgsym
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 gnome-shell-dbgsym : Depends: gnome-shell (= 3.36.1-5ubuntu1) but 3.36.2-1ubuntu1~20.04.1 is to be installed
E: Unable to correct problems, you have held broken packages.

[Daniel van Vugt (vanvugt)]

Thanks. Those both show gnome-shell is idle and not frozen in any obvious way. So that gives me two ideas:

  * I wonder if this is one of those rare cases where it's starting on the wrong VT. Please try: Ctrl+Alt+F4, Ctrl+Alt+F2 etc and see if you can find the missing session when it appears to freeze.

  * I wonder if the 'qxl' kernel graphics driver is failing to report (virtual) presentation events with correct timestamps. That would cause the apparently idle symptoms we are seeing and is also a bug we've seen in the VMware kernel driver (fixed a long time ago).

[John Jacob (johnjacob10)]

I think it is starting on the wrong VT. However, the missing session does not reappear. I took two cellphone videos, one of my laptop (335MB) and one of my VM (624MB) showing the issue in both cases. Let me see if I can upload them.

[John Jacob (johnjacob10)]

I was unable to upload the videos to Launchpad, so I uploaded them to Youtube. Here are the links:

Video from laptop: https://youtu.be/rPCjCMRwtoo
Video from VM : https://youtu.be/9enU0jCbtVI

[Daniel van Vugt (vanvugt)]

Thanks for the videos.

One thing you didn't seem to try, and I think is the most likely to work, would be:

  Ctrl+Alt+F3, Ctrl+Alt+F2

because VT2 is where you want to go in the end but it might be ignored until you switch to VT3 or higher first.

Secondly, I can now see the ssh problem you mentioned...

  Enter passphrase for /home/john/.ssh/id_rsa:
   * ssh-add: Identities added: /home/john/.ssh/id_rsa

Yes that sounds like a general account problem which would explain why you experience trouble with both physical and virtual machines. So I suspect the real problem is coming from:

 "I have keychain installed for password-less SSH"

Please remove "keychain" and your bash_profile modifications.

[Launchpad Janitor (janitor)]

[Expired for gdm3 (Ubuntu) because there has been no activity for 60 days.]