[nvidia][xorg] display hangs on boot LOGO due to race of gdm and nvidia driver probe

[jeremyszu]

[Impact]

 * In Ubuntu 20.04 (either impish, jammy, upstream gdm) (which using Xorg with proprietary nvidia driver), in some cases, the nvidia driver will probe later than launching gdm.
 * If above race condition happens in iGPU + nvidia cases and monitor connects to dGPU, which will cause gdm starts with wayland as opposed to Xorg. Which may lead the monitor stuck in black-screen or boot LOGO.

[Test Plan]

 * The environment:
  1. A desktop or workstation which containing an iGPU.
  2. Plug a nvidia graphic card to the system and installing proprietary
  nvidia driver (470 in my case)
  3. Attach a monitor to dGPU and leave iGPU connect to nothing.
  (in my test environment, there is the other ethernet card and TBT4 cards)
 * Setup a cronjob,
   e.g. @reboot /home/u/test.sh
 * Have a test script in something like /home/u/test.sh as

#!/bin/bash

sleep 20
count="$(cat /home/ubuntu/count)"
count=$((count+1))
echo $count | tee /home/ubuntu/count
journalctl -b | grep -q -i wayland || sudo reboot

 * the system will probably stuck in black-screen or boot LOGO.
 * Before applying the fix, the fail rate is 6/24 (fail 6 time in 24 runs).
 * After applying the fix, it got pass within 1000+ reboot cycles.
 * Test PPA can be found here https://launchpad.net/~os369510/+archive/ubuntu/lp1958488

[Fix]
 * The patch makes gpu-manager to probe nvidia (if needed) first and waiting for the /run/u-d-c-nvidia-drm-was-loaded be touched by 71-u-d-c-gpu-detection.rules.
 * Also, the gdm is using 61-gdm.rules to configure the gdm mode by checking the nvidia driver presents or not.
 * gpu-manager is before display-manager. Thus, gpu-manager will wait for nvidia uevent be processed and then continue to work. When gdm be launched, the targeted nvidia uevent has been processed already. (71-u-d-c-gpu-detection.rules is later than 61-gdm.rules）

[Where problems could occur]
 * there is not potential regression from my mind but it will lead the boot time be longer.
 * In my test cycles, it leads extra 0~1000ms in boot time. Usually, 0~200ms. Worst case, over 1 s in 8xx runs (of 1000).
 * I think the stability is important than performance in this case.

[Other Info]
 * For non-ubuntu-desktop (which doesn't have gpu-manager), which using gdm will meet this issue still. The other potential fix (from either gdm or logind) is under discussion in https://gitlab.gnome.org/GNOME/gdm/-/issues/763#note_1385786.
 * u-d-c upstream fix: https://github.com/tseliot/ubuntu-drivers-common/pull/67

---

Test environment/steps:
1. A desktop or workstation which containing an iGPU.
2. Plug a nvidia graphic card to the system and installing proprietary nvidia driver (470 in my case)
3. Attach a monitor to dGPU and leave iGPU connect to nothing.
(in my test environment, there is the other ethernet card and TBT4 cards)
4. Reboot system.

Based on:
$ cat /lib/udev/rules.d/61-gdm.rules
# disable Wayland on Hi1710 chipsets
ATTR{vendor}=="0x19e5", ATTR{device}=="0x1711", RUN+="/usr/lib/gdm3/gdm-disable-wayland"
# disable Wayland when using the proprietary nvidia driver
DRIVER=="nvidia", RUN+="/usr/lib/gdm3/gdm-disable-wayland"

It will disable wayland by default if proprietary nvidia driver load.
But in some race condition cases, the nvidia probe is later than gnome launches. (The fail rate is 6/24.)
Thus, ubuntu-gdm has a fix for Bug#1794280 to add "ExecStartPre=@libexecdir@/gdm-wait-for-drm".

The gdm-wait-for-drm is intend to make sure all drm udev devices enumerated before launching gdm.

It rely on at least one "master-of-seat" graphic card for gdm but it's not rigorous enough.
Since most of graphic cards are own "master-of-seat"[1].

In my case, it detects the iGPU is probed but dGPU.
However, the display is attached to dGPU.

We need to make sure the targeted gpu (connecting to monitor) is probe before launching gdm.

debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1004131
upstream bug: https://gitlab.gnome.org/GNOME/gdm/-/issues/763

[1] https://www.freedesktop.org/wiki/Software/systemd/multiseat/
/lib/udev/rules.d/71-seat.rules
/lib/udev/rules.d/71-nvidia.rules

[jeremyszu (os369510)]

for focal

[jeremyszu (os369510)]

This patch doesn't need to apply to jammy/impish because
debian/patches/gdm.rules-Keep-wayland-enabled-in-all-nvidia-configuratio.patch

Please let me know if my assumption is incorrect.

[Daniel van Vugt (vanvugt)]

Note that gnome-shell (which also renders the login screen for gdm) always uses the primary iGPU for rendering regardless of whether some monitors are plugged into a dGPU port. The latter get their framebuffers rendered on the iGPU but copied to the dGPU for output on every frame. This makes sense for the common dual-GPU case of a laptop where you want to be able to connect an external monitor to any port (even if that port is wired to dGPU) while still using the iGPU for efficiency by default.

But enough from me. You should propose any patches to https://gitlab.gnome.org/GNOME/gdm/ (or to https://gitlab.gnome.org/GNOME/mutter) to get more authoritative information from the developers.

[Kai-Heng Feng (kaihengfeng)]

"boot_vga" only exists for PCI device, so this change will make VMs and boards that don't use PCI GFX wait for straight 10 seconds.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1958488_gdm-wait-for-drm_wait_for_boot_vga.debdiff" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[jeremyszu (os369510)]

the "gdm-wait-for-drm" is only maintain in ubuntu-focal also upstream doesn't use "gdm-disable-wayland" anymore but "gdm-runtime-config" instead.

Anyway, I will try the gdm from upstream version to see whether having the same issue.

[Kai-Heng Feng (kaihengfeng)]

Last time I checked, "gdm_prefer_xorg" is still required in 61-gdm.rules otherwise the issue still occurs.

[jeremyszu (os369510)]

I tried the upstream gdm on focal (60 reboot cycles) and can not reproduce the issue.

I'm trying jammy and could reproduce this issue.
Let me build upstream version on jammy.

upstream on jammy works good (32 runs).
Let me find the difference.

[jeremyszu (os369510)]

The issue is happening in debian/ubuntu only. Thus, I'm not sure the upstream is correct place to discuss instead of debian/ubuntu.

This commit (536491641) works good.

| * 1c9e4c93d (HEAD, refs/bisect/bad) New upstream release
| * 4106a87d5 Update upstream source from tag 'upstream/41.0'
| |\
| | * 5fcf89011 (debian/upstream/latest) New upstream version 41.0
| | |\
| | | * 536491641 (tag: 41.0, refs/bisect/good-536491641156ba88beab2c881278b09e52df6813) 41.0

The difference between 1c9e4c93d .. 536491641 that is only debian/* (we don't care .gitignore)

[jeremyszu (os369510)]

When I build the 536491641, I use meson build.
When 1c9e4c93d, I use gbp buildpackage.

Thus, I tried dh_quilt_patch then gbp buildpackage on 1c9e4c93d.
The issue is gone.

Based on above test, I think it's likely related to debian/rules.

Something cause it (or has higher fail rate) happens.

Also report the bug to debian mailing list.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1004131

[jeremyszu (os369510)]

u-d-c patch for jammy

[jeremyszu (os369510)]

u-d-c for impish

[jeremyszu (os369510)]

u-d-c for focal

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-drivers-common - 1:0.9.5.1

---------------
ubuntu-drivers-common (1:0.9.5.1) jammy; urgency=medium

  * Change maximum open file descriptors limit for the process in
    the test suite. This should fix the FTBFS.

 -- Alberto Milone <email address hidden> Mon, 07 Mar 2022 17:16:26 +0100

[Sebastien Bacher (seb128)]

is there anything needed from gdm there?

[jeremyszu (os369510)]

Hi Seb,

FWIK, the fix from gdm is not mandatory.
The gpu-manager (inside u-d-c) will handle it.

but if any flavor which doesn't use u-d-c then gdm upstream also provide this commit
https://gitlab.gnome.org/GNOME/gdm/-/commit/ecbd9694458194e24f356ba9d37fc4f3a515c3dd#note_1402863
to fix this issue in GDM.

Says, if no one from a flavor (which doesn't use u-d-c) reports this issue then we don't hurry to SRU gdm patches.

[Chris Halse Rogers (raof)]

Hello jeremyszu, or anyone else affected,

Accepted ubuntu-drivers-common into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-drivers-common/1:0.9.0~0.20.04.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Hello jeremyszu, or anyone else affected,

Accepted ubuntu-drivers-common into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-drivers-common/1:0.9.2.4~0.21.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[jeremyszu (os369510)]

Tested focal version
```
ubuntu-drivers-common:
  Installed: 1:0.9.0~0.20.04.6

```

and the log shows
```
...
Takes 640ms to wait for nvidia udev rules completed.
```

thus, the result is PASS.

[jeremyszu (os369510)]

Tested impish version
```
ubuntu-drivers-common:
  Installed: 1:0.9.2.4~0.21.10.1

```

and the log shows
```
...
Takes 660ms to wait for nvidia udev rules completed.
```

thus, the result is PASS.

[Launchpad Janitor (janitor)]

This bug was fixed in the package gdm3 - 42.0-1ubuntu1

---------------
gdm3 (42.0-1ubuntu1) jammy; urgency=medium

  * Merge with debian, remaining changes:
    + readme.debian: update for correct paths in ubuntu
    + control.in:
      - don't recommend desktop-base
      - depend on bash for config_error_dialog.patch
      - update vcs field
    + rules:
      - don't override default user/group
      - -dgdm-xsession=true to install upstream xsession script
      - override dh_installinit with --no-start to avoid session being killed
    + rules, readme.debian, gdm3.8.pod:
      use upstream custom.conf instead of daemon.conf
    + gdm3.{postinst,postrm}: rename user and group back to gdm
    + gdm3.*.pam: make pam_env read ~/.pam_environment, as we use in g-c-c
      settings
    + gdm3.install:
      - don't install debian/xsession
    + add run_xsession.d.patch
    + add xresources_is_a_dir.patch
      - fix loading from /etc/x11/xresources/*
    + add nvidia_prime.patch:
      - add hook to run prime-offload (as root) and prime-switch if
        nvidia-prime is installed
    + add revert_override_lang_with_accountservices.patch:
      - on ubuntu accountservices only stores the language and not the
        full locale as needed by lang.
    + add dont_set_language_env.patch:
      - don't run the set_up_session_language() function, since it
        overrides variable values set by ~/.pam_environment
    + add config_error_dialog.patch:
      - show warning dialog in case of error in ~/.profile etc. and
        don't let a syntax error make the login fail
    + add debian/patches/revert_nvidia_wayland_blacklist.patch:
      - don't blacklist nvidia for wayland
    + add gdm3.service-wait-for-drm-device-before-trying-to-start-i.patch:
      - wait for the first valid gdm device on pre-start
    + add debian/default.pa
      - disable bluetooth audio devices in pulseaudio from gdm3.
    + debian/gdm3.install
      - added details of the default.pa file
    + debian/gdm3.postinst
      - added installation of default.pa and creation of dir if it doesn't
        exist.
    + debian/greeter.dconf-defaults: don't set debian settings in the
      greeter's dconf db

gdm3 (42.0-1) unstable; urgency=medium

  [ Jeremy Bicha ]
  * New upstream release
    - Fix hang caused by GDM starting sooner than nvidia_drm
      (Closes: #1004131, LP: #1958488)
    - Default to Wayland for nvidia 510 drivers (LP: #1962523)
  * debian/control.in: Build-Depend on libgudev-1.0-dev

  [ Simon McVittie ]
  * Add a NEWS.Debian entry for the removal of "System X11 Default"

 -- Jeremy Bicha <email address hidden> Tue, 22 Mar 2022 16:59:12 -0400

[Daniel van Vugt (vanvugt)]

Note that if nvidia_drm is starting so late then plymouth won't display in that case. Because Plymouth prioritises DRM and won't fall back to any other options (hence won't display anything!) unless boot takes longer than 8 seconds. It's related to https://gitlab.freedesktop.org/plymouth/plymouth/-/issues/103 despite us now using a different kernel driver.

I've been thinking for years that we should just go straight to efifb instead (see bug 1869655).

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-drivers-common - 1:0.9.0~0.20.04.6

---------------
ubuntu-drivers-common (1:0.9.0~0.20.04.6) focal; urgency=medium

  [ Jeremy Szu ]
  * 71-u-d-c-gpu-detection.rules,
    gpu-manager.c:
    - Wait for the nvidia-drm module to be loaded. This works around
      a race condition, where nvidia-drm is loaded after gdm, and
      this causes gdm to hang (LP: #1958488).

 -- Alberto Milone <email address hidden> Fri, 04 Mar 2022 16:35:45 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for ubuntu-drivers-common has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-drivers-common - 1:0.9.2.4~0.21.10.1

---------------
ubuntu-drivers-common (1:0.9.2.4~0.21.10.1) impish; urgency=medium

  [ Jeremy Szu ]
  * 71-u-d-c-gpu-detection.rules,
    gpu-manager.c:
    - Wait for the nvidia-drm module to be loaded. This works around
      a race condition, where nvidia-drm is loaded after gdm, and
      this causes gdm to hang (LP: #1958488).

 -- Alberto Milone <email address hidden> Fri, 04 Mar 2022 16:57:33 +0100

[ysfr (ysfr)]

Hi,

seems that the change from 1:0.9.0~0.20.04.5 to 1:0.9.0~0.20.04.6
did introduce for me a 10s extra delay at each startup,
despite the fact that it's a desktop with only one (pretty old) Nvidia discrete graphic card.

systemd-analyze blame
   10.403s gpu-manager.service
   9.927s plymouth-quit-wait.service
   2.641s dev-sda3.device

While investigating, I found in
/var/log/gpu-manager.log
   can't access /run/u-d-c-nvidia-drm-was-loaded file
   ...
   can't access /run/u-d-c-nvidia-drm-was-loaded file
   Takes 10000ms to wait for nvidia udev rules completed.
   Single card detected

which lead me here ...

Has anyone an idea to workaround this ?

Thanks

[jeremyszu (os369510)]

Hi, would you please share:

1. lspci -nvk
2. /var/log/gpu-manager.log
3. journalctl -b -k
4. dpkg -l | grep nvidia
5. lsmod | grep nvidia

[ysfr (ysfr)]

Hi,

did it reverting to ubuntu-drivers-common 1:0.8.1 (focal) > XXX_before.txt logs in ZIP

accountname@hostname:~$ systemd-analyze time
 Startup finished in 5.469s (kernel) + 5.395s (userspace) = 10.864s
 graphical.target reached after 4.814s in userspace
accountname@hostname:~$ systemd-analyze blame
 2.343s dev-sda3.device
 2.285s man-db.service
 ...

& updating to:
ubuntu-drivers-common 1:0.9.0~0.20.04.7 (focal-updates)

accountname@hostname:~$ systemd-analyze time
  Startup finished in 5.557s (kernel) + 14.014s (userspace) = 19.572s
  graphical.target reached after 13.983s in userspace
accountname@hostname:~$ systemd-analyze blame
  10.488s gpu-manager.service
  10.142s plymouth-quit-wait.service
  2.704s dev-sda3.device
  ...

Hope it helps !

[jeremyszu (os369510)]

Hi,

I aware you don't have "nvidia_drm" which is needed for gdm as well.
u-d-c was wait the "nvidia_drm" be probed and your nvidia driver is 340 is quite old as what I known.

FWIK, the legacy driver should be nvidia-390 and the next is 470.
I wondering why nvidia-340 is not yet be transited to 390 or newer.

Hi Alberto,

Will nvidia-340 be transited?

[jeremyszu (os369510)]

ok, I aware the `nvidia-drm` should support in nvidia-340.

Did you blacklist it?
Can you check the nvidia-drm whether presenting in your system?
you could probably found it under
find ... /lib/modules/$(uname -r)/...

[jeremyszu (os369510)]

update:

no transition plan for nvidia-340.
Then let's see why your nvidia-drm gone.

[ysfr (ysfr)]

Hi,

I do not remember blacklisting nvidia-drm intentionally, but will check !
Anyway don't you think this line from journalctl:
  kernel: [drm] Initialized nvidia-drm 0.0.0 20150116 for 0000:01:00.0 on minor 0
indicates nvidia-drm is fine ?

Running Mint 20.3 still with kernel 5.4 if that matters ...
seems also Nvidia driver version 340 is the last to support my GeForce 210 based card ...
https://nvidia.custhelp.com/app/answers/detail/a_id/3142/%7E/support-timeframes-for-unix-legacy-gpu-releases

Yes it's old ... not only the GPU ..., but still fairly usable ;-)

[jeremyszu (os369510)]

but it doesn't show in your lsmod.

how about `modinfo nvidia_drm`?

[ysfr (ysfr)]

modinfo nvidia-drm
  modinfo: ERROR: Module nvidia-drm not found.
modinfo nvidia_drm
  modinfo: ERROR: Module nvidia_drm not found.

modinfo nvidia
  filename: /lib/modules/5.4.0-110-generic/kernel/drivers/char/drm/nvidia.ko

inxi -G
  Graphics: Device-1: NVIDIA GT218 [GeForce 210] driver: nvidia v: 340.108
           Display: x11 server: X.Org 1.20.13 driver: nvidia
           unloaded: fbdev,modesetting,nouveau,vesa resolution: 1600x900~60Hz
           OpenGL: renderer: GeForce 210/PCIe/SSE2 v: 3.3.0 NVIDIA 340.108

https://www.nvidia.fr/Download/find.aspx?lang=fr shows
Linux x64 (AMD64/EM64T) Display Driver Recommandé par NVIDIA 340.108 23.12.2019
* Updated the nvidia-drm kernel module for compatibility with the removal of the DRIVER_PRIME flag in recent Linux kernel versions.

lsmod | grep nvidia
  Module Size Used by
  nvidia_uvm 36864 0
  nvidia 10588160 50 nvidia_uvm
  drm 491520 3 nvidia

modinfo drm
  filename: /lib/modules/5.4.0-110-generic/kernel/drivers/gpu/drm/drm.ko
  description: DRM shared core routines
  description: DRM bridge infrastructure
  description: DRM panel infrastructure
  author: Thierry Reding <email address hidden>
  name: drm
  vermagic: 5.4.0-110-generic SMP mod_unload modversions

Could that just mean that 'nvidia_drm' we are seeking
was just named 'drm' in 3.40 ??

[jeremyszu (os369510)]

nope, drm is not nvidia_drm.

I don't have 5.4 + nvidia-340 environment.
Thus, I would like to get more information from you.

Could you please share
1. dkms status
2. find /lib/modules/$(uname -r) -name 'nvidia*'
3. your "/var/lib/dkms/" probably have nvidia-340 build log with your current kernel, could you please share the contents from the directory?
(e.g. /var/lib/dkms/tp_smapi/kernel-5.14.0-1036-oem-x86_64)

[ysfr (ysfr)]

Thanks for investigating !
below is what you requested ...

1.dkms status
nvidia-340, 340.108, 5.4.0-109-generic, x86_64: installed
nvidia-340, 340.108, 5.4.0-110-generic, x86_64: installed
virtualbox, 6.1.32, 5.4.0-109-generic, x86_64: installed
virtualbox, 6.1.32, 5.4.0-110-generic, x86_64: installed

2.find /lib/modules/$(uname -r) -name 'nvidia*'
/lib/modules/5.4.0-110-generic/kernel/drivers/char/drm/nvidia.ko
/lib/modules/5.4.0-110-generic/kernel/drivers/net/ethernet/nvidia
/lib/modules/5.4.0-110-generic/kernel/drivers/video/fbdev/nvidia
/lib/modules/5.4.0-110-generic/kernel/drivers/video/fbdev/nvidia/nvidiafb.ko
/lib/modules/5.4.0-110-generic/kernel/drivers/video/nvidia_uvm.ko

3.Found /var/lib/dkms/nvidia-340/340.108/5.4.0-110-generic/x86_64/log/make.log
(also reachable from symlink /var/lib/dkms/nvidia-340/kernel-5.4.0-110-generic-x86_64)
> see attachement

[jeremyszu (os369510)]

looks like nvidia-drm is not built in nvidia-340.
could you report the other bug for it takes 10 seconds when booting in nvidia-340?
and we can leave the information there.

[ysfr (ysfr)]

Thanks, just opened a separate bug:
[nvidia340] display hangs 10s on boot with nvidia-340
https://bugs.launchpad.net/ubuntu/+source/gdm3/+bug/1973604

[Brian Murray (brian-murray)]

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.

[Julian Andres Klode (juliank)]

ubuntu-sponsors was still subscribed and a task open for gdm3/focal. AFAICT, the issue was somehow resolved in ubuntu-drivers in focal and there's nothing to actually sponsor.

If sponsorship is still needed, please provide more details and resubscribe ubuntu-sponsors.