Ubuntu 22.04 in the middle of updating drivers from 530.41.03 to 535.54.03 video output cuts out

[Rovano]

[Impact]

The current version nvidia-graphics-drivers-535 and nvidia-graphics-drivers-535-server, from Focal on, use the default values (from the debian compat 10) for dh_installsystemd which cause all the systemd services to be restarted on upgrade. This will cause the screen to go blank and never wake up.

[Test Case]

Install the nvidia-driver:
sudo ubuntu-drivers install nvidia:535

Try re-installing nvidia-kernel-common-535 and nvidia-compute-utils-535:

sudo apt --reinstall nvidia-kernel-common-535 nvidia-compute-utils-535

The screen will go black.

[Where problems could occur]

On servers with no display server in use, if, for some reason, restarting services such as nvidia-persistenced on upgrade is a desired feature, this will not take place any more.

_________
https://forums.developer.nvidia.com/t/ubuntu-22-04-in-the-middle-of-updating-drivers-from-530-41-03-to-535-54-03-video-input-cuts-out/258588

https://forums.linuxmint.com/viewtopic.php?p=2344294

timeline 10:36 - 10:39
switch to TTY not helped-no video output, SSH not tested

After emergency sync, reboot (by sysrq), is all fine.

Jul 3 10:37:14 Panter nvidia-persistenced: Received signal 15
Jul 3 10:37:14 Panter systemd[1]: Stopping NVIDIA Persistence Daemon...
Jul 3 10:37:14 Panter nvidia-persistenced: Socket closed.
Jul 3 10:37:14 Panter nvidia-persistenced: PID file unlocked.
Jul 3 10:37:14 Panter nvidia-persistenced: PID file closed.
Jul 3 10:37:14 Panter nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
Jul 3 10:37:14 Panter nvidia-persistenced: Shutdown (1085)
Jul 3 10:37:14 Panter systemd[1]: nvidia-persistenced.service: Deactivated successfully.
Jul 3 10:37:14 Panter systemd[1]: Stopped NVIDIA Persistence Daemon.
Jul 3 10:37:19 Panter systemd[1]: Reloading.
Jul 3 10:37:19 Panter systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab...
Jul 3 10:37:19 Panter systemd[1]: Starting NVIDIA system hibernate actions...
Jul 3 10:37:19 Panter hibernate: nvidia-hibernate.service
Jul 3 10:37:19 Panter logger[9260]: <13>Jul 3 10:37:19 hibernate: nvidia-hibernate.service
Jul 3 10:37:19 Panter kernel: [ 424.892353] snd_hda_codec_hdmi hdaudioC0D0: HDMI: invalid ELD data byte 16
Jul 3 10:37:19 Panter systemd[1]: nvidia-hibernate.service: Deactivated successfully.
Jul 3 10:37:19 Panter systemd[1]: Finished NVIDIA system hibernate actions.
Jul 3 10:37:20 Panter systemd[1]: Reloading.
Jul 3 10:37:20 Panter systemd[1]: Starting NVIDIA system resume actions...
Jul 3 10:37:20 Panter suspend: nvidia-resume.service
Jul 3 10:37:20 Panter logger[9305]: <13>Jul 3 10:37:20 suspend: nvidia-resume.service
Jul 3 10:37:20 Panter systemd[1]: nvidia-resume.service: Deactivated successfully.
Jul 3 10:37:20 Panter systemd[1]: Finished NVIDIA system resume actions.
Jul 3 10:37:20 Panter acpid: client 1346[0:0] has disconnected
Jul 3 10:37:20 Panter rtkit-daemon[1537]: Supervising 7 threads of 3 processes of 1 users.
Jul 3 10:37:20 Panter rtkit-daemon[1537]: Successfully made thread 9312 of process 1530 owned by '1000' RT at priority 5.
Jul 3 10:37:20 Panter rtkit-daemon[1537]: Supervising 8 threads of 3 processes of 1 users.
Jul 3 10:37:20 Panter acpid: client connected from 1346[0:0]
Jul 3 10:37:20 Panter acpid: 1 client rule loaded
Jul 3 10:37:20 Panter rtkit-daemon[1537]: Supervising 7 threads of 3 processes of 1 users.
Jul 3 10:37:20 Panter rtkit-daemon[1537]: Successfully made thread 9315 of process 1530 owned by '1000' RT at priority 5.
Jul 3 10:37:20 Panter rtkit-daemon[1537]: Supervising 8 threads of 3 processes of 1 users.
Jul 3 10:37:20 Panter systemd[1]: Reloading.
Jul 3 10:37:21 Panter systemd[1]: Starting NVIDIA system suspend actions...
Jul 3 10:37:21 Panter suspend: nvidia-suspend.service
Jul 3 10:37:21 Panter logger[9348]: <13>Jul 3 10:37:21 suspend: nvidia-suspend.service
Jul 3 10:37:21 Panter kernel: [ 426.645430] snd_hda_codec_hdmi hdaudioC0D0: HDMI: invalid ELD data byte 12
Jul 3 10:37:21 Panter systemd[1]: nvidia-suspend.service: Deactivated successfully.
Jul 3 10:37:21 Panter systemd[1]: Finished NVIDIA system suspend actions.
Jul 3 10:37:21 Panter systemd[1]: Reloading.
Jul 3 10:37:22 Panter systemd[1]: Starting NVIDIA system hibernate actions...
Jul 3 10:37:22 Panter hibernate: nvidia-hibernate.service
Jul 3 10:37:22 Panter logger[9399]: <13>Jul 3 10:37:22 hibernate: nvidia-hibernate.service
Jul 3 10:37:22 Panter systemd[1]: Starting NVIDIA system resume actions...
Jul 3 10:37:22 Panter suspend: nvidia-resume.service
Jul 3 10:37:22 Panter logger[9403]: <13>Jul 3 10:37:22 suspend: nvidia-resume.service
Jul 3 10:37:22 Panter nvidia-sleep.sh[9400]: /usr/bin/nvidia-sleep.sh: řádek 20: echo: chyba zápisu: Chyba vstupu/výstupu
Jul 3 10:37:22 Panter systemd[1]: Starting NVIDIA system suspend actions...
Jul 3 10:37:22 Panter systemd[1]: nvidia-hibernate.service: Main process exited, code=exited, status=1/FAILURE
Jul 3 10:37:22 Panter suspend: nvidia-suspend.service
Jul 3 10:37:22 Panter systemd[1]: nvidia-hibernate.service: Failed with result 'exit-code'.
Jul 3 10:37:22 Panter logger[9407]: <13>Jul 3 10:37:22 suspend: nvidia-suspend.service
Jul 3 10:37:22 Panter systemd[1]: Failed to start NVIDIA system hibernate actions.
Jul 3 10:37:22 Panter kernel: [ 427.641772] snd_hda_codec_hdmi hdaudioC0D0: HDMI: invalid ELD data byte 0
Jul 3 10:37:22 Panter systemd[1]: nvidia-resume.service: Deactivated successfully.
Jul 3 10:37:22 Panter systemd[1]: Finished NVIDIA system resume actions.
Jul 3 10:37:22 Panter systemd[1]: nvidia-suspend.service: Deactivated successfully.
Jul 3 10:37:22 Panter systemd[1]: Finished NVIDIA system suspend actions.
Jul 3 10:37:24 Panter systemd[1]: Reloading.
Jul 3 10:37:24 Panter systemd[1]: Starting Daily apt download activities...
Jul 3 10:37:24 Panter systemd[1]: Reloading.
Jul 3 10:37:25 Panter systemd[1]: apt-daily.service: Deactivated successfully.
Jul 3 10:37:25 Panter systemd[1]: Finished Daily apt download activities.

[Rovano (rovano)]

edited/shorted syslog dmesg dpkg.log Xorg.0.log

And yes. I'm a cheater because this is from Linux Mint 21.1. But the same packages base. Forgive me then.

[Manfred Hampl (m-hampl)]

1. You are not supposed to confirm your own bug reports
2. You are not running Ubuntu, but Linux Mint.
Support for Linux Mint can be found at https://www.linuxmint.com/contactus.php and https://forums.linuxmint.com/

[Rovano (rovano)]

On the LM forum they write that some changes were made to the packages in Ubuntu. Also, there is no problem with the driver from Nvidia's website.

Manfred Hampl (m-hampl) wrote on 2023-07-03: #2

1. You are not supposed to confirm your own bug reports

I changed it first because it was obvious there was a reason for it here. This has been confirmed to me by multiple people elsewhere.
We got the version of Linux Mint wrong. I'm not going on 21.2, but on 21.1 (22.04.xx)
This is not request for support. This is a bug.

https://launchpad.net/ubuntu/+source/nvidia-graphics-drivers-535

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alexander Brissman (alexbriss)]

Same thing happened to me when updating from 525 to 535 via additional drivers on Ubuntu 23.04. Video output completely cuts off, couldn't switch to TTY. After a reboot no Nvidia drivers where loaded. In the journal I see:

juli 06 06:15:03 ubuntu gdm3[2630]: modprobe: FATAL: Module nvidia not found in directory /lib/modules/6.2.0-24-generic

no make.log under /var/lib/dkms/nvidia

Edit: Ran the following commands, the driver now appears to be working.

sudo dpkg --configure -a (to fix broken state)
sudo apt install nvidia-dkms-535
flatpak update (otherwise flatpak apps wouldn't recognize my GPU)
sudo reboot

[Daniel Eckl (daniel-eckl)]

Can confirm this on Lunar when updating from 530.41.03-0ubuntu2 to 535.54.03-0ubuntu0.23.04.2

Educated guess: It seems that some newly installed nvidia systemd unit is being restarted but cannot start because new user space binaries mismatch old kernel module. I also saw nvidia-persistenced.service being restarted, but I'm not sure if that's the culprit. Screens then go dark, but X is still running as do all X applications, so update process can finish if given enough time and if you reboot afterwards it works. SSH still working as well. systemctl reboot works at first and takes network down but gets stuck tho, so in the end you need a hard reset or sysrq reboot.

[Keith Myers (keith-myers)]

I had the same issue on upgrading five hosts from the 525 metapackage to the 535 metapackage via apt install.

The screen goes blank right at the building process and you lose control of the keyboard and mouse.

But the build process continues normally unseen. As long as you wait long enough for the kernel build with the new drivers, when you do a host reboot/reset, you will find the new 535 drivers installed correctly with no errors in the logs.

I just wait for the hard drive access light to quit blinking showing the writes, wait a minute to be sure it has completed and reboot.

[Stéphane (stephane-treboux)]

I faced the same issue and waited some time but it did not help.
My graphical session was a Wayland session so maybe apt was killed with Wayland, which would not have happened with X (can someone confirm?).

Mint users reported the same issue here: https://forums.linuxmint.com/viewtopic.php?t=399231

[Stéphane (stephane-treboux)]

I had to update from nvidia-driver-525 to nvidia-driver-535 again today.
I started the Ubuntu driver manager from a X session so even though the screen went black installation was successful.

[Night Train (nighttrain)]

Same thing happened to me when updating from 535.54 to 535.86 via jammy-updates on Ubuntu 22.04.

The screen goes blank right at the building process.

I just wait for 10 minutes, then ctrl+alt+del and the system reboot after other time.

Ubuntu 22.04 not completed reboot, then ctrl+alt+F3 and I log in shell, then sudo dpkg --configure -a complete the installation.

sudo reboot now and the system restarts normally.

[Rovano (rovano)]

Night Train (nighttrain) wrote on 2023-08-01:
Yep. Same here.
Unplug cable from card and plug back, nothing. No signal for monitor.

[stupid user (mc6312)]

Same problem with the nvidia-driver-535 in Ubuntu 23.04.

[Tiziano Tommasini (elenio)]

Same problem with Ubuntu 23.04 and nvidia driver version 535.104.05.

[Dylan Borg (borgdylan)]

The supposed fix of not starting systemd services did nothing to fix the issue as I upgraded to 535.104.04 today on UBuntu 23.10 (mantic).

[Night Train (nighttrain)]

The problem also persists with 535.113.1
the trouble is that the monitors turn off!

[Mark Fraser (launchpad-mfraz)]

I have found that I can still SSH into the computer after the screen has gone blank, but issuing a reboot or shutdown command doesn't completely shutdown the computer and I have to hold the power button so that I can shut it down.

[Mark Fraser (launchpad-mfraz)]

Sep 28 17:25:25 Rachael systemd[1]: Starting NVIDIA system hibernate actions...
Sep 28 17:25:25 Rachael systemd[1]: Starting NVIDIA system resume actions...
Sep 28 17:25:25 Rachael hibernate: nvidia-hibernate.service
Sep 28 17:25:25 Rachael logger[140999]: <13>Sep 28 17:25:25 hibernate: nvidia-hibernate.service
Sep 28 17:25:25 Rachael suspend: nvidia-resume.service
Sep 28 17:25:25 Rachael logger[141000]: <13>Sep 28 17:25:25 suspend: nvidia-resume.service
Sep 28 17:25:25 Rachael systemd[1]: nvidia-resume.service: Deactivated successfully.
Sep 28 17:25:25 Rachael systemd[1]: Finished NVIDIA system resume actions.
Sep 28 17:25:25 Rachael systemd[1]: Starting NVIDIA system suspend actions...
Sep 28 17:25:25 Rachael suspend: nvidia-suspend.service
Sep 28 17:25:25 Rachael logger[141006]: <13>Sep 28 17:25:25 suspend: nvidia-suspend.service
Sep 28 17:25:26 Rachael acpid: client 1329[0:0] has disconnected
Sep 28 17:25:26 Rachael nvidia-sleep.sh[141001]: /usr/bin/nvidia-sleep.sh: line 20: echo: write error: Input/output error
Sep 28 17:25:26 Rachael systemd[1]: nvidia-suspend.service: Deactivated successfully.
Sep 28 17:25:26 Rachael systemd[1]: Finished NVIDIA system suspend actions.
Sep 28 17:25:26 Rachael systemd[1]: nvidia-hibernate.service: Main process exited, code=exited, status=1/FAILURE
Sep 28 17:25:26 Rachael systemd[1]: nvidia-hibernate.service: Failed with result 'exit-code'.
Sep 28 17:25:26 Rachael systemd[1]: Failed to start NVIDIA system hibernate actions.

[Lastique (andysem)]

See also: https://github.com/tseliot/nvidia-graphics-drivers/issues/61.

[Benjamin Bock (crapshit)]

Same happened to me today.
I'm using a Lenovo ThinkPad P16 Gen 1 with Ubuntu 22.04.
I was using "nvidia-firmware-535-535.86.05" before the update and during the update to "nvidia-firmware-535-535.113.01" my screen went black.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted nvidia-graphics-drivers-535-server (535.104.12-0ubuntu0.23.04.2) for lunar have finished running.
The following regressions have been reported in tests triggered by the package:

pyopencl/2022.3.1-2build1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/lunar/update_excuses.html#nvidia-graphics-drivers-535-server

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Rovano (rovano)]

The update arrived. I just don't know if it works, because of course someone didn't think to distribute the repair patch first, but it was installed together with the Nvidia driver, so the monitor no signal again. I see room for reflection here.

Today I also happened to meet a person on the Ubuntu forum who damaged his filesystem in this way because he does not know emergency or the console. And so you probably turned off your computer hard and EXT4 was not breathing.

I thank the others for their work.

PS: From the blacked out screen I went the CTRL+ALT+F1 and right ALT+SYSRQ+REISUB route.
It worked the first time and restart succesfully done.

[Andreas Kromke (dandreas)]

I just installed 535.113 on 23.04, extremely carefully by using an SSH connection. During installation I watched the login screen, and suddenly it went black and never came back.

After installation I ran "sudo reboot", still via SSH, was automatically logged out, and the system --- FROZE.

After having waited about a minute I tried the Alt-PrtScr-reisub magic sequence, and after the second (!) attempt the system rebooted automatically into Windows, as if the firmware skipped a boot step. I rebooted from Windows to Ubuntu and was greeted with some "file system clean" messages. Obviously the volumes had not been unmounted correctly, despite the magic key sequence.

However, the system is running again, and the 535.113 drivers seem to be installed successfully.

[dlotton (yellow56)]

My experience has been identical to Andreas above. I'm on Mint Linux 21.2, updating to 515.113 (and also a prior driver version). I've experienced this on five different machines with various different nVidia cards, all but one running 5.15 kernels, and the remaining running a 6.5 kernel.

After the screen goes black, I've tried to use CTRL_ALT+F1 to go to a terminal (outside of the display manager), but that doesn't yield anything.

I can ssh to the machine. I've tried restarting the display manager from the ssh session, but that doesn't appear to do anything. When I try to reboot or shutdown from the ssh session, the session is terminated (as one would expect), but the machine never appears to complete the shutdown sequence and a forced shutdown (hold power button) is required.

When the machine reboots, the new driver is installed, and everything seems to be functioning correctly.

It is deeply concerning to have to force a reboot, wondering if the file system is going to be trashed as a result.

[Ernst Persson (ernstp)]

Same issue in #2027614 ?

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535-server - 535.104.12-0ubuntu2

---------------
nvidia-graphics-drivers-535-server (535.104.12-0ubuntu2) mantic; urgency=medium

  * debian/rules:
    - Add override_dh_installsystemd.
    - Pass in --no-stop-on-upgrade --no-restart-after-upgrade to
      dh_installsystemd (LP: #2025640).

 -- Alberto Milone <email address hidden> Fri, 29 Sep 2023 19:12:51 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535 - 535.113.01-0ubuntu3

---------------
nvidia-graphics-drivers-535 (535.113.01-0ubuntu3) mantic; urgency=medium

  * debian/rules:
    - Add override_dh_installsystemd.
    - Pass in dh_installsystemd --no-restart-after-upgrade.

nvidia-graphics-drivers-535 (535.113.01-0ubuntu2) mantic; urgency=medium

  * debian/rules:
   - Pass in --no-stop-on-upgrade to dh_installsystemd.
     Prevent dh_installsystemd from stopping services (LP: #2025640).

 -- Alberto Milone <email address hidden> Fri, 29 Sep 2023 18:07:07 +0000

[Daniel Eckl (daniel-eckl)]

Fix confirmed with 535.113.01-0ubuntu0.23.04.3 lunar proposed

[michalq (michalq)]

Is it possible to estimate when the final fix will be available at standard Ubuntu Jammy (!!!) repos?

[Alberto Milone (albertomilone)]

Verified in Lunar, Jammy, and Focal.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535 - 535.113.01-0ubuntu0.23.04.3

---------------
nvidia-graphics-drivers-535 (535.113.01-0ubuntu0.23.04.3) lunar; urgency=medium

  * debian/rules:
    - Add override_dh_installsystemd.
    - Pass in dh_installsystemd --no-restart-after-upgrade.

nvidia-graphics-drivers-535 (535.113.01-0ubuntu0.23.04.2) lunar; urgency=medium

  * debian/rules:
   - Pass in --no-stop-on-upgrade to dh_installsystemd.
     Prevent dh_installsystemd from stopping services (LP: #2025640).

 -- Alberto Milone <email address hidden> Fri, 29 Sep 2023 18:09:19 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535-server - 535.104.12-0ubuntu0.23.04.2

---------------
nvidia-graphics-drivers-535-server (535.104.12-0ubuntu0.23.04.2) lunar; urgency=medium

  * debian/rules:
    - Add override_dh_installsystemd.
    - Pass in --no-stop-on-upgrade --no-restart-after-upgrade to
      dh_installsystemd (LP: #2025640).

 -- Alberto Milone <email address hidden> Fri, 29 Sep 2023 19:16:20 +0000

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for nvidia-graphics-drivers-535 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535 - 535.113.01-0ubuntu0.22.04.3

---------------
nvidia-graphics-drivers-535 (535.113.01-0ubuntu0.22.04.3) jammy; urgency=medium

  * debian/rules:
    - Add override_dh_installsystemd.
    - Pass in dh_installsystemd --no-restart-after-upgrade.

nvidia-graphics-drivers-535 (535.113.01-0ubuntu0.22.04.2) jammy; urgency=medium

  * debian/rules:
   - Pass in --no-stop-on-upgrade to dh_installsystemd.
     Prevent dh_installsystemd from stopping services (LP: #2025640).

 -- Alberto Milone <email address hidden> Fri, 29 Sep 2023 18:11:41 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535-server - 535.104.12-0ubuntu0.22.04.2

---------------
nvidia-graphics-drivers-535-server (535.104.12-0ubuntu0.22.04.2) jammy; urgency=medium

  * debian/rules:
    - Add override_dh_installsystemd.
    - Pass in --no-stop-on-upgrade --no-restart-after-upgrade to
      dh_installsystemd (LP: #2025640).

 -- Alberto Milone <email address hidden> Fri, 29 Sep 2023 19:17:05 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535 - 535.113.01-0ubuntu0.20.04.3

---------------
nvidia-graphics-drivers-535 (535.113.01-0ubuntu0.20.04.3) focal; urgency=medium

  * debian/rules:
    - Add override_dh_installsystemd.
    - Pass in dh_installsystemd --no-restart-after-upgrade.

nvidia-graphics-drivers-535 (535.113.01-0ubuntu0.20.04.2) focal; urgency=medium

  * debian/rules:
   - Pass in --no-stop-on-upgrade to dh_installsystemd.
     Prevent dh_installsystemd from stopping services (LP: #2025640).

 -- Alberto Milone <email address hidden> Fri, 29 Sep 2023 18:03:34 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535-server - 535.104.12-0ubuntu0.20.04.2

---------------
nvidia-graphics-drivers-535-server (535.104.12-0ubuntu0.20.04.2) focal; urgency=medium

  * debian/rules:
    - Add override_dh_installsystemd.
    - Pass in --no-stop-on-upgrade --no-restart-after-upgrade to
      dh_installsystemd (LP: #2025640).

 -- Alberto Milone <email address hidden> Fri, 29 Sep 2023 19:17:37 +0000

[Night Train (nighttrain)]

Today I successfully updated to nvidia-graphics-drivers-535 (535.113.01-0ubuntu0.22.04.3) jammy.

Thanks Alberto