After suspend CUDA is not working any more

[Thomas]

After suspend the CUDA library can not get a connection to the GPU any more.

There is following workaround possible:

sudo rmmod nvidia_uvm
sudo modprobe nvidia_uvm

This could be automated in an event script however it is not helping if there are applications running already which depend on CUDA connection. The module will be refused to unload due to its usage then.

System environment:

Description: Ubuntu 20.04.2 LTS
Release: 20.04

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1650 On | 00000000:01:00.0 Off | N/A |
| N/A 53C P5 9W / N/A | 2015MiB / 3911MiB | 5% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 2924 G /usr/lib/xorg/Xorg 768MiB |
| 0 N/A N/A 5018 G xfwm4 2MiB |
| 0 N/A N/A 8227 G /opt/idea/jbr/bin/java 2MiB |
| 0 N/A N/A 255790 G ...AAAAAAAAA= --shared-files 66MiB |
| 0 N/A N/A 267569 C+G ....jdks/openjdk-15/bin/java 1168MiB |
+-----------------------------------------------------------------------------+

[Thomas (t-hartwig)]

It looks like following is working also. Create a file /etc/modules-load.d/nvidia.conf with contents below and restart.

options nvidia NVreg_PreserveVideoMemoryAllocations=1
options nvidia NVreg_TemporaryFilePath=/run

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Roel (roel15)]

Still an issue with Driver Version 515.65.01 and Ubuntu 20.04.4. "sudo rmmod nvidia_uvm" does work, but requires me to kill my current Jupyter kernel.

[Dev El Cuy (develcuy)]

The fix proposed at #1 doesn't work anymore. I have to do #3, that one works.

FTR, I got a Quadro P1000 on Ubuntu 22.04 with up to date NVIDIA drivers