LRM packages do not upgrade when the kernel is upgraded

[Francis Ginther]

Upgrading the kernel on a host with the nvidia 525-server driver LRM modules installed results in a kernel/module mismatch as the kernel upgrades, but the LRM packages do not. I've seen this so far with the 525-server and 535-server packages on jammy.

Here's how I produced this.

# Install the latest kernel in -updates (this is currently 5.15.0-78-generic)
$ sudo apt-get install kernel-testing--linux--full--generic

# Install the LRM packages for 525-server:
$ sudo apt-get install nvidia-headless-no-dkms-525-server linux-modules-nvidia-525-server-generic nvidia-utils-525-server

# At this point in time everything works, as indicated by `nvidia-smi`.
# Now attempt to install the latest kernel from the proposed PPA (now 5.15.0-79-generic):
$ sudo apt-add-repository ppa:canonical-kernel-team/proposed
$ sudo apt-get install --dry-run kernel-testing--linux--full--generic
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'linux-generic' instead of 'kernel-testing--linux--full--generic'
The following package was automatically installed and is no longer required:
  libnuma1
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  linux-headers-5.15.0-79 linux-headers-5.15.0-79-generic linux-headers-generic
  linux-image-5.15.0-79-generic linux-image-generic linux-modules-5.15.0-79-generic
  linux-modules-extra-5.15.0-79-generic
Suggested packages:
  fdutils linux-doc | linux-source-5.15.0 linux-tools
The following NEW packages will be installed:
  linux-headers-5.15.0-79 linux-headers-5.15.0-79-generic linux-image-5.15.0-79-generic
  linux-modules-5.15.0-79-generic linux-modules-extra-5.15.0-79-generic
The following packages will be upgraded:
  linux-generic linux-headers-generic linux-image-generic
3 upgraded, 5 newly installed, 0 to remove and 26 not upgraded.

# The above did not attempt to upgrade the LRM packages. However, if I try to update the LRM packages instead, it will attempt to upgrade both the LRM packages and the kernel:
$ sudo apt-get install nvidia-headless-no-dkms-525-serv[237/903]
modules-nvidia-525-server-generic nvidia-utils-525-server --dry-run
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
nvidia-headless-no-dkms-525-server is already the newest version (525.125.06-0ubuntu0.22.04.1
).
nvidia-utils-525-server is already the newest version (525.125.06-0ubuntu0.22.04.1).
The following package was automatically installed and is no longer required:
  libnuma1
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  linux-image-5.15.0-79-generic linux-modules-5.15.0-79-generic
  linux-modules-nvidia-525-server-5.15.0-79-generic
  linux-objects-nvidia-525-server-5.15.0-79-generic
  linux-signatures-nvidia-5.15.0-79-generic
Suggested packages:
  fdutils linux-doc | linux-source-5.15.0 linux-tools linux-headers-5.15.0-79-generic
  linux-modules-extra-5.15.0-79-generic
The following NEW packages will be installed:
  linux-image-5.15.0-79-generic linux-modules-5.15.0-79-generic
  linux-modules-nvidia-525-server-5.15.0-79-generic
  linux-objects-nvidia-525-server-5.15.0-79-generic
  linux-signatures-nvidia-5.15.0-79-generic
The following packages will be upgraded:
  linux-modules-nvidia-525-server-generic
1 upgraded, 5 newly installed, 0 to remove and 28 not upgraded.

Upgrading the kernel but not the LRMs results in a partially working system. It still boots, but the nvidia driver can no longer communicate with the hardware. A local fix would be to also upgrade the LRM packages or just do an 'apt-get dist-upgrade'.