Update the 535 driver series - 25/09/2023

[Alberto Milone]

[Impact]
These releases provide both bug fixes and new features, and we would like to
make sure all of our users have access to these improvements.

See the changelog entry below for a full list of changes and bugs.

[Test Case]
The following development and SRU process was followed:
https://wiki.ubuntu.com/NVidiaUpdates

Certification test suite must pass on a range of hardware:
https://git.launchpad.net/plainbox-provider-sru/tree/units/sru.pxu

The QA team that executed the tests will be in charge of attaching the artifacts and console output of the appropriate run to the bug. nVidia maintainers team members will not mark ‘verification-done’ until this has happened.

[Regression Potential]
In order to mitigate the regression potential, the results of the
aforementioned system level tests are attached to this bug.

[Discussion]

[Changelog]

535 (535.113.01):
https://www.nvidia.com/Download/driverResults.aspx/211711/en-us/

535-server (535.104.12):
https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-535-104-12/index.html

[Francis Ginther (fginther)]

I've completed the cuda based testing for these drivers. This includes

 * amd64 testing for the DKMS for bionic, focal, jammy and lunar.
 * arm64 testing for the DKMS for focal and jammy
 * amd64 testing of the LRM packages against the generic kernel for bionic, focal, jammy and lunar
 * amd64 testing of the LRM packages against the nvidia kernel for jammy

This matches the testing done for prior driver releases.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535-server - 535.104.12-0ubuntu0.23.04.1

---------------
nvidia-graphics-drivers-535-server (535.104.12-0ubuntu0.23.04.1) lunar; urgency=medium

  * New upstream release (LP: #2037266):
    - Fixed an issue where the NVSwitch driver would not retrain
      NVLinks on init correctly on HGX 8 H100, in case they faulted
      earlier (such a due to GPU resets). This would result in links
      being down and CUDA workloads failing with "system not yet
      initialized" error. The issue was introduced in the 535.86.10
      driver and fixed in 535.104.12 and later drivers.

 -- Alberto Milone <email address hidden> Mon, 25 Sep 2023 16:18:16 +0000

[Andy Whitcroft (apw)]

The verification of the Stable Release Update for nvidia-graphics-drivers-535-server has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535 - 535.113.01-0ubuntu0.23.04.1

---------------
nvidia-graphics-drivers-535 (535.113.01-0ubuntu0.23.04.1) lunar; urgency=medium

  * New upstream release (LP: #2037266):
    - Fixed a bug that could cause GPU memory utilization to be
      reported incorrectly for Multi-Instance GPU (MIG) partitions on
      Grace Hopper systems.
    - Fixed a bug that intermittently caused the display to freeze
      when resuming from suspend on some Ada GPUs.

 -- Alberto Milone <email address hidden> Mon, 25 Sep 2023 09:32:34 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535-server - 535.104.12-0ubuntu0.22.04.1

---------------
nvidia-graphics-drivers-535-server (535.104.12-0ubuntu0.22.04.1) jammy; urgency=medium

  * New upstream release (LP: #2037266):
    - Fixed an issue where the NVSwitch driver would not retrain
      NVLinks on init correctly on HGX 8 H100, in case they faulted
      earlier (such a due to GPU resets). This would result in links
      being down and CUDA workloads failing with "system not yet
      initialized" error. The issue was introduced in the 535.86.10
      driver and fixed in 535.104.12 and later drivers.

 -- Alberto Milone <email address hidden> Mon, 25 Sep 2023 16:30:44 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535 - 535.113.01-0ubuntu0.22.04.1

---------------
nvidia-graphics-drivers-535 (535.113.01-0ubuntu0.22.04.1) jammy; urgency=medium

  * New upstream release (LP: #2037266):
    - Fixed a bug that could cause GPU memory utilization to be
      reported incorrectly for Multi-Instance GPU (MIG) partitions on
      Grace Hopper systems.
    - Fixed a bug that intermittently caused the display to freeze
      when resuming from suspend on some Ada GPUs.

 -- Alberto Milone <email address hidden> Mon, 25 Sep 2023 09:45:05 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535-server - 535.104.12-0ubuntu0.20.04.1

---------------
nvidia-graphics-drivers-535-server (535.104.12-0ubuntu0.20.04.1) focal; urgency=medium

  * New upstream release (LP: #2037266):
    - Fixed an issue where the NVSwitch driver would not retrain
      NVLinks on init correctly on HGX 8 H100, in case they faulted
      earlier (such a due to GPU resets). This would result in links
      being down and CUDA workloads failing with "system not yet
      initialized" error. The issue was introduced in the 535.86.10
      driver and fixed in 535.104.12 and later drivers.

 -- Alberto Milone <email address hidden> Mon, 25 Sep 2023 16:29:52 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535 - 535.113.01-0ubuntu0.20.04.1

---------------
nvidia-graphics-drivers-535 (535.113.01-0ubuntu0.20.04.1) focal; urgency=medium

  * New upstream release (LP: #2037266):
    - Fixed a bug that could cause GPU memory utilization to be
      reported incorrectly for Multi-Instance GPU (MIG) partitions on
      Grace Hopper systems.
    - Fixed a bug that intermittently caused the display to freeze
      when resuming from suspend on some Ada GPUs.

 -- Alberto Milone <email address hidden> Mon, 25 Sep 2023 10:02:47 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535-server - 535.104.12-0ubuntu2

---------------
nvidia-graphics-drivers-535-server (535.104.12-0ubuntu2) mantic; urgency=medium

  * debian/rules:
    - Add override_dh_installsystemd.
    - Pass in --no-stop-on-upgrade --no-restart-after-upgrade to
      dh_installsystemd (LP: #2025640).

 -- Alberto Milone <email address hidden> Fri, 29 Sep 2023 19:12:51 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-535 - 535.113.01-0ubuntu3

---------------
nvidia-graphics-drivers-535 (535.113.01-0ubuntu3) mantic; urgency=medium

  * debian/rules:
    - Add override_dh_installsystemd.
    - Pass in dh_installsystemd --no-restart-after-upgrade.

nvidia-graphics-drivers-535 (535.113.01-0ubuntu2) mantic; urgency=medium

  * debian/rules:
   - Pass in --no-stop-on-upgrade to dh_installsystemd.
     Prevent dh_installsystemd from stopping services (LP: #2025640).

 -- Alberto Milone <email address hidden> Fri, 29 Sep 2023 18:07:07 +0000