nvidia-dkms-* FTBS with linux 6.5

[Paolo Pisati]

[Impact]

...
In file included from /var/lib/dkms/nvidia/390.157/build/common/inc/nv-linux.h:21,
                 from /var/lib/dkms/nvidia/390.157/build/nvidia/nv-instance.c:13:
/var/lib/dkms/nvidia/390.157/build/common/inc/nv-mm.h: In function ‘NV_GET_USER_PAGES_REMOTE’:
/var/lib/dkms/nvidia/390.157/build/common/inc/nv-mm.h:164:45: error: passing argument 1 of ‘get_user_pages_remote’ from incompatible pointer type [-Werror=incompatible-pointer-types]
  164 | return get_user_pages_remote(tsk, mm, start, nr_pages, flags,
      | ^~~
      | |
      | struct task_struct *
...

[Fix]

Apply the attached fix.

[How to test]

Install (and build) the patched packet.

[Regression potential]

The fix is composed of two patches:

1) the first patch simply garbage collect a reference to a function that was never used but that had the API changed in Linux 6.5 - so, it's a trivial change.

2) the second patch actually reimplement part of the vma scanning that was removed in __get_user_pages_locked() in upstream commit b2cac248191b7466c5819e0da617b0705a26e197 "mm/gup: removed vmas
array from internal GUP functions" - here is where most likely any regression could be found.

[Paolo Pisati (p-pisati)]

New patch for the 470 driver

[Paolo Pisati (p-pisati)]

525-server fix for Linux 6.5

[Paolo Pisati (p-pisati)]

450-server Linux 6.5 fix.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-525-server - 525.125.06-0ubuntu2

---------------
nvidia-graphics-drivers-525-server (525.125.06-0ubuntu2) mantic; urgency=medium

  * debian/dkms_nvidia/patches/buildfix_kernel_6.5-fix-get_user_pages-get_user_pages_remote.patch,
    debian/dkms_nvidia/patches/buildfix_kernel_6.5-fix-pin_user_pages.patch,
    debian/dkms_nvidia/patches/buildfix_kernel_6.5-fix-pin_user_pages_remote.patch:
    - Fix build with Linux 6.5 (LP: #2028165)

 -- Paolo Pisati <email address hidden> Wed, 26 Jul 2023 09:50:27 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-470 - 470.199.02-0ubuntu2

---------------
nvidia-graphics-drivers-470 (470.199.02-0ubuntu2) mantic; urgency=medium

  * debian/dkms_nvidia/patches/buildfix_kernel_6.5-fix-get_user_pages-get_user_pages_remote.patch,
    debian/dkms_nvidia/patches/buildfix_kernel_6.5-fix-pin_user_pages.patch,
    debian/dkms_nvidia/patches/buildfix_kernel_6.5-fix-pin_user_pages_remote.patch:
    - Fix build with Linux 6.5 (LP: #2028165)

 -- Paolo Pisati <email address hidden> Fri, 21 Jul 2023 10:56:36 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-470-server - 470.199.02-0ubuntu2

---------------
nvidia-graphics-drivers-470-server (470.199.02-0ubuntu2) mantic; urgency=medium

  * debian/dkms_nvidia/patches/buildfix_kernel_6.5-fix-get_user_pages-get_user_pages_remote.patch,
    debian/dkms_nvidia/patches/buildfix_kernel_6.5-fix-pin_user_pages.patch,
    debian/dkms_nvidia/patches/buildfix_kernel_6.5-fix-pin_user_pages_remote.patch:
    - Fix build with Linux 6.5 (LP: #2028165)

 -- Paolo Pisati <email address hidden> Tue, 25 Jul 2023 15:23:06 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-450-server - 450.248.02-0ubuntu2

---------------
nvidia-graphics-drivers-450-server (450.248.02-0ubuntu2) mantic; urgency=medium

  * debian/dkms_nvidia/patches/buildfix_kernel_6.5-fix-get_user_pages-get_user_pages_remote.patch,
    debian/dkms_nvidia/patches/buildfix_kernel_6.5-fix-pin_user_pages.patch,
    debian/dkms_nvidia/patches/buildfix_kernel_6.5-fix-pin_user_pages_remote.patch:
    - Fix build with Linux 6.5 (LP: #2028165)

 -- Paolo Pisati <email address hidden> Wed, 26 Jul 2023 15:21:26 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-525 - 525.125.06-0ubuntu4

---------------
nvidia-graphics-drivers-525 (525.125.06-0ubuntu4) mantic; urgency=medium

  * debian/open-kernel/patches/buildfix_kernel_6.5-fix-get_user_pages-get_user_pages_remote.patch,
    debian/open-kernel/patches/buildfix_kernel_6.5-fix-pin_user_pages.patch,
    debian/open-kernel/patches/uildfix_kernel_6.5-fix-pin_user_pages_remote.patch:
    - Fix build with Linux 6.5 for the -open variant (LP: #2028165)

 -- Paolo Pisati <email address hidden> Tue, 29 Aug 2023 08:57:20 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-390 - 390.157-0ubuntu8

---------------
nvidia-graphics-drivers-390 (390.157-0ubuntu8) mantic; urgency=medium

  * debian/dkms_nvidia.conf,
    debian/templates/dkms_nvidia.conf.in,
    debian/dkms_nvidia/patches/buildfix_kernel_6.5-garbage-collect-all-references-to-get_user.patch,
    debian/dkms_nvidia/patches/buildfix_kernel_6.5-handle-get_user_pages-vmas-argument-remova.patch:
    - Support linux 6.5 ABI (LP: #2028165).

 -- Paolo Pisati <email address hidden> Wed, 19 Jul 2023 13:06:56 +0000

[Daniel van Vugt (vanvugt)]

I've opened a jammy task for nvidia 390. We're getting a bunch of bug reports from jammy users complaining their Nvidia driver doesn't work anymore since they got automatically upgraded to kernel 6.5.

[Daniel van Vugt (vanvugt)]

This is a regression for jammy in the update of the linux-generic-hwe-22.04 package to version 6.5. So I guess that's more regression-update than regression-release. It won't be a regression-release unless the issue is still unresolved in 22.04.4.

[Sunit (sunit-parab)]

so are we saying this would be 'fixed' in 22.04.4? And when is that expected? Asking cause this has rather let me either using the noveau drive(which is laggy) or get a new graphic card(which am unsure would even work?)

[Daniel Letzeisen (dtl131)]

@Sunit: use the PPA until(if?) Ubuntu fixes this: https://launchpad.net/~dtl131/+archive/ubuntu/nvidiaexp

[Daniel van Vugt (vanvugt)]

If you need nvidia-390 on 22.04 then the best workaround I can suggest is to boot an older kernel like 6.2 instead.

[Daniel Letzeisen (dtl131)]

@vanvugt: Isn't 6.2 EOL now though?
https://ubuntu.com/about/release-cycle#ubuntu-kernel-release-cycle

[Daniel van Vugt (vanvugt)]

It's only a workaround. Sometimes workarounds force you to use older software.

[Sunit (sunit-parab)]

Thanks Daniel Letzesien that worked

[Daniel Letzeisen (dtl131)]

@Steve Langesak, I added the regression-release tag based on vanvugt's comment that this this would be such a bug if not fixed by 22.04.4 release. Please explain why you removed it.

[Brian Murray (brian-murray)]

Probably because regression-update is more appropriate as this was caused by a package update in a stable release of Ubuntu.

https://wiki.ubuntu.com/Bugs/Tags

[Dmitry Lapshin (lapshin-dv)]

Why nvidia-graphics-drivers-390 for jammy isn't assigned and just confirmed? @p-pisati was the last uploader of the package for jammy, and @albertomilone handled the mantic one, fix from the PPA seem to work fine.

[Daniel Letzeisen (dtl131)]

@Dmitry: Ubuntu devs made a conscious decision to drop the 390 driver and some other branches: https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers-390/+bug/2035189
So if they haven't fixed the Jammy version by now, it's probably not going to happen.

Good news though: I patched the package in my PPA for kernel 6.8 and added a version for Ubuntu 24.04.
You should be able to use the versions in the PPA until the year 2029.

I will try to keep up with future kernel version, but no guarantees. I'm very good at getting distracted by life and/or procrastinating.

[Test (3560holy)]

@dtl131 Why did the Ubuntu devs make this decision? Does this mean that the driver functionality will not work correctly and Ubuntu users will not be able to use Ubuntu?

[Daniel van Vugt (vanvugt)]

I don't think the "simpledrm" reason listed in 2035189 holds up. Unless you're in a virtual machine, SimpleDRM is only used for the few seconds of initrd, or for the duration of the disk unlock prompt. Even if old Nvidia drivers can't support SimpleDRM, they can support legacy framebuffers which we are continuing to ship in initrd. So old Nvidia drivers should remain usable, even if low res, during boot.

Also the unresolved Jammy task at the top of this page would not still say Confirmed if it was decided to never be fixed.

[Daniel Letzeisen (dtl131)]

@vanvugt: I appreciate you opening the Jammy task, but I'm not holding my breath for this to get fixed after reading https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers-390/+bug/2035189/comments/1
Basically, they knew this bug was coming when they removed the 390 package, but they seem to be okay with that as long as people can use 5.15

The 6.5 kernel has been out for months and 6.8 is coming down the pipe soon. The fix is simple and they already had the appropriate patches applied for mantic. So maybe this wasn't marked "Won't Fix" yet, but to me, the silence speaks volumes..

[Daniel van Vugt (vanvugt)]

Everyone is very busy with the 24.04 release as well as other responsibilities. So don't take silence to imply anything :)

[Daniel Letzeisen (dtl131)]

Sorry, but I don't believe that for a second. The fix is not difficult or time-consuming, and they already had a patched version.

"Eventually 6.5+ based jammy kernels will not have it in lrm either. And we will only keep it against older kernels in bionic..jammy (ga only)"
Unless they reverse ^that decision, it means Won't Fix. I'll believe otherwise when I see it.

[Daniel Letzeisen (dtl131)]

Bad news - I copied the packages I was using in my PPA to make 24.04/Noble packages, but they are broken (black screen /unbootable) according to the reports I've received, and my own personal test with a GTX950. The packages build successfully in the PPA, they install correctly, the kernel modules build correctly, and the kernel modules load correctly at boot, but Xorg and/or nvidia-smi are crashing.

If anyone has any ideas how to fix it, I'd appreciate it. If you want to test the Noble packages, see: https://launchpad.net/~dtl131/+archive/ubuntu/nv390test

[Ginés Utrera Pavón (myzras)]

Thank you very much @Daniel for your work, I am trying to install the package from the test repository, but I get a dependency error.

[Daniel Letzeisen (dtl131)]

Ginés: I'm guessing you have leftover packages from when I had the 390 package for noble in the nvidiaexp PPA. You probably need to remove at least libnvidia-compute-390 and then try again. This will help you see what packages you still have installed:
dpkg -l | grep 390 | grep ii

And I should have renamed the source package from nvidiaexp to nvidiatest, but this thought always seems to occur to me about 2 minutes after I upload the package to the PPA. *Shrug*

[Ginés Utrera Pavón (myzras)]

Thank you very much Daniel, indeed the libnvidia-compute-390 package was uninstalled. I was finally able to test the package, and the result was... black screen again. :(

[Daniel Letzeisen (dtl131)]

There's another ppa, "Nvidia-legacy" that has also packaged 390 for noble, but that fails on boot too. Well, at least I know it's not just my package.

I don't know what else to try. I've tried different kernels (liquorix, xanmod), older mainline kernels (6.5.11), replacing lightdm with sddm, and a bunch of other things I can't remember. Considering I don't personally use an Nvidia Fermi card, and it seems my package is okay, I don't have much motivation to spend more time on this. All I can say is good luck :(

[Ginés Utrera Pavón (myzras)]

Thank you very much for your work and effort, Daniel.

[Ginés Utrera Pavón (myzras)]

I have installed the latest nvidia-driver-390 package that has been uploaded in "nvidia-legacy" that supports kernel 6.8, but when I restart I have the black screen again...

[Daniel Letzeisen (dtl131)]

New idea: Try booting with nvidia-drm.modeset=1

[Ginés Utrera Pavón (myzras)]

Hi Daniel, yes I already tried but it didn't work...

[Sunit (sunit-parab)]

Any update on this one? as am struggling with 390 as it currently does not work with the latest kernel 6.5.34. I have to literally start twice for the 'repair' menu to load and select the older kernel 6.5.28 to get it working

[DiagonalArg (diagonalarg)]

I'm having to stay with 22.04, with a prayer that I won't be stuck with my over half dozen thinpads of the 520/530/540 generation will no longer be able to run Ubuntu.

[William Norfleet (wnorfl)]

Seems to work now under 6.8.0-40-generic

[Daniel van Vugt (vanvugt)]

Please clarify which driver version "seems to work now".

[William Norfleet (wnorfl)]

Noob here. I readily admit I don't know what I'm doing.

390.157 seems to work now:

$ grep "X Driver" /var/log/Xorg.0.log
[ 38.313] (II) NVIDIA dlloader X Driver 390.157 Wed Oct 12 09:21:41 UTC 2022

Boot failed after an upgrade to kernel 6.5, but boot works now (no more messing with grub configuration)

[Daniel van Vugt (vanvugt)]

That's great news, but surprising because the package hasn't changed for a year:
https://launchpad.net/ubuntu/+source/nvidia-graphics-drivers-390/390.157-0ubuntu0.22.04.2