Bug #2006408 “nvidia-uvm module does not exist, making CUDA and ...” : Bugs : nvidia-graphics-drivers-390 package : Ubuntu

Revision history for this message

Aaron Rainbolt (arraybolt3) wrote on 2023-02-07:

#1

clinfo_strace.log Edit (83.3 KiB, text/plain)

strace of "clinfo -l" from the machine of the affected user.

Aaron Rainbolt (arraybolt3) on 2023-02-07

description:	updated
summary:	- nvidia-uvm module does not exist, making CUDA unusable with the 390 - driver + nvidia-uvm module does not exist, making CUDA and OpenCL unusable with + the 390 driver

Revision history for this message

Aaron Rainbolt (arraybolt3) wrote on 2023-02-08:

#2

Download full text (12.7 KiB)

IRC conversation log (partially redacted for privacy of other users):

[Jan24 12:30:44] <Liver_K> BTW Oblomov, I was just looking over our convo a while ago about ICDs and ICD loaders, and I finally checked /etc/OpenCL/vendors and saw the nvidia icd, but I still don't see it in clinfo
[Jan24 12:33:09] <Liver_K> So you said that could mean that there is missing hardware, which is not true, or that there are missing libraries, which I am not sure how to look for
[Jan24 12:36:14] <Oblomov> ldd $(cat /etc/OpenCL/vendors/nvidia.icd)
[Jan24 12:38:27] <Liver_K> Yeah no such file or directory
[Jan24 12:42:54] <Oblomov> so the file is not there?
[Jan24 12:42:40] <Liver_K> Oblomov: libnvidia-opencl.so.1 is the output so it is there
[Jan24 12:43:12] <Oblomov> locate libnvidia-opencl.so.1
[Jan24 12:43:59] <Liver_K> Wait lol no I just thought about it, Oblomov where is it supposed to be?
[Jan24 12:44:10] <Liver_K> nvidia.icd is a regular file
[Jan24 12:44:36] <Oblomov> yes
[Jan24 12:44:51] <Oblomov> the file contains a string which is the name or path of the library
[Jan24 12:45:07] <Oblomov> but the actual library is searched for in the due paths
[Jan24 12:44:50] <Liver_K> Oh so libnvidia-opencl.so.1 is the name of an apt package?
[Jan24 12:45:12] <Oblomov> so you have to find that library
[Jan24 12:45:20] <Oblomov> no it's the name of the library
[Jan24 12:45:37] <Oblomov> most likely installd by some nvidia-compute-* package if this is ubuntu
[Jan24 12:45:33] <Liver_K> Ok it's not anywhere in PATH then, where would it be
[Jan24 12:52:11] <Liver_K> Like what "due paths" do you mean
[Jan24 12:53:03] <Liver_K> I have not looked through my libs before, I apologize if it's a dumb question, but where am I supposed to look to find such a library?
[Jan24 13:43:46] <Oblomov> Liver_K: dpkg -L nvidia-compute-525
[Jan24 14:46:54] <Oblomov> fwiw I'll be tagging the next clinfo release soon (ideally before the end of week, so if any of you guys wants to do some testing of the latest properties please do
[Jan24 14:53:28] <Oblomov> Athas: btw, if you want to test that double vs float thing, ARM on android doesn't support cl_khr_fp64
[Jan24 15:19:31] <Liver_K> Oblomov: Ok; I'll substitute my version after nvidia-compute-
[Jan24 15:35:19] <Oblomov> does anybody have access to IBM's opencl platform?
[Jan24 15:35:31] <Oblomov> I think it's the only thing I haven't seen a clinfo for
[Jan24 18:28:30] <Liver_K> Oblomov: I have libnvidia-compute-390 - libnvidia-opencl.so.1 is at /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.1
[Jan24 18:48:23] <Liver_K> Now what lol
[Jan24 19:39:46] <Liver_K> Wait it is a symlink
[Jan24 19:40:06] <Liver_K> Points to /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.1
[Jan24 19:41:19] <Liver_K> *Point to libnvidia-opencl.so.390.157
[Jan24 19:43:19] <Liver_K> What do I do with that linked file?
[Jan25 00:13:43] <Oblomov> can you ldd it?
[Jan25 10:35:20] <Liver_K> Oblomov: https://dpaste.com/AHM6UB8AQ
[Jan25 10:36:49] <Oblomov> looks like it should load, so I don't understand why it wouldn't work
[Jan25 10:36:57] <Oblomov> do you have the nvidia and nvidia-uvm modules loaded?
[Jan25 10:38:29] <Liver_K> I have nvi...

IRC conversation log (partially redacted for privacy of other users):

[Jan24 12:30:44]  <Liver_K> BTW Oblomov, I was just looking over our convo a while ago about ICDs and ICD loaders, and I finally checked /etc/OpenCL/vendors and saw the nvidia icd, but I still don't see it in clinfo
[Jan24 12:33:09]  <Liver_K> So you said that could mean that there is missing hardware, which is not true, or that there are missing libraries, which I am not sure how to look for
[Jan24 12:36:14]  <Oblomov> ldd $(cat /etc/OpenCL/vendors/nvidia.icd)
[Jan24 12:38:27]  <Liver_K> Yeah no such file or directory
[Jan24 12:42:54]  <Oblomov> so the file is not there?
[Jan24 12:42:40]  <Liver_K> Oblomov: libnvidia-opencl.so.1 is the output so it is there 
[Jan24 12:43:12]  <Oblomov> locate libnvidia-opencl.so.1
[Jan24 12:43:59]  <Liver_K> Wait lol no I just thought about it, Oblomov where is it supposed to be?
[Jan24 12:44:10]  <Liver_K> nvidia.icd is a regular file 
[Jan24 12:44:36]  <Oblomov> yes
[Jan24 12:44:51]  <Oblomov> the file contains a string which is the name or path of the library
[Jan24 12:45:07]  <Oblomov> but the actual library is searched for in the due paths
[Jan24 12:44:50]  <Liver_K> Oh so libnvidia-opencl.so.1 is the name of an apt package?
[Jan24 12:45:12]  <Oblomov> so you have to find that library
[Jan24 12:45:20]  <Oblomov> no it's the name of the library
[Jan24 12:45:37]  <Oblomov> most likely installd by some nvidia-compute-* package if this is ubuntu
[Jan24 12:45:33]  <Liver_K> Ok it's not anywhere in PATH then, where would it be
[Jan24 12:52:11]  <Liver_K> Like what "due paths" do you mean
[Jan24 12:53:03]  <Liver_K> I have not looked through my libs before, I apologize if it's a dumb question, but where am I supposed to look to find such a library?
[Jan24 13:43:46]  <Oblomov> Liver_K: dpkg -L nvidia-compute-525
[Jan24 14:46:54]  <Oblomov> fwiw I'll be tagging the next clinfo release soon (ideally before the end of week, so if any of you guys wants to do some testing of the latest properties please do
[Jan24 14:53:28]  <Oblomov> Athas: btw, if you want to test that double vs float thing, ARM on android doesn't support cl_khr_fp64
[Jan24 15:19:31]  <Liver_K> Oblomov: Ok; I'll substitute my version after nvidia-compute-
[Jan24 15:35:19]  <Oblomov> does anybody have access to IBM's opencl platform?
[Jan24 15:35:31]  <Oblomov> I think it's the only thing I haven't seen a clinfo for
[Jan24 18:28:30]  <Liver_K> Oblomov: I have libnvidia-compute-390 - libnvidia-opencl.so.1 is at /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.1
[Jan24 18:48:23]  <Liver_K> Now what lol
[Jan24 19:39:46]  <Liver_K> Wait it is a symlink
[Jan24 19:40:06]  <Liver_K> Points to /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.1
[Jan24 19:41:19]  <Liver_K> *Point to libnvidia-opencl.so.390.157
[Jan24 19:43:19]  <Liver_K> What do I do with that linked file?
[Jan25 00:13:43]  <Oblomov> can you ldd it?
[Jan25 10:35:20]  <Liver_K> Oblomov: https://dpaste.com/AHM6UB8AQ
[Jan25 10:36:49]  <Oblomov> looks like it should load, so I don't understand why it wouldn't work
[Jan25 10:36:57]  <Oblomov> do you have the nvidia and nvidia-uvm modules loaded?
[Jan25 10:38:29]  <Liver_K> I have nvidia-headless-390 and all its deps
[Jan25 10:40:05]  <Oblomov> lsmod | grep nvidia
[Jan25 10:41:39]  <Oblomov> Liver_K: ^
[Jan25 10:42:54]  <Liver_K> And everything there in /proc/modules is already loaded?
[Jan25 10:44:34]  <Oblomov> yes, but lsmod is typically used instead
[Jan25 10:44:43]  <Oblomov> do you have the nvidia and nvidia-uvm modules loaded?
[Jan25 10:45:17]  <Liver_K> Humm would have been helpful to know that a few times before heh. nvidia, nvidia_drm, and nvidia_modeset are loaded, but nothing with nvidia_uvm
[Jan25 10:47:47]  <Oblomov> modprobe nvidia-uvm
[Jan25 10:47:52]  <Liver_K> How do I know that is even an available module?
[Jan25 10:48:19]  <Liver_K> Where do I check all of them, loaded or not?
[Jan25 10:49:38]  <Oblomov> lsmod tells you if it's loaded
[Jan25 10:50:47]  <Liver_K> Oh is that the "used by" column?
[Jan25 10:51:23]  <Oblomov> no, it should be in the first column if it's loaded
[Jan25 10:51:03]  <Liver_K> It displays all of them and only loaded ones are used by something else?
[Jan25 10:51:05]  <Liver_K> Oh
[Jan25 10:51:38]  <Oblomov> the used by are the users
[Jan25 10:51:56]  <Oblomov> but if it says the nvidia module is used by nvidia_uvm, it means nvidia-uvm is loaded
[Jan25 10:51:58]  <Liver_K> Oblomov: I was asking where to check for ALL modules, not just loaded ones, but it looks like modprobe manpage says they are all in /lib/modules
[Jan25 10:59:05]  <Liver_K> Oblomov: None of them have uvm in the used by to clarify
[Jan25 10:59:17]  <Liver_K> I am just trying to find it somewhere before I try to load it
[Jan25 11:11:34]  <Oblomov> Liver_K: you need the uvm module to run compute loads
[Jan25 11:11:39]  <Oblomov> be it cuda or opencl
[Jan25 11:12:46]  <Oblomov> I actually have an enable-gpu module that modprobes nvidia and nvidia-uvm
[Jan25 11:14:59]  <Liver_K> Oblomov: I don't think there exists an nvidia-umv module
[Jan25 11:15:47]  <Liver_K> Modprobing it fails, but I expected that since I grepped for it everywhere under /lib/modules/$(uname -r) and absolutely nothing with the term "uvm"
[Jan25 11:16:00]  <Liver_K> So I don't have it for some reason
[Jan25 11:56:01]  <Oblomov> Liver_K: ok maybe 390 didn't have that yet
[Jan25 11:56:18]  <Oblomov> in which case I have no idea why it doesn't appear in your clinfo
[Jan25 11:56:24]  <Oblomov> even if cuda works
[Jan25 11:56:31]  <Oblomov> what gpu is this?
[Jan25 12:08:55]  <Liver_K> GeForce GTX 570
[Jan25 12:09:09]  <Liver_K> 390 is the newest version that works
[Jan25 12:09:19]  <Liver_K> Still being maintained AFAIK
[Jan25 12:09:45]  <Liver_K> Oblomov: CUDA does not work, I have not done anything to set that up
[Jan25 12:10:55]  <Oblomov> oh god fermi
[Jan25 12:11:12]  <Liver_K> Fermi?
[Jan25 12:12:56]  <Liver_K> Oblomov: What is this "nvidia-uvm" module supposed to do?
[Jan25 12:13:33]  <Oblomov> Liver_K: enable unified virtual memory
[Jan25 12:13:39]  <Oblomov> but it's a “recent” thing
[Jan25 12:13:43]  <Liver_K> Yeah lol no way is that gonna be on this card
[Jan25 12:13:48]  <Liver_K> But is it really required?
[Jan25 12:14:17]  <Oblomov> no
[Jan25 12:14:30]  <Oblomov> do you have the nvidia-smi tool installed?
[Jan25 12:14:09]  <Liver_K> Ok so must be something else
[Jan25 12:14:23]  <Liver_K> Oblomov: Not currently, I was going to do that soon
[Jan25 12:14:26]  <Liver_K> I'll do it now
[Jan25 12:24:51]  <Liver_K> https://dpaste.com/HJZFR4UC5
[Jan25 12:28:57]  <Oblomov> does clinfo work now?
[Jan25 12:30:14]  <Oblomov> wait, is this a headless setup?
[Jan25 12:29:59]  <Liver_K> No the GPU was visible with lshw before, clinfo still only reports my pocl platform
[Jan25 12:30:15]  <Liver_K> Oblomov: Here is the bug report though if you want it: http://0x0.st/oFZI.log.gz
[Jan25 12:30:51]  <Liver_K> Should I restart the machine though and run clinfo after that to try?
[Jan25 12:31:32]  <Oblomov> no
[Jan25 12:31:14]  <Liver_K> Oblomov: Oh sorry didn't see your last question, yes this is a headless setup
[Jan25 12:31:21]  <Liver_K> nvidia-headless-390 is installed
[Jan25 12:32:16]  <Oblomov> can you run strace -o /tmp/strace.log clinfo -l and pastebin /tmp/strace.log?
[Jan25 12:32:15]  <Liver_K> Sure give me a min
[Jan25 12:36:16]  <Liver_K> Oblomov: http://0x0.st/oFZD.log
[Jan25 12:36:25]  <Liver_K> Oh sorry you said pastebin that
[Jan25 12:38:01]  <Liver_K> Here https://dpaste.com/4SS4J8V54
[Jan25 12:40:53]  <Oblomov> so, it's looking for the nvidia-uvm stuff without finding it
[Jan25 12:41:04]  <Oblomov> so I'm guessing that's the reason why it won't work
[Jan25 12:41:19]  <Oblomov> it even tries to modprobe it
[Jan25 12:41:15]  <Liver_K> Lol
[Jan25 12:41:46]  <Liver_K> Does that mean the platform is actually there but clinfo doesn't like the old driver without uvm?
[Jan25 12:42:10]  <Liver_K> "there" as in installed and usable
[Jan25 12:42:47]  <Oblomov> it means the ubuntu driver is fsckd up for some reason because it's missing components
[Jan25 12:43:16]  <Oblomov> do you have apt-file installed?
[Jan25 12:43:05]  <Liver_K> Heh fsckd up thas funny
[Jan25 12:43:30]  <Liver_K> Oblomov: Yes
[Jan25 12:44:41]  <Oblomov> which version of ubuntu is this?
[Jan25 12:44:28]  <Liver_K> 22.04 LTS
[Jan25 12:44:33]  <Liver_K> *Ubuntu server
[Jan25 12:44:57]  <Oblomov> jammy?
[Jan25 12:44:40]  <Liver_K> yea
[Jan25 12:46:21]  <Liver_K> I mean should I try getting the actual Linux driver from nvidia and trying to install that?
[Jan25 12:46:48]  <Oblomov> no
[Jan25 12:46:33]  <Liver_K> Ok
[Jan25 12:48:00]  <Oblomov> did you install nvidia-dkms-390
[Jan25 12:47:48]  <Liver_K> It was a dep yes
[Jan25 12:49:58]  <Oblomov> and nvidia-kernel-common-390
[Jan25 12:50:23]  <Oblomov> so from what I'm seeing it should do th thing: https://packages.ubuntu.com/jammy/amd64/nvidia-kernel-common-390/filelist
[Jan25 12:50:22]  <Liver_K> Yes I have that installed as well
[Jan25 12:50:51]  <Oblomov> sudo you run sudo /sbin/create-uvm-dev-node ?
[Jan25 12:50:59]  <Oblomov> if it's there
Jan25 12:58:06]  <Liver_K> Yeah just let me look through that first
[Jan25 13:06:50]  <Liver_K> Alright Oblomov I looked for nvidia-uvm manually in the /proc/devices file like the script does, and there is none, so I ran it anyway to see what happened and still nothing happened that I can see (since the string isn't found there)
[Jan25 13:07:41]  <Oblomov> I give up
[Jan25 13:07:44]  <Oblomov> report the issue to ubuntu
[Jan25 13:07:49]  <Liver_K> So this is a driver issue or ubuntu's fault?
[Jan25 13:09:35]  <Liver_K> Cause if ubuntu has the wrong driver or something in their apt db then why can't I get the one from nvidia?
[Jan25 13:14:35]  <Liver_K> Except I don't think I could find a headless driver from nvidia when i looked
[Jan25 13:16:04]  <Oblomov> it's weird because judging from the files everything should be there, but the uvm component is not being loaded
[Jan25 13:16:15]  <Oblomov> and that's needed for compute
[Jan25 13:16:10]  <Liver_K> What about the loader profile? Could that be causing any problem? I don't know if it's something you set or it's automatically set for each loaded platform, but what I see for pocl is an opencl 3.0 profile
[Jan25 13:16:24]  <Liver_K> And I don't think this older nvidia driver would be ocl 3.0 
[Jan25 13:16:54]  <Oblomov> that's irrelevant
[Jan25 13:16:36]  <Liver_K> K
[Jan25 13:17:20]  <Oblomov> the nvidia-opencl platform doesn't load up because it can't find the uvm device
[Jan25 13:17:24]  <Oblomov> (by the strace)
[Jan25 13:17:45]  <Oblomov> so the question is, why isn't the uvm device being created and/or why don't you have nvidia-uvm
[Jan25 13:17:57]  <Oblomov> find /lib/moduls -name \*uvm\* finds nothing then?
[Jan25 13:17:44]  <Liver_K> Maybe I can look for a package relating to that
[Jan25 13:18:20]  <Oblomov> no it's built with the nvidia dkms kernel sources
[Jan25 13:18:51]  <Oblomov> can you tell me what this finds? find /lib/modules -name \*uvm*
[Jan25 13:18:54]  <Oblomov> ehm
[Jan25 13:18:55]  <Oblomov> can you tell me what this finds? find /lib/modules -name \*uvm\*
[Jan25 13:23:39]  <Liver_K> No that finds nothing
[Jan25 13:25:17]  <Liver_K> Also la -R /lib/modules/5.15.0-58-generic/ | grep uvm outputs nothing
[Jan25 13:25:46]  <Liver_K> And double checked with something I knew was there to make sure it is working
[Jan25 13:29:06]  <Oblomov> what about in the dkms tree?
[Jan25 13:28:53]  <Liver_K> Hmm where's that again
[Jan25 13:29:18]  <Oblomov> something like /var/lib/dkms
[Jan25 13:31:37]  <Liver_K> Yeah grepping recursively in there also yields nil
[Jan25 13:31:46]  <Liver_K> Checked with a known true term again
[Jan25 13:33:08]  <Oblomov> you definitely have some issue. the uvm part should be part of the nvidia 390 kernel module
[Jan25 13:33:09]  <Liver_K> Gah I am mad at ubuntu now
[Jan25 13:33:12]  <Liver_K> And nvidia
[Jan25 14:00:24]  <Liver_K> Oblomov: You are probably more qualified than I to report that bug, if you are willing to :)
[Jan25 14:01:22]  <Oblomov> I don't report bugs for situations I don't have full access to
[Jan25 14:52:03]  <Liver_K> Alright then so I can report it better, can you tell me a bit more about what these components for the kernel module are?
[Jan25 14:54:12]  <Liver_K> Gotta love vague question heh
[Jan25 14:59:00]  <Oblomov> Liver_K: there should be either an nvidia-uvm module produced with the other, or something that produces a /dev/nvidia-uvm device
[Jan25 14:59:06]  <Oblomov> and this is not happening
[Jan25 14:59:47]  <Liver_K> Alright I'll slap it on launchpad sometime then
[Jan25 15:00:07]  <Liver_K> This is extremely annoying >| I will have to use cuda 
[Jan25 15:02:05]  <Liver_K> Oh so nvidia-uvm is supposed to be its own kernel module
[Jan25 19:14:54]  <Liver_K> Oblomov: And you're sure the nvidia provided driver wouldn't be any different?
[Jan26 01:45:55]  <Oblomov> Liver_K: you can try it if you want

Ubuntu
nvidia-graphics-drivers-390 package

nvidia-uvm module does not exist, making CUDA and OpenCL unusable with the 390 driver

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntunvidia-graphics-drivers-390 package