root escalation via /dev/nvidia0

[Bryce Harrington]

It was raised to me just now that there is a security issue with /dev/nvidia0 where an unprivileged account can access kernel memory and gain root access. An example exploit is attached.

[Bryce Harrington (bryce)]

The previous code apparently is written for 64-bit.

Attached is my (probably wrong) attempt at making it work on 32-bit.

Output:

bryce@porlock:~$ gcc ./nvidia_exploit.c -o nvidia_exploit ; ./nvidia_exploit
Card type: 0x92200a2, chip 92
627
c34500b4 488
Linux kernel found at 70000
bryce@porlock:~$ gcc ./nvidia_exploit.c -o nvidia_exploit ; ./nvidia_exploit
Card type: 0x92200a2, chip 92
627
c34500b4 488
Linux kernel found at 70000

I've no idea whether the c34500b4 488 numbers correspond to something pertaining to the kernel, but those were the only numbers that got printed out.

[Alberto Milone (albertomilone)]

@Daniel, can you investigate this issue, please?

[Daniel Dadap (ddadap)]

Thank you for reporting this vulnerability. We are investigating ways to close it. In the meantime, as additional protection against this vulnerability, there are mechanisms available to override the default security settings of the device nodes: please see the Q/A item "How and when are the NVIDIA device files created?" in Chapter 7, "Frequently Asked Questions" of the driver README for details on how to set custom owner, group, and permissions on these files:

ftp://download.nvidia.com/XFree86/Linux-x86/295.33/README/faq.html#devicenodes

[Chris Halse Rogers (raof)]

Thanks for that pointer. To what extent does the user need access to the device nodes? I would expect that removing user-writability from the nodes would cause direct rendering via libGL to fail; is this indeed the case?

If we can safely restrict these devices to root, then we're done.

[Daniel Dadap (ddadap)]

Yes, direct rendering requires user access to the device nodes. (Indirect rendering does not.) Restricting device node write access to root would still allow X and indirect rendering to work, provided that X runs as root.

Users who wish to run CUDA/OpenCL applications would also need write access to the device nodes.

We're investigating strategies to block this exploit on the driver end. We're having another internal meeting about this tonight; I'll make it one of the agenda items to discuss other potential workarounds that don't require a driver change.

Since this is an existing vulnerability, how does it impact the release schedule for Precise, if at all, given that previous releases are equally affected? This is obviously a high priority issue for NVIDIA; for our tracking purposes, I just wanted to know how urgent this is on your end.

[Daniel Dadap (ddadap)]

We'll be pushing out an updated driver soon that will address this issue. We're still working out the schedule; hopefully by the Canonical/NVIDIA meeting tomorrow morning we'll be able to share the schedule with you.

[Daniel Dadap (ddadap)]

Examination of the exploit code helped us to identify the driver vulnerability that it attacks, and we're testing some driver changes right now and to ensure that they'll be able to block this attack; however, we haven't yet actually gotten the exploit to successfully find the payload, when compiled exactly as written.

We are able to find the payload if we remove the hardcoded offsets to pmem and instead loop over the mapped memory, but for completeness, we want to be able to reproduce with the code exactly it was provided: for reference, what kind of system was this exploit originally developed for, running which kernel version, etc?

(An nvidia-bug-report.log file will capture other details we may be interested in, so if one can be provided, that would be helpful.)

[Daniel Dadap (ddadap)]

Status update: a fix for this issue in the 295 driver series is imminent. We hope to be posting updated drivers and a public announcement before the end of the week.

In addition to updated drivers, we will be providing a patch to the kernel module's Linux kernel interface layer, which is open source, so that older driver versions can be secured as well. I recommend that the packages for the older driver versions which are supplied for previous Ubuntu releases which are still receiving security updates be patched.

[Marc Deslauriers (mdeslaur)]

Thanks for the update.

Would it be possible to get the patch in advance and a coordinated release date so we can prepare updates and have them ready once you publically announce the vulnerability?

[Bryce Harrington (bryce)]

On Fri, Mar 30, 2012 at 05:01:56PM -0000, Daniel Dadap wrote:
> Examination of the exploit code helped us to identify the driver
> vulnerability that it attacks, and we're testing some driver changes
> right now and to ensure that they'll be able to block this attack;
> however, we haven't yet actually gotten the exploit to successfully find
> the payload, when compiled exactly as written.
>
> We are able to find the payload if we remove the hardcoded offsets to
> pmem and instead loop over the mapped memory, but for completeness, we
> want to be able to reproduce with the code exactly it was provided: for
> reference, what kind of system was this exploit originally developed
> for, running which kernel version, etc?
>
> (An nvidia-bug-report.log file will capture other details we may be
> interested in, so if one can be provided, that would be helpful.)

I think the individual wishes to remain anonymous, however he passed
along some additional comments about this:

"""
oh it seems to be a bit picky, sometimes the kernel won't align
right. The real problem already happens when you can succesfully map it.
I just tried to identify the zero page of the linux kernel.
Compare with a copy from /dev/mem. That's all my code is doing, showing
that /dev/nvidia0 was a world read/writable version of /dev/mem :-)
"""

> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/959842
>
> Title:
> root escalation via /dev/nvidia0
>
> Status in “nvidia-graphics-drivers” package in Ubuntu:
> Triaged
>
> Bug description:
> It was raised to me just now that there is a security issue with
> /dev/nvidia0 where an unprivileged account can access kernel memory
> and gain root access. An example exploit is attached.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers/+bug/959842/+subscriptions

[Daniel Dadap (ddadap)]

We're working on providing the updated driver and patchset to Alberto via the partners site. It should be posted momentarily.

We are doing some final testing and will likely be posting the driver on Monday. Please do not release packages with the fixed drivers until after our public announcement is made. I will update this bug to indicate when that has happened.

[Daniel Dadap (ddadap)]

BTW: applying the patch will break the CUDA debugger. This is a known issue that will be noted in the public announcement about the vulnerability. The updated driver (295.40) includes additional changes to restore compatibility with the CUDA driver, but applying the patch on its own will break usage.

I'll let you decide which tradeoff you want to make (better security vs. convenience for a relatively small set of users) - users who must use the CUDA debugger will have to choose whether to use an unpatched driver, or an updated one, since a patched but not updated driver will break the CUDA debugger.

[Marc Deslauriers (mdeslaur)]

Would it be possible to get the announcement delayed for a few days while we prepare updated packages with the patch for all our releases and put them through proper QA?

Having the patch dropped on us friday afternoon of easter weekend for publishing on monday is kind of tight, especially for a root escalation issue.

[Daniel Dadap (ddadap)]

I'll make a request to delay the posting, but I don't know how much time I can get you.

Since it might be a while before the partner portal posting is up, I'm attaching the patches to this bug. There are three separate versions of the patch, for different driver versions:

a) nvidia-blacklist-register-mapping-290-295.diff for drivers from releases 290 and 295.
b) nvidia-blacklist-register-mapping-256-285.diff for drivers from releases 256 through 285, inclusive.
c) nvidia-blacklist-register-mapping-195.diff for drivers from releases 195 and earlier. (There were no releases between 195 and 256.)

The patch is intended for use as an argument to the --apply-patch option to the NVIDIA installer script. You can also just use `patch` directly; there are only two files that get patched.

(It looks like I can only attach one file at a time: I'll post separate comments to attach the other two files.)

[Marc Deslauriers (mdeslaur)]

Do you need a CVE number for this issue?
Have the other linux vendors been notified? (Do any other vendors ship the nvidia drivers?)

[Daniel Dadap (ddadap)]

We haven't gotten a CVE ourselves. Let me circle back with management to see if we were planning to get ourselves a CVE or if we'd get one from Canonical.

Novell and Red Hat have been notified. Red Hat does not package the NVIDIA drivers. Novell does.

There are other vendors who package the NVIDIA drivers as well, but we don't have a direct relationship with them, as we do with Canonical, Novell, and Red Hat.

[Daniel Dadap (ddadap)]

Management agrees that getting a CVE from Canonical should be fine. How quickly can we get one?

I'll have limited access to e-mail today and tomorrow, so subscribing additional NVIDIA contacts to the bug.

[Marc Deslauriers (mdeslaur)]

Pretty quickly :)

This issue is CVE-2012-0946. Please communicate the assigned CVE number to the other Linux vendors.

Thanks.

[Marc Deslauriers (mdeslaur)]

Here is a slightly modified version of the patch necessary to get it working with release 96 of the driver, which we still have in Ubuntu for compatibility reasons.

[Marc Deslauriers (mdeslaur)]

We have updates ready and tested for all releases in Lucid, Natty and Oneiric.

Please let us know once this issue is public so we can push out updated packages with the security patch.

Thanks!

[Daniel Dadap (ddadap)]

The patch isn't necessary on release 96, unless people are using a release 96 driver with G80 or newer hardware. The vulnerability is only exploitable on G80 or newer.

Unfortunately, we've discovered some issues with the patch as provided during our extended testing: on some GPUs, blacklisting the mapping ranges that are currently in the patch breaks OpenGL. We're looking into the best way to ensure that exploitable mappings are forbidden while maintaining compatibility across our full line of supported GPUs.

[Daniel Dadap (ddadap)]

Hardy posted a comment by e-mail to Alberto Milone and Henry Hall, and I just realized that this bug wasn't updated. After further analysis of the issues we observed, we have decided to go ahead with the release: the problems we encountered during our extended testing were limited to a rarely used fallback path which is already almost unusably slow, and only on certain GPUs. We are looking into ways to resolve this issue, but in the meantime, saw no need to further delay the release of this security fix to address an issue that is unlikely to be encountered by users in practice.

We will be posting the driver update and patches shortly. I'll update the bug once the update is posted.

Also, I fail to see what is different about the release 96 version of the patch above, besides line numbers: we expect the line numbers to have shifted a bit; in my testing, the `patch` tool was always able to resolve this. Once again, though, we don't recommend applying the patch to the 96 series of drivers, as those drivers target older GPUs which aren't affected by this security issue.

[Marc Deslauriers (mdeslaur)]

Thanks for the update Daniel.

Does 96 refuse to work with G80 or newer? If that's not the case, we may have some users running 96 with an affected GPU simply because it was the first one they tried in our driver installation tool.

If 96 doesn't work with G80 and newer, then we will ignore it.

The 96 patch simply does "return -EINVAL;" instead of "status = -EINVAL;goto done;".

[Daniel Dadap (ddadap)]

Ah, thanks: I hadn't looked at the patch closely enough to notice that.

the 96 series of drivers isn't expressly prohibited from running on G80, so it's possible that a user could use it with G80. The products that are supported by the 96 series of drivers (NV2x) are not affected by the vulnerability, and we haven't done any internal testing to see how the patch might affect the proper operation of these products.

The drivers are now posted to the NVIDIA download website and FTP site.

[Marc Deslauriers (mdeslaur)]

OK, our driver install tool actually filters by PCI ID, so nobody running recent cards should be running the 96 driver. As you recommend, we will skip the 96 driver. Thanks!

[Daniel Dadap (ddadap)]

The 173 drivers are also intended for older GPUs (NV3x generation), but I do know of several cases where users elect to use the 173 drivers on G80 and newer, to avoid issues in newer drivers. The combination of patched 173 driver + NV3x is also untested.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers - 280.13-0ubuntu6.1

---------------
nvidia-graphics-drivers (280.13-0ubuntu6.1) oneiric-security; urgency=low

  * SECURITY UPDATE: privilege escalation via kernel memory access
    (LP: #959842)
    - debian/dkms/patches/blacklist-register-mapping.patch: blacklist
      certain offsets in nv.{c,h}.
    - debian/dkms.conf{.in}: added new patch.
    - Thanks to NVIDIA for providing the patch.
    - CVE number pending
 -- Marc Deslauriers <email address hidden> Sat, 07 Apr 2012 17:00:50 -0400

[Daniel Dadap (ddadap)]

I noticed the Launchpad CVE tracker page for CVE-2012-0946 still displays the "**RESERVED**" placeholder. Is any action required from NVIDIA (e.g. additional details) to get the CVE tracker page updated?

[Marc Deslauriers (mdeslaur)]

No, the description comes from the MITRE database. They will update the description once they notice that updates have been issued. It usually takes a few days.

[Daniel Dadap (ddadap)]

Okay, will they need any information from NVIDIA to populate the database entry, or will they do their own research?

[Marc Deslauriers (mdeslaur)]

They write the description themselves based on the security advisories that get published, so presumably they will include information from here:

http://nvidia.custhelp.com/app/answers/detail/a_id/3109

[Jamie Strandboge (jdstrand)]

The stable releases were fixed in http://www.ubuntu.com/usn/usn-1420-1. Alberto is working on 12.04 now.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Patch for 195 and earlier" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers - 295.40-0ubuntu1

---------------
nvidia-graphics-drivers (295.40-0ubuntu1) precise; urgency=low

  * New upstream release:
    - Added support for the following GPUs:
      o GeForce GT 635M
      o GeForce 610M
    - SECURITY UPDATE: privilege escalation via kernel
      memory access (LP: #959842).
    - Fixed a bug that caused DisplayPort devices to
      occasionally fail to turn back on after the system
      is resumed from suspend.
    - Added a ModeValidation X configuration option token,
      "AllowNon3DVisionModes", to allow modes that are not
      optimized for NVIDIA 3D Vision to be validated on 3D
      Vision monitors.
 -- Alberto Milone <email address hidden> Thu, 12 Apr 2012 15:08:57 +0200

[Bryce Harrington (bryce)]

There's still a few tasks open for this bug (against precise). Do these still need some attention?

[Jamie Strandboge (jdstrand)]

It seems like 12.04 is fixed, but it was assigned to Alberto after the fix was published. Is 12.04 still affected?

[Bryce Harrington (bryce)]

It looks like only the -173 drivers still have open tasks, however the -173 in precise (173.14.35-0ubuntu0.2) does include this security fix.

It looks like the bug # and CVE was not included in the changelog, so presumably that's why the tasks didn't close. I'm going to go ahead and assume this is all resolved and close out the remaining tasks.