nbd mounter leaks nbd devices

Bug #1088083 reported by Michael Still
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Michael Still

Bug Description

If the nbd device is in used by a non-nova-compute user, it is leaked forever.

Tags: ops security
Michael Still (mikal)
information type: Public → Private Security
Revision history for this message
Michael Still (mikal) wrote :

Ok, I can see at least one race condition and a leak in the current version of the nbd code (nova/virt/disk/mount/nbd.py). They are:

 - the leak is: we check to see if there is a pid associated with a nbd device file. If there is we remove the device from the list of possible devices that we can use. We never re-add it. This is the bug that this bug was originally intended to track and was found the other day by Robert Collins and I. I am working on a patch for this problem now. The attack vector here is that a _local_ user could consume all the nbd devices for a short time which coincides with nova attempting to use nbd, and then nbd is broken for nova until it is restarted.

 - the race is: we check that a pid doesn't exist for a nbd device file. We then execute qemu using that device file, and then declare success when there is a pid associated with the device file. However -- we never check that the pid associated is the pid we created. This means someone else could be providing something which we think is qemu but is something else. I guess this could be a disk image or something like that, but I'm not too sure what attack would be meaningful here.

I find none of these attacks individually too concerning, but I want to see what people think before I send of a series of reviews to resolve them.

Revision history for this message
Robert Collins (lifeless) wrote :

fix it all

Revision history for this message
Michael Still (mikal) wrote :

Sure, but can we think of an exploit severe enough to warrant going through the CVE process, or should we just fix it like any other bug?

Revision history for this message
Thierry Carrez (ttx) wrote :

Once public, forever public.

information type: Private Security → Public Security
Revision history for this message
Thierry Carrez (ttx) wrote :

The leak and race need to be fixed, no question about that. That said, the attack vector is a bit convoluted, so I would not consider this a vulnerability that needs a security advisory issued.

Revision history for this message
Thierry Carrez (ttx) wrote :

In all cases feel free to issue patches and public reviews. Bugs filed in public are considered public knowledge anyway.

Changed in nova:
importance: Critical → High
Revision history for this message
Robert Collins (lifeless) wrote :

It looks local only to me. So no, IMNSHO.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/17980

Changed in nova:
status: Triaged → In Progress
Thierry Carrez (ttx)
tags: added: security
information type: Public Security → Public
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/17980
Committed: http://github.com/openstack/nova/commit/e4377fdb0ef2087a36d2b1fbee96543f735040de
Submitter: Jenkins
Branch: master

commit e4377fdb0ef2087a36d2b1fbee96543f735040de
Author: Michael Still <email address hidden>
Date: Wed Dec 12 17:13:53 2012 +1100

    Stop nbd leaks, remove pid race.

    With the previous implementation, if a nbd device was found to be in
    use by something other than nova it was removed from the class scoped
    list of devices, but never re-added. This meant we "leaked" devices
    away over time if we were competing with other nbd users on the
    machine.

    Instead of tracking our use at all, we should rely on the presence of
    a user pid in /sys.

    Resolves bug 1088083.

    Change-Id: If777e270a0dda12034ea7ef1bc7fd688cadde8a9

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-2 → 2013.1
Revision history for this message
Kurt Seifried (kseifried) wrote :

This sounds like a denial of service attack:

The attack vector here is that a _local_ user could consume all the nbd devices for a short time which coincides with nova attempting to use nbd, and then nbd is broken for nova until it is restarted.

Can someone explain to me why this is not a security issue (sounds like one, no?).

Revision history for this message
Michael Still (mikal) wrote :

@Kurt -- this was discussed earlier in the bug. What we are actually discussing above is if we think this is severe enough to issue a CVE for, which we decided it wasn't.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.