[OSSA 2016-012] qemu-img calls need to be restricted by ulimit (CVE-2015-5162)

[Tristan Cacqueray]

Reported via private E-mail from Richard W.M. Jones.

Turns out qemu image parser is not hardened against malicious input and can be abused to allocated an arbitrary amount of memory and/or dump a lot of information when used with "--output=json".

The solution seems to be: limit qemu-img ressource using ulimit.

Example of abuse:

-- afl1.img --

$ /usr/bin/time qemu-img info afl1.img
image: afl1.img
[...]
0.13user 0.19system 0:00.36elapsed 92%CPU (0avgtext+0avgdata 642416maxresident)k
0inputs+0outputs (0major+156927minor)pagefaults 0swaps

The original image is 516 bytes, but it causes qemu-img to allocate 640 MB.

-- afl2.img --

$ qemu-img info --output=json afl2.img | wc -l
589843

This is a 200K image which causes qemu-img info to output half a
million lines of JSON (14 MB of JSON).

Glance runs the --output=json variant of the command.

-- afl3.img --

$ /usr/bin/time qemu-img info afl3.img
image: afl3.img
[...]
0.09user 0.35system 0:00.47elapsed 94%CPU (0avgtext+0avgdata 1262388maxresident)k
0inputs+0outputs (0major+311994minor)pagefaults 0swaps

qemu-img allocates 1.3 GB (actually, a bit more if you play with
ulimit -v). It appears that you could change it to allocate
arbitrarily large amounts of RAM.

[Daniel Berrange (berrange)]

FYI, I tested a recentish git master to see if I can trigger it and was succesful:

What I did was

 $ cd /usr/bin
 $ mv qemu-img qemu-img.real

 $ cat > qemu-img <<EOF
  #!/bin/sh

  echo "$@" >> /tmp/usage.log
  /usr/bin/time /usr/bin/qemu-img.real "$@" 2>> /tmp/usage.log
  echo >> /tmp/usage.log
  EOF

 $ chmod +x qemu-img

Then

 $ glance image-create --name afl3 --disk-format vmdk --container-format bare --file /home/berrange/afl3.img --is-public true

Glance did not appear to run qemu-img info at this stage, but when I then boot the image iwth Nova:

 $ nova boot --image afl3 --flavor m1.tiny afl3

Nova correctly refuses to boot the guest, because the disk image is larger
than the maximum it permits:

2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] return f(*args, **kwargs)
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] File
+"/home/berrange/src/cloud/nova/nova/virt/libvirt/imagebackend.py", line 221, in fetch_func_sync
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] fetch_func(target=target, *args, **kwargs)
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] File
+"/home/berrange/src/cloud/nova/nova/virt/libvirt/utils.py", line 501, in fetch_image
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] max_size=max_size)
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] File
+"/home/berrange/src/cloud/nova/nova/virt/images.py", line 119, in fetch_to_raw
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] raise exception.FlavorDiskTooSmall()
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] FlavorDiskTooSmall: Flavor's disk is too small for
+requested image.
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b]

However in checking that maximum limit, it has indeed run qemu-img info
and suffered the memory usage DOS.

 $ cat /tmp/usage.log

 info /home/berrange/src/cloud/data/nova/instances/_base/30854c86815b92c21c8af45cf8ff5757c04046aa.part
 0.08user 0.94system 0:01.24elapsed 82%CPU (0avgtext+0avgdata 1260680maxresident)k
 0inputs+0outputs (0major+311876minor)pagefaults 0swaps

So someone needs to see just what code paths in glance might trigger the qemu-img info call. I also wonder if there is any risk to cinder, because it too has various invokations of qemu-img

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Projects security liaison and original reporter have been subscribe.

While being an issue in a dependency (qemu), it's not clear whenever OpenStack is really missing restriction on the qemu-img process...

However, depending on what a malicious user can effectively do with this issue, we may want to consider this OSSA worthy. Thought ?

[Nikhil Komawar (nikhil-komawar)]

Excellent catch! And thanks for bringing this up, Richard. I was going to do some testing myself this week and file a corresponding bug.

This is a valid concern and something that we realized after the code was merged. One more point to consider here is that we need to run qemu-img in a sandbox vs. directly as a "glance" user on the the API node as possible in certain configurations. A approach for fixing this could be, using a chroot environment.

[Richard Jones (rjones-redhat)]

Here's an image which causes 4 GB to be allocated.

If you change a single byte in this image, then you can get qemu-img to crash with an allocation error:

00000000 4b 44 4d 56 01 00 00 00 03 00 00 00 01 00 ff ff |KDMV............|
00000010 ff ff 00 00 01 04 00 00 00 00 00 00 01 00 00 00 |................|
                      ^
                      +--- this byte to 0

[Nikhil Komawar (nikhil-komawar)]

qemu-img is run only on Glance API/worker nodes when user makes a tasks import call while the operator wants it converted into a specific format, currently.

Ref:-
http://specs.openstack.org/openstack/glance-specs/specs/kilo/conversion-of-images.html
https://review.openstack.org/#/c/159129/

[Daniel Berrange (berrange)]

> However, depending on what a malicious user can effectively do with this issue, we may want to consider this OSSA worthy. Thought ?

From Nova's POV I consider it to be OSSA worthy. Nova uses flavours to limit the amount of RAM that a user may consume. So if using the m1.tiny flavour, a guest is supposed to only have 500 MB RAM usage. The scheduler will consider this when deciding if a host has sufficient free RAM to run the guest. Now regardless of what the flavour limit is set to, the unprivileged user can consume as much as 4 GB of RAM on the host by uploading a malicious disk image. This is enough to cause OOM on the compute host and so affect running of other tenants VMs in a negative way.

[Tristan Cacqueray (tristan-cacqueray)]

Any progess on patch for this bug ?

[Richard Jones (rjones-redhat)]

Although this bug is marked as secret, I have published elsewhere some information about this (it was anyway an open secret that I was fuzzing qemu-img), and it is also fixed in libguestfs.

https://bugs.launchpad.net/qemu/+bug/1462944
https://bugs.launchpad.net/qemu/+bug/1462949
https://github.com/libguestfs/libguestfs/commit/9db0f807a917ec00d24297d5bf63c19065f930cf

[Nikhil Komawar (nikhil-komawar)]

Thanks, I will have a Glance patch this week.

[Daniel Berrange (berrange)]

This is the simplest patch I could come up with for limiting qemu-img info commands. There are various other qemu-img commands run, eg to create snapshots and the like, but I believe checking the qemu-img info command is sufficient as it is used to validate images after obtaining them from glance.

[Daniel Berrange (berrange)]

Added cinder, since the file cinder/image/image_utils.py invokes qemu-img info without protection too

[Tristan Cacqueray (tristan-cacqueray)]

Thanks Daniel, cinder-coresec is now subscribed. Can Glance and Cinder use the same prlimit fix ?

[Daniel Berrange (berrange)]

And now I've fixed the patch to make the unit tests pass again. Opps :-)

[Daniel Berrange (berrange)]

Yes, from looking at their code, the change to cinder & glance could be essentially identical.

[Richard Jones (rjones-redhat)]

> "util-limit package in Linux distros."

That should say "util-linux".

[Tristan Cacqueray (tristan-cacqueray)]

Here is a proposed patch for cinder

[Tristan Cacqueray (tristan-cacqueray)]

Hum, prlimit is not readily available on stable Ubuntu and Debian...

[Daniel Berrange (berrange)]

Sigh, "only" been part of util-linux since early 2012

http://karelzak.blogspot.co.uk/2012/01/prlimit1.html
https://www.kernel.org/pub/linux/utils/util-linux/v2.21/v2.21-ReleaseNotes

[Daniel Berrange (berrange)]

The alternative approach is to modify oslo.concurrency processutils.execute method to allow limits to be passed in explicitly, and then use the pre-exec function to set them between fork + exec. This is actually the more desirable architectural approach, but I didn't go that way, since while it is fine for master, it would not be something that's easily backported to stable branches due to the need to update the min required oslo.concurrency version dependancy.

[Michael Still (mikal)]

I actually much prefer the ulimit argument to processutils.execute() plan. I'm going to go to the oslo PTL and ask for an opinion of backporting that.

[Tristan Cacqueray (tristan-cacqueray)]

Since this bugs requires change in 4 different projects, and the impacts is moderate, I propose we open this bug in a few days for a faster solution. Is there any objections ?

[Davanum Srinivas (DIMS) (dims-v)]

@mikal,

yes, i'd support a ulimit argument to processutils.execute and willing to help backport it to stable/kilo (and release if needed)

[Daniel Berrange (berrange)]

This is a *completely untested* patch to illustrate adding support for process resource limits to oslo.concurrency's execute() call. If people agree with this approach, I'll just submit it to gerrit against oslo, as adding this feature to olso is probably useful regardless of this OSSA

[Davanum Srinivas (DIMS) (dims-v)]

Hi, we've released a oslo.concurrency 1.8.1 (stable/kilo) with support to call a user specified preexec_fn to support this bug/fix. yes, same is in master as well.

80bfd1a Allow preexec_fn method for processutils.execute

thanks,
dims

[Grant Murphy (gmurphy)]

Draft impact description:

Title: Malicious input to qemu-img may result in resource exhaustion
Reporter: Richard W.M. Jones
Product: Nova, Cinder, Glance
Affects:
    Nova: 2012.1.0 versions through 2015.1.0,
    Cinder: 2013.1.0 versions through 2015.1.0,
    Glance: 2015.1.0

Description:
Richard W.M. Jones of Red Hat reported a vulnerability that affects several
OpenStack projects including Nova, Glance, and Cinder. By providing a
maliciously crafted disk image an attacker can consume considerable amounts
of RAM and CPU time resulting in a denial of service via resource exhaustion.
Any project which makes calls to qemu-img without appropriate ulimit restrictions
in place is affected by this flaw.

[Jeremy Stanley (fungi)]

I think this would more properly be (assuming the fixes won't make it in time for 2015.1.1)...

    Nova: versions through 2014.2.3, 2015.1 versions through 2015.1.1
    Cinder: versions through 2014.2.3, 2015.1 versions through 2015.1.1
    Glance: 2015.1 versions through 2015.1.1

[Grant Murphy (gmurphy)]

CVE requested.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/209627

[Tristan Cacqueray (tristan-cacqueray)]

Is there a review in progress for Glance ?

[Nikhil Komawar (nikhil-komawar)]

Yep.

[Daniel Berrange (berrange)]

I've attached another patch which this time uses the new preexec_fn feature of oslo.concurrency 2.3.0.

This patch is *UNTESTED* beyond running the unit tests, so don't assume this actually fixes the problem without testing it in real world.

[Tristan Cacqueray (tristan-cacqueray)]

Daniel, could you please have a look at https://review.openstack.org/#/c/209627/ ?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/209627
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=da217205f53f9a38a573fb151898fbbeae41021d
Submitter: Jenkins
Branch: master

commit da217205f53f9a38a573fb151898fbbeae41021d
Author: Tristan Cacqueray <email address hidden>
Date: Wed Aug 5 17:17:04 2015 +0000

    virt: Use preexec_fn to ulimit qemu-img info call

    This uses the preexec_fn oslo.concurrency execute parameter to set user limit
    before calling qemu-img info. The process is limitted to 2 seconds of cpu
    and 1GB of virtual memory

    Change-Id: Ib47f1116c94c8f76d2a4a525af24439a4aa15854
    Closes-Bug: #1449062

[Daniel Berrange (berrange)]

FYI, we are reverting the patch for this bug because it caused a spike in gate failures

https://bugs.launchpad.net/nova/+bug/1506012
https://review.openstack.org/#/c/234696/

[Daniel Berrange (berrange)]

The way https://review.openstack.org/#/c/209627/ applied the limits to qemu-img was by setting rlimits in between fork() and execve(). The problem with this is that before we reach exec() we have a full copy-on-write mapping of the nova python process. So if Nova memory usage is too high, this could tickle the limit we're setting for qemu-img.

The only way to avoid this is to *not* set limits until after we have exec'd. This would require using an external command like my original prlimit patch.

Since not all distros have prlimit, we could create a simple nova-prlimit python script todo this. We would use native prlimit if it was present in the distro (as its fast) and fallacck to nova-prlimit if missing.

It is a shame oslo.concurrency went for the preexec_fn approach, rather than my suggestion of explicitly representing resource limits in the API, as that would have allowed us to hide this compat code in oslo.concurrency, instead of dealing with it in nova, glance and cinder :-(

[Tristan Cacqueray (tristan-cacqueray)]

Can this prlimit python script could be provided by oslo or a requirement since it could be used by Cinder and Glance as well?

[Tristan Cacqueray (tristan-cacqueray)]

Note that such prlimit utility is being worked on in oslo.concurrency here: https://review.openstack.org/#/c/243829/

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/nova 13.0.0.0b1 development milestone.

[Tristan Cacqueray (tristan-cacqueray)]

The proposed change did not effectively fixed that issue.

[Daniel Berrange (berrange)]

The oslo.concurrency feature we require merged in Feb 1st this year. So we can now retry fixing nova once again, and also backport it to Mitaka stable series once merged in master

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/307663

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/326327

[Nikhil Komawar (nikhil-komawar)]

For Glance: The task API is now admin only by default since Mitaka. So, the risk of this issue to glance tasks API is significantly low. I've degraded the importance accordingly.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/307663
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=068d851561addfefb2b812d91dc2011077cb6e1d
Submitter: Jenkins
Branch: master

commit 068d851561addfefb2b812d91dc2011077cb6e1d
Author: Daniel P. Berrange <email address hidden>
Date: Mon Apr 18 16:32:19 2016 +0000

    virt: set address space & CPU time limits when running qemu-img

    This uses the new 'prlimit' parameter for oslo.concurrency execute
    method, to set an address space limit of 1GB and CPU time limit
    of 2 seconds, when running qemu-img.

    This is a re-implementation of the previously reverted commit

    commit da217205f53f9a38a573fb151898fbbeae41021d
    Author: Tristan Cacqueray <email address hidden>
    Date: Wed Aug 5 17:17:04 2015 +0000

        virt: Use preexec_fn to ulimit qemu-img info call

    Closes-Bug: #1449062
    Change-Id: I135b5242af1bfdcb0ea09a6fcda21fc03a6fbe7d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/327624

[Tristan Cacqueray (tristan-cacqueray)]

Newton is now fixed, but mitaka needs an oslo.concurrency requirements bump to 3.8.0.

For liberty, it is still using 2.6.x version and we need to first merge: https://review.openstack.org/#/c/327630/, then release a new version of olso.concurrency in order to bump requirements for stable/liberty.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/326327
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c8ec9ebf379c61d73c5671a75dd2a4e4ae1403fb
Submitter: Jenkins
Branch: stable/mitaka

commit c8ec9ebf379c61d73c5671a75dd2a4e4ae1403fb
Author: Daniel P. Berrange <email address hidden>
Date: Mon Apr 18 16:32:19 2016 +0000

    virt: set address space & CPU time limits when running qemu-img

    This uses the new 'prlimit' parameter for oslo.concurrency execute
    method, to set an address space limit of 1GB and CPU time limit
    of 2 seconds, when running qemu-img.

    This is a re-implementation of the previously reverted commit

    commit da217205f53f9a38a573fb151898fbbeae41021d
    Author: Tristan Cacqueray <email address hidden>
    Date: Wed Aug 5 17:17:04 2015 +0000

        virt: Use preexec_fn to ulimit qemu-img info call

    NOTE(mriedem): The backport depends on raising the minimum
    required oslo.concurrency>=3.7.1 which contains the prlimits
    fix for this change, the code fails without it.

    Depends-on: I157a6fc88836da73814ae46c8991ddefc5066c96
    Closes-Bug: #1449062
    Change-Id: I135b5242af1bfdcb0ea09a6fcda21fc03a6fbe7d
    (cherry picked from commit 068d851561addfefb2b812d91dc2011077cb6e1d)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 13.1.0 release.

[Chris J Arges (arges)]

Hello Tristan, or anyone else affected,

Accepted python-oslo.concurrency into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-oslo.concurrency/3.7.1-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hi Chris,

python-oslo.concurrency 3.7.1-0ubuntu1 has been tested successfully.

Thanks,
Corey

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-oslo.concurrency - 3.7.1-0ubuntu1

---------------
python-oslo.concurrency (3.7.1-0ubuntu1) xenial; urgency=medium

  * New upstream point release (LP: #1449062).

 -- Corey Bryant <email address hidden> Mon, 13 Jun 2016 12:34:15 -0400

[Martin Pitt (pitti)]

The verification of the Stable Release Update for python-oslo.concurrency has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Tristan Cacqueray (tristan-cacqueray)]

Here is a recap of all related patches,

master/newton (oslo.concurrency-3.8.0):
* https://review.openstack.org/243829 oslo.concurrency prlimit (merged in 3.7.1)
* https://review.openstack.org/307813 oslo.concurrency process limit (not merged in 3.7.1)
* https://review.openstack.org/307663 nova fix

stable/mitaka (oslo.concurrency-3.7.1):
* https://review.openstack.org/326327 nova backport
* https://review.openstack.org/327774 oslo.concurrency process limit backport

stable/liberty (oslo.concurrency-2.6.1):
* https://review.openstack.org/327624 nova backport
* https://review.openstack.org/327630 oslo.concurrency prlimit backport
* https://review.openstack.org/332222 oslo.concurrency process limit backport

It seems like we only need 327624 to move on the OSSA task. Is there something like a global requirement change we are waiting for ?

[Chris J Arges (arges)]

Hello Tristan, or anyone else affected,

Accepted python-oslo.concurrency into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-oslo.concurrency/2.6.1-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 14.0.0.0b2 development milestone.

[Corey Bryant (corey.bryant)]

Just a note that the Ubuntu Cloud Archive for Liberty includes the fix with python-oslo.concurrency 2.6.1-0ubuntu1~cloud0. (Wily has gone EOL since the update from Chris Arges above.)

[Jeremy Stanley (fungi)]

Based on the thread at http://lists.openstack.org/pipermail/openstack-dev/2016-September/104091.html we may need to figure out how to adjust the messaging to indicate that it was a severe enough bug to fix in stable/mitaka but that stable/liberty will be left unfixed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/327624
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6bc37dcceca823998068167b49aec6def3112397
Submitter: Jenkins
Branch: stable/liberty

commit 6bc37dcceca823998068167b49aec6def3112397
Author: Daniel P. Berrange <email address hidden>
Date: Mon Apr 18 16:32:19 2016 +0000

    virt: set address space & CPU time limits when running qemu-img

    This uses the new 'prlimit' parameter for oslo.concurrency execute
    method, to set an address space limit of 1GB and CPU time limit
    of 2 seconds, when running qemu-img.

    This is a re-implementation of the previously reverted commit

    commit da217205f53f9a38a573fb151898fbbeae41021d
    Author: Tristan Cacqueray <email address hidden>
    Date: Wed Aug 5 17:17:04 2015 +0000

        virt: Use preexec_fn to ulimit qemu-img info call

    NOTE (kchamart) [stable/liberty]: Add a check for the presence of
    'ProcessLimits' attribute (which is only present in
    oslo.concurrency>=2.6.1; and a conditional check for 'prlimit' parameter
    in qemu_img_info() method.

    Upstream discussion[1][2] that led to merging this patch to
    stable/liberty branch.

    [1] http://lists.openstack.org/pipermail/openstack-dev/2016-September/104091.html
    [2] http://lists.openstack.org/pipermail/openstack-dev/2016-September/104303.html

    Closes-Bug: #1449062
    Change-Id: I135b5242af1bfdcb0ea09a6fcda21fc03a6fbe7d
    (cherry picked from commit 068d851561addfefb2b812d91dc2011077cb6e1d)

[Jeremy Stanley (fungi)]

I'm resurrecting Grant's proposed impact description from comment #28 and updating for the year of time which has passed since. I've also edited it to remove references to Cinder and Glance... are those effectively still impacted in any supported branches? I see that the tasks API in Glance becoming admin-only in Mitaka results in this being impractical there, but what about for Liberty? And there's little input from Cinder on this bug at all but the claim is that it's exploitable there as well. Is that still the case today?

--

Title: Malicious input to qemu-img may result in resource exhaustion
Reporter: Richard W.M. Jones
Product: Nova
Affects: <=12.0.4, ==13.0.0

Description:
Richard W.M. Jones of Red Hat reported a vulnerability that affects OpenStack
Nova. By providing a maliciously crafted disk image an attacker can consume
considerable amounts of RAM and CPU time resulting in a denial of service via
resource exhaustion. Any project which makes calls to qemu-img without
appropriate ulimit restrictions in place is affected by this flaw.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/375099

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/375102

[Jeremy Stanley (fungi)]

Following discussion with Sean and Hemanth, it looks like we ought to get fixes for this into supported branches of Cinder and Glance after all. Hopefully getting them merged goes quickly now that Nova has already trodden this ground and the fixes are basically identical between them.

Assuming all fixes merge before Newton releases and prior to any stable branch point releases, this is the updated proposed impact description. I'm using our YAML format here both for convenience and clarity due to the number of repos and releases involved. I've also shortened the title and referenced the affected software in it.

Should I include a note about the updated versions of oslo.concurrency required, or are the references to patches in consuming projects sufficient for this purpose? I'd like to avoid unnecessary additional complexity if possible here.

--

date: TBD

id: TBD

title: Malicious qemu-img input may exhaust resources in Cinder, Glance, Nova

description: >
  Richard W.M. Jones of Red Hat reported a vulnerability that affects OpenStack
  Cinder, Glance and Nova. By providing a maliciously crafted disk image an
  attacker can consume considerable amounts of RAM and CPU time resulting in a
  denial of service via resource exhaustion. Any project which makes calls to
  qemu-img without appropriate ulimit restrictions in place is affected by this
  flaw.

affected-products:
  - product: cinder
    version: "<=7.0.2, >=8.0.0 <=8.1.1"
  - product: glance
    version: "<=11.0.1, ==12.0.0"
  - product: nova
    version: "<=12.0.4 and ==13.0.0"

vulnerabilities:
  - cve-id: CVE-2015-5162

reporters:
  - name: Richard W.M. Jones
    affiliation: Red Hat
    reported:
      - CVE-2015-5162

issues:
  links:
    - https://launchpad.net/bugs/1449062

reviews:
  ocata:
    - https://review.openstack.org/375099 (cinder)
    - https://review.openstack.org/TBD (glance)
  newton:
    - https://review.openstack.org/375102 (cinder)
    - https://review.openstack.org/TBD (glance)
    - https://review.openstack.org/307663 (nova)
  mitaka:
    - https://review.openstack.org/TBD (cinder)
    - https://review.openstack.org/TBD (glance)
    - https://review.openstack.org/326327 (nova)
  liberty:
    - https://review.openstack.org/TBD (cinder)
    - https://review.openstack.org/TBD (glance)
    - https://review.openstack.org/327624 (nova)

notes:
  - >
    Separate Ocata patches are listed for Cinder and Glance, as they were fixed
    during the Newton release freeze after it branched from master.

[Ian Cordasco (icordasc)]

Thank you for that info Jeremy, I've targetted it to the appropriate series in Glance so it's clear.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/375099
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=78f17f0ad79380ee3d9c50f2670252bcc559b62b
Submitter: Jenkins
Branch: master

commit 78f17f0ad79380ee3d9c50f2670252bcc559b62b
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500

    Limit memory & CPU when running qemu-img info

    It was found that a modified or corrupted image file can cause a DoS
    on the host when getting image info with qemu-img.

    This uses the newer 'prlimit' parameter for oslo.concurrency execute
    to set an address space limit of 1GB and CPU time limit of 2 seconds
    when running the qemu-img info command.

    Change-Id: If5b7129b266ef065642bc7898ce9dcf93722a053
    Closes-bug: #1449062

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/375102
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=8547444775e406a50d9d26a0003e9ba6554b0d70
Submitter: Jenkins
Branch: stable/newton

commit 8547444775e406a50d9d26a0003e9ba6554b0d70
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500

    Limit memory & CPU when running qemu-img info

    It was found that a modified or corrupted image file can cause a DoS
    on the host when getting image info with qemu-img.

    This uses the newer 'prlimit' parameter for oslo.concurrency execute
    to set an address space limit of 1GB and CPU time limit of 2 seconds
    when running the qemu-img info command.

    Change-Id: If5b7129b266ef065642bc7898ce9dcf93722a053
    Closes-bug: #1449062
    (cherry picked from commit 78f17f0ad79380ee3d9c50f2670252bcc559b62b)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/375625

[Hemanth Makkapati (hemanth-makkapati)]

It's just a theory (credits to Brian Rosmaita) at this point, but looks like "qemu-img convert" will try to infer the format of input image if "-f" is not provided. So, "qemu-img convert" may be susceptible to the same attack. Any thoughts?

[Daniel Berrange (berrange)]

Yes, *any* qemu-img command that you run without providing '-f' will try to guess the image format. Rather than trying to figure out whether a particular invokation may or may not be susceptible to attack, the safe approach is to use '-f' every time.

[Tristan Cacqueray (tristan-cacqueray)]

Jeremy, the impact description looks good. Though note that prlimit implementation before oslo.concurrency-3.8.0 doesn't support all the required resources limit, thus I would mention that oslo.concurrency>=3.8.0 is required to fix that issue.

[Jeremy Stanley (fungi)]

Hemanth, Daniel: So that means the current patches to Nova are insufficient because they missed `qemu-image convert` invocations? For example at http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/xenapi/vm_utils.py#n1128

Tristan: Thanks, it looked like oslo.concurrency got backports to stable/mitaka and stable/liberty to support it too though. Is that correct? If so we may need to get much more verbose about the library versions required by the service fixes.

[Kashyap Chamarthy (kashyapc)]

Jeremy: Hemanth (in comment#72) seems to have mixed up this bug (which sets limits for memory / CPU usage for `qemu-img` calls) with *another* bug[x] that is about disk image format guessing.

So, the Nova patches that fix this bug (1449062) are sufficient for the problem it is solving (setting a cap on memory / CPU usage).

[x] https://bugs.launchpad.net/nova/+bug/1415087 --
    [OSSA 2015-011] Format-guessing and file disclosure
    in image convert (CVE-2015-1850, CVE-2015-1851)

[Nikhil Komawar (nikhil-komawar)]

Thanks Kashyap.

That bug isn't cross listed with Glance and the current one applies to Glance in a very generic way. So, Hemanth did the right thing from our perspective. I will leave the rest of the story to the VMT team.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/375526
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f
Submitter: Jenkins
Branch: master

commit 69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f
Author: Hemanth Makkapati <email address hidden>
Date: Fri Sep 23 09:29:12 2016 -0500

    Adding constraints around qemu-img calls

    * All "qemu-img info" calls are now run under resource limitations
      that limit CPU time to 2 seconds and address space usage to 1 GB.
      This helps avoid any DoS attacks via malicious images.
    * All "qemu-img convert" calls now specify the import format so that
      it does not have to be inferred by qemu-img.

    SecurityImpact

    Change-Id: Ib900bbc05cb9ccd90c6f56ccb4bf2006e30cdc80
    Closes-Bug: #1449062

[Tristan Cacqueray (tristan-cacqueray)]

Jeremy, the missing bit for oslo-concurrency 3.7.1 is https://review.openstack.org/#/q/I164c4b35e1357a0f80ed7fe00a7ae8f49df92e31 and it was merged to stable branches. Fortunately it seems like all version >= 3.8.0 are good enough to support correct resources limit, so I guess we could just mention that.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/377734

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/377736

[Jeremy Stanley (fungi)]

Tristan: I'm still a little confused on the oslo.concurrency recommendation. Are you saying that we should suggest stable/liberty and stable/mitaka deployments to also use oslo.concurrency>=3.8.0? At the moment the tips of stable/liberty and stable/mitaka branches for oslo.concurrency are tagged 2.6.1 and 3.7.1 respectively (and that's what we have pinned in upper-constraints.txt for testing purposes as well). I don't want to imply in an advisory that all deployments should upgrade oslo.concurrency to 3.8.0 or later if there's a risk it will break liberty or mitaka deployments (which is why I was leaning toward not mentioning oslo.concurrency versions as that would just add to confusion).

[Sean McGinnis (sean-mcginnis)]

I had the same question for mitaka, so had to dig a little deeper. oslo.concurrency 3.7.1 was the follow on release to 3.7.0 to include the prlimit option. So based on current requirements for stable/mitaka, it should be safe to backport this fix.

I did not investigate stable/liberty, but I'm assuming 2.6.1 is the same situation.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/377734
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=6cba6b18a348e8fc2eb4ae25b636d065456a633c
Submitter: Jenkins
Branch: stable/newton

commit 6cba6b18a348e8fc2eb4ae25b636d065456a633c
Author: Hemanth Makkapati <email address hidden>
Date: Fri Sep 23 09:29:12 2016 -0500

    Adding constraints around qemu-img calls

    * All "qemu-img info" calls are now run under resource limitations
      that limit CPU time to 2 seconds and address space usage to 1 GB.
      This helps avoid any DoS attacks via malicious images.
    * All "qemu-img convert" calls now specify the import format so that
      it does not have to be inferred by qemu-img.

    SecurityImpact

    Change-Id: Ib900bbc05cb9ccd90c6f56ccb4bf2006e30cdc80
    Closes-Bug: #1449062
    (cherry picked from commit 69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f)

[Tristan Cacqueray (tristan-cacqueray)]

Indeed my bad, oslo-concurrency 2.6.1, 3.7.1 and 3.8.0 are all good and referenced in the upper-constraint of supported stable branches. Thus I agree we could omit that note on the advisory.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/377736
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=c90830d71969f68768d898c1c178489f602214e2
Submitter: Jenkins
Branch: stable/mitaka

commit c90830d71969f68768d898c1c178489f602214e2
Author: Hemanth Makkapati <email address hidden>
Date: Fri Sep 23 09:29:12 2016 -0500

    Adding constraints around qemu-img calls

    * All "qemu-img info" calls are now run under resource limitations
      that limit CPU time to 2 seconds and address space usage to 1 GB.
      This helps avoid any DoS attacks via malicious images.
    * All "qemu-img convert" calls now specify the import format so that
      it does not have to be inferred by qemu-img.

    SecurityImpact

    Change-Id: Ib900bbc05cb9ccd90c6f56ccb4bf2006e30cdc80
    Closes-Bug: #1449062
    (cherry picked from commit 69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 13.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 9.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 12.0.5 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/378012
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=58311904a73f931404416649dc6ed3958adc59c8
Submitter: Jenkins
Branch: stable/liberty

commit 58311904a73f931404416649dc6ed3958adc59c8
Author: Brian Rosmaita <email address hidden>
Date: Tue Sep 27 16:11:17 2016 -0400

    Adding constraints around qemu-img calls

    * All "qemu-img info" calls are now run under resource limitations that
      limit CPU time to 2 seconds and address space usage to 1 GB. This
      helps avoid any DoS attacks via malicious images.
    * All "qemu-img convert" calls now specify the import format so that it
      does not have to be inferred by qemu-img.

    SecurityImpact

    (Hemanth did all the work on this, I'm just doing the backport.)

    Co-authored-by: Hemanth Makkapati <email address hidden>
    Closes-Bug: #1449062
    (cherry picked from commit 69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f)

    Change-Id: I65f30b85439a8811545b0ca590555528631954df

[Jeremy Stanley (fungi)]

Status update: it looks like all Glance and Nova fixes have merged; so too have the master and stable/newton changes for Cinder. At this point we're waiting for https://review.openstack.org/375625 (Cinder's stable/mitaka fix) to merge, and we don't seem to have a stable/liberty backport for Cinder.

Sean: were you planning to work on a liberty change for this? Or is stable/liberty of Cinder unaffected (in which case I need to adjust the impact description accordingly)?

[Sean McGinnis (sean-mcginnis)]

I think the impact description is good. I will push through those backports.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/382573

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/375625
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c6adc020a67ae77e3645d4f6e80fa93b19432177
Submitter: Jenkins
Branch: stable/mitaka

commit c6adc020a67ae77e3645d4f6e80fa93b19432177
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500

    Limit memory & CPU when running qemu-img info

    It was found that a modified or corrupted image file can cause a DoS
    on the host when getting image info with qemu-img.

    This uses the newer 'prlimit' parameter for oslo.concurrency execute
    to set an address space limit of 1GB and CPU time limit of 2 seconds
    when running the qemu-img info command.

    Change-Id: If5b7129b266ef065642bc7898ce9dcf93722a053
    Closes-bug: #1449062
    (cherry picked from commit 8547444775e406a50d9d26a0003e9ba6554b0d70)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/382573
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=455b318ced717fb38dfe40014817d78fbc47dea5
Submitter: Jenkins
Branch: stable/liberty

commit 455b318ced717fb38dfe40014817d78fbc47dea5
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500

    Limit memory & CPU when running qemu-img info

    It was found that a modified or corrupted image file can cause a DoS
    on the host when getting image info with qemu-img.

    This uses the newer 'prlimit' parameter for oslo.concurrency execute
    to set an address space limit of 1GB and CPU time limit of 2 seconds
    when running the qemu-img info command.

    Change-Id: If5b7129b266ef065642bc7898ce9dcf93722a053
    Closes-bug: #1449062
    (cherry picked from commit 8547444775e406a50d9d26a0003e9ba6554b0d70)

[Brian Rosmaita (brian-rosmaita)]

See comment #90 for glance fix to stable/liberty

[Brian Rosmaita (brian-rosmaita)]

Fix released to all stable branches

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 12.0.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 9.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 13.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 7.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 10.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 14.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 13.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 11.0.2 release.