[OSSA 2016-012] qemu-img calls need to be restricted by ulimit (CVE-2015-5162)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
Medium
|
Sean McGinnis | |||
Mitaka |
Undecided
|
Hemanth Makkapati | |||
Newton |
Medium
|
Sean McGinnis | |||
Glance |
High
|
Hemanth Makkapati | |||
Liberty |
Undecided
|
Unassigned | |||
Mitaka |
High
|
Hemanth Makkapati | |||
Newton |
Critical
|
Hemanth Makkapati | |||
OpenStack Compute (nova) |
Medium
|
Daniel Berrange | |||
OpenStack Security Advisory |
Medium
|
Jeremy Stanley | |||
Ubuntu Cloud Archive |
Medium
|
Unassigned | |||
Liberty |
Medium
|
Unassigned | |||
Mitaka |
Medium
|
Unassigned | |||
Newton |
Medium
|
Unassigned | |||
python-oslo.concurrency (Ubuntu) |
Medium
|
Unassigned | |||
Wily |
Medium
|
Unassigned | |||
Xenial |
Medium
|
Corey Bryant | |||
Yakkety |
Medium
|
Unassigned |
Bug Description
Reported via private E-mail from Richard W.M. Jones.
Turns out qemu image parser is not hardened against malicious input and can be abused to allocated an arbitrary amount of memory and/or dump a lot of information when used with "--output=json".
The solution seems to be: limit qemu-img ressource using ulimit.
Example of abuse:
-- afl1.img --
$ /usr/bin/time qemu-img info afl1.img
image: afl1.img
[...]
0.13user 0.19system 0:00.36elapsed 92%CPU (0avgtext+0avgdata 642416maxresident)k
0inputs+0outputs (0major+
The original image is 516 bytes, but it causes qemu-img to allocate 640 MB.
-- afl2.img --
$ qemu-img info --output=json afl2.img | wc -l
589843
This is a 200K image which causes qemu-img info to output half a
million lines of JSON (14 MB of JSON).
Glance runs the --output=json variant of the command.
-- afl3.img --
$ /usr/bin/time qemu-img info afl3.img
image: afl3.img
[...]
0.09user 0.35system 0:00.47elapsed 94%CPU (0avgtext+0avgdata 1262388maxresid
0inputs+0outputs (0major+
qemu-img allocates 1.3 GB (actually, a bit more if you play with
ulimit -v). It appears that you could change it to allocate
arbitrarily large amounts of RAM.
Daniel Berrange (berrange) wrote : | #4 |
Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.
Projects security liaison and original reporter have been subscribe.
While being an issue in a dependency (qemu), it's not clear whenever OpenStack is really missing restriction on the qemu-img process...
However, depending on what a malicious user can effectively do with this issue, we may want to consider this OSSA worthy. Thought ?
Nikhil Komawar (nikhil-komawar) wrote : | #6 |
Excellent catch! And thanks for bringing this up, Richard. I was going to do some testing myself this week and file a corresponding bug.
This is a valid concern and something that we realized after the code was merged. One more point to consider here is that we need to run qemu-img in a sandbox vs. directly as a "glance" user on the the API node as possible in certain configurations. A approach for fixing this could be, using a chroot environment.
Changed in glance: | |
status: | New → Triaged |
importance: | Undecided → High |
Richard Jones (rjones-redhat) wrote : | #7 |
Here's an image which causes 4 GB to be allocated.
If you change a single byte in this image, then you can get qemu-img to crash with an allocation error:
00000000 4b 44 4d 56 01 00 00 00 03 00 00 00 01 00 ff ff |KDMV............|
00000010 ff ff 00 00 01 04 00 00 00 00 00 00 01 00 00 00 |................|
Nikhil Komawar (nikhil-komawar) wrote : | #8 |
qemu-img is run only on Glance API/worker nodes when user makes a tasks import call while the operator wants it converted into a specific format, currently.
Ref:-
http://
https:/
Daniel Berrange (berrange) wrote : | #9 |
> However, depending on what a malicious user can effectively do with this issue, we may want to consider this OSSA worthy. Thought ?
From Nova's POV I consider it to be OSSA worthy. Nova uses flavours to limit the amount of RAM that a user may consume. So if using the m1.tiny flavour, a guest is supposed to only have 500 MB RAM usage. The scheduler will consider this when deciding if a host has sufficient free RAM to run the guest. Now regardless of what the flavour limit is set to, the unprivileged user can consume as much as 4 GB of RAM on the host by uploading a malicious disk image. This is enough to cause OOM on the compute host and so affect running of other tenants VMs in a negative way.
Changed in ossa: | |
importance: | Undecided → Medium |
status: | Incomplete → Confirmed |
Changed in nova: | |
status: | New → Confirmed |
Changed in glance: | |
milestone: | none → liberty-1 |
assignee: | nobody → nikhil komawar (nikhil-komawar) |
Any progess on patch for this bug ?
Changed in nova: | |
assignee: | nobody → Daniel Berrange (berrange) |
Richard Jones (rjones-redhat) wrote : | #11 |
Although this bug is marked as secret, I have published elsewhere some information about this (it was anyway an open secret that I was fuzzing qemu-img), and it is also fixed in libguestfs.
https:/
https:/
https:/
Nikhil Komawar (nikhil-komawar) wrote : | #12 |
Thanks, I will have a Glance patch this week.
Daniel Berrange (berrange) wrote : | #13 |
This is the simplest patch I could come up with for limiting qemu-img info commands. There are various other qemu-img commands run, eg to create snapshots and the like, but I believe checking the qemu-img info command is sufficient as it is used to validate images after obtaining them from glance.
Daniel Berrange (berrange) wrote : | #14 |
Added cinder, since the file cinder/
Thanks Daniel, cinder-coresec is now subscribed. Can Glance and Cinder use the same prlimit fix ?
Daniel Berrange (berrange) wrote : | #16 |
And now I've fixed the patch to make the unit tests pass again. Opps :-)
Daniel Berrange (berrange) wrote : | #17 |
Yes, from looking at their code, the change to cinder & glance could be essentially identical.
Richard Jones (rjones-redhat) wrote : | #18 |
> "util-limit package in Linux distros."
That should say "util-linux".
Here is a proposed patch for cinder
Hum, prlimit is not readily available on stable Ubuntu and Debian...
Daniel Berrange (berrange) wrote : | #21 |
Sigh, "only" been part of util-linux since early 2012
http://
https:/
Daniel Berrange (berrange) wrote : | #22 |
The alternative approach is to modify oslo.concurrency processutils.
Changed in glance: | |
milestone: | liberty-1 → liberty-2 |
Michael Still (mikal) wrote : | #23 |
I actually much prefer the ulimit argument to processutils.
Since this bugs requires change in 4 different projects, and the impacts is moderate, I propose we open this bug in a few days for a faster solution. Is there any objections ?
Davanum Srinivas (DIMS) (dims-v) wrote : | #25 |
@mikal,
yes, i'd support a ulimit argument to processutils.
description: | updated |
information type: | Private Security → Public Security |
Daniel Berrange (berrange) wrote : | #26 |
This is a *completely untested* patch to illustrate adding support for process resource limits to oslo.concurrency's execute() call. If people agree with this approach, I'll just submit it to gerrit against oslo, as adding this feature to olso is probably useful regardless of this OSSA
Davanum Srinivas (DIMS) (dims-v) wrote : | #27 |
Hi, we've released a oslo.concurrency 1.8.1 (stable/kilo) with support to call a user specified preexec_fn to support this bug/fix. yes, same is in master as well.
80bfd1a Allow preexec_fn method for processutils.
thanks,
dims
Grant Murphy (gmurphy) wrote : | #28 |
Draft impact description:
Title: Malicious input to qemu-img may result in resource exhaustion
Reporter: Richard W.M. Jones
Product: Nova, Cinder, Glance
Affects:
Nova: 2012.1.0 versions through 2015.1.0,
Cinder: 2013.1.0 versions through 2015.1.0,
Glance: 2015.1.0
Description:
Richard W.M. Jones of Red Hat reported a vulnerability that affects several
OpenStack projects including Nova, Glance, and Cinder. By providing a
maliciously crafted disk image an attacker can consume considerable amounts
of RAM and CPU time resulting in a denial of service via resource exhaustion.
Any project which makes calls to qemu-img without appropriate ulimit restrictions
in place is affected by this flaw.
Jeremy Stanley (fungi) wrote : | #29 |
I think this would more properly be (assuming the fixes won't make it in time for 2015.1.1)...
Nova: versions through 2014.2.3, 2015.1 versions through 2015.1.1
Cinder: versions through 2014.2.3, 2015.1 versions through 2015.1.1
Glance: 2015.1 versions through 2015.1.1
Grant Murphy (gmurphy) wrote : | #30 |
CVE requested.
summary: |
- qemu-img calls need to be restricted by ulimit + qemu-img calls need to be restricted by ulimit (CVE-2015-5162) |
Changed in glance: | |
milestone: | liberty-2 → liberty-3 |
Fix proposed to branch: master
Review: https:/
Changed in nova: | |
assignee: | Daniel Berrange (berrange) → Tristan Cacqueray (tristan-cacqueray) |
status: | Confirmed → In Progress |
Changed in glance: | |
status: | Triaged → In Progress |
Tristan Cacqueray (tristan-cacqueray) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162) | #32 |
Is there a review in progress for Glance ?
Nikhil Komawar (nikhil-komawar) wrote : | #33 |
Yep.
Changed in glance: | |
milestone: | liberty-3 → liberty-rc1 |
tags: | added: liberty-rc-potential |
Changed in glance: | |
milestone: | liberty-rc1 → ongoing |
Changed in nova: | |
assignee: | Tristan Cacqueray (tristan-cacqueray) → Dan Smith (danms) |
Daniel Berrange (berrange) wrote : | #34 |
I've attached another patch which this time uses the new preexec_fn feature of oslo.concurrency 2.3.0.
This patch is *UNTESTED* beyond running the unit tests, so don't assume this actually fixes the problem without testing it in real world.
Daniel, could you please have a look at https:/
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit da217205f53f9a3
Author: Tristan Cacqueray <email address hidden>
Date: Wed Aug 5 17:17:04 2015 +0000
virt: Use preexec_fn to ulimit qemu-img info call
This uses the preexec_fn oslo.concurrency execute parameter to set user limit
before calling qemu-img info. The process is limitted to 2 seconds of cpu
and 1GB of virtual memory
Change-Id: Ib47f1116c94c8f
Closes-Bug: #1449062
Changed in nova: | |
status: | In Progress → Fix Committed |
Daniel Berrange (berrange) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162) | #37 |
FYI, we are reverting the patch for this bug because it caused a spike in gate failures
https:/
https:/
Daniel Berrange (berrange) wrote : | #38 |
The way https:/
The only way to avoid this is to *not* set limits until after we have exec'd. This would require using an external command like my original prlimit patch.
Since not all distros have prlimit, we could create a simple nova-prlimit python script todo this. We would use native prlimit if it was present in the distro (as its fast) and fallacck to nova-prlimit if missing.
It is a shame oslo.concurrency went for the preexec_fn approach, rather than my suggestion of explicitly representing resource limits in the API, as that would have allowed us to hide this compat code in oslo.concurrency, instead of dealing with it in nova, glance and cinder :-(
Can this prlimit python script could be provided by oslo or a requirement since it could be used by Cinder and Glance as well?
Note that such prlimit utility is being worked on in oslo.concurrency here: https:/
Changed in nova: | |
status: | Fix Committed → Fix Released |
Changed in nova: | |
status: | Fix Released → Confirmed |
Changed in glance: | |
assignee: | nikhil komawar (nikhil-komawar) → nobody |
Changed in nova: | |
assignee: | Dan Smith (danms) → nobody |
importance: | Undecided → Medium |
Changed in nova: | |
assignee: | nobody → Daniel Berrange (berrange) |
status: | Confirmed → In Progress |
Changed in glance: | |
importance: | High → Low |
Changed in nova: | |
status: | In Progress → Fix Released |
tags: | added: in-stable-mitaka |
Changed in python-oslo.concurrency (Ubuntu Yakkety): | |
status: | New → Fix Released |
Changed in python-oslo.concurrency (Ubuntu Xenial): | |
status: | New → Triaged |
importance: | Undecided → Medium |
Changed in python-oslo.concurrency (Ubuntu Yakkety): | |
importance: | Undecided → Medium |
Changed in python-oslo.concurrency (Ubuntu Xenial): | |
assignee: | nobody → Corey Bryant (corey.bryant) |
Changed in python-oslo.concurrency (Ubuntu Xenial): | |
status: | Triaged → Fix Committed |
tags: | added: verification-needed |
tags: |
added: verification-done removed: verification-needed |
Changed in python-oslo.concurrency (Ubuntu Xenial): | |
status: | Fix Committed → Fix Released |
status: | Fix Committed → Fix Released |
Changed in python-oslo.concurrency (Ubuntu Wily): | |
importance: | Undecided → Medium |
Changed in python-oslo.concurrency (Ubuntu Wily): | |
status: | New → Fix Committed |
tags: | removed: verification-done |
tags: | added: verification-needed |
affects: | cinder → ubuntu-translations |
no longer affects: | ubuntu-translations |
affects: | glance → ubuntu-translations |
Changed in ubuntu-translations: | |
milestone: | ongoing → none |
no longer affects: | ubuntu-translations |
Changed in cloud-archive: | |
status: | New → Fix Released |
importance: | Undecided → Medium |
Changed in ossa: | |
status: | Confirmed → Incomplete |
Changed in ossa: | |
assignee: | nobody → Jeremy Stanley (fungi) |
status: | Incomplete → In Progress |
Changed in cinder: | |
importance: | Undecided → Medium |
assignee: | nobody → Sean McGinnis (sean-mcginnis) |
Changed in cinder: | |
status: | New → In Progress |
Fix proposed to branch: stable/newton
Review: https:/
Jeremy Stanley (fungi) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162) | #67 |
Following discussion with Sean and Hemanth, it looks like we ought to get fixes for this into supported branches of Cinder and Glance after all. Hopefully getting them merged goes quickly now that Nova has already trodden this ground and the fixes are basically identical between them.
Assuming all fixes merge before Newton releases and prior to any stable branch point releases, this is the updated proposed impact description. I'm using our YAML format here both for convenience and clarity due to the number of repos and releases involved. I've also shortened the title and referenced the affected software in it.
Should I include a note about the updated versions of oslo.concurrency required, or are the references to patches in consuming projects sufficient for this purpose? I'd like to avoid unnecessary additional complexity if possible here.
--
date: TBD
id: TBD
title: Malicious qemu-img input may exhaust resources in Cinder, Glance, Nova
description: >
Richard W.M. Jones of Red Hat reported a vulnerability that affects OpenStack
Cinder, Glance and Nova. By providing a maliciously crafted disk image an
attacker can consume considerable amounts of RAM and CPU time resulting in a
denial of service via resource exhaustion. Any project which makes calls to
qemu-img without appropriate ulimit restrictions in place is affected by this
flaw.
affected-products:
- product: cinder
version: "<=7.0.2, >=8.0.0 <=8.1.1"
- product: glance
version: "<=11.0.1, ==12.0.0"
- product: nova
version: "<=12.0.4 and ==13.0.0"
vulnerabilities:
- cve-id: CVE-2015-5162
reporters:
- name: Richard W.M. Jones
affiliation: Red Hat
reported:
- CVE-2015-5162
issues:
links:
- https:/
reviews:
ocata:
- https:/
- https:/
newton:
- https:/
- https:/
- https:/
mitaka:
- https:/
- https:/
- https:/
liberty:
- https:/
- https:/
- https:/
notes:
- >
Separate Ocata patches are listed for Cinder and Glance, as they were fixed
during the Newton release freeze after it branched from master.
Ian Cordasco (icordasc) wrote : | #68 |
Thank you for that info Jeremy, I've targetted it to the appropriate series in Glance so it's clear.
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 78f17f0ad79380e
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500
Limit memory & CPU when running qemu-img info
It was found that a modified or corrupted image file can cause a DoS
on the host when getting image info with qemu-img.
This uses the newer 'prlimit' parameter for oslo.concurrency execute
to set an address space limit of 1GB and CPU time limit of 2 seconds
when running the qemu-img info command.
Change-Id: If5b7129b266ef0
Closes-bug: #1449062
Changed in cinder: | |
status: | In Progress → Fix Released |
Changed in glance: | |
assignee: | nobody → Hemanth Makkapati (hemanth-makkapati) |
importance: | Undecided → High |
status: | New → In Progress |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/newton
commit 8547444775e406a
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500
Limit memory & CPU when running qemu-img info
It was found that a modified or corrupted image file can cause a DoS
on the host when getting image info with qemu-img.
This uses the newer 'prlimit' parameter for oslo.concurrency execute
to set an address space limit of 1GB and CPU time limit of 2 seconds
when running the qemu-img info command.
Change-Id: If5b7129b266ef0
Closes-bug: #1449062
(cherry picked from commit 78f17f0ad79380e
Fix proposed to branch: stable/mitaka
Review: https:/
Hemanth Makkapati (hemanth-makkapati) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162) | #72 |
It's just a theory (credits to Brian Rosmaita) at this point, but looks like "qemu-img convert" will try to infer the format of input image if "-f" is not provided. So, "qemu-img convert" may be susceptible to the same attack. Any thoughts?
Daniel Berrange (berrange) wrote : | #73 |
Yes, *any* qemu-img command that you run without providing '-f' will try to guess the image format. Rather than trying to figure out whether a particular invokation may or may not be susceptible to attack, the safe approach is to use '-f' every time.
Jeremy, the impact description looks good. Though note that prlimit implementation before oslo.concurrenc
Jeremy Stanley (fungi) wrote : | #75 |
Hemanth, Daniel: So that means the current patches to Nova are insufficient because they missed `qemu-image convert` invocations? For example at http://
Tristan: Thanks, it looked like oslo.concurrency got backports to stable/mitaka and stable/liberty to support it too though. Is that correct? If so we may need to get much more verbose about the library versions required by the service fixes.
Kashyap Chamarthy (kashyapc) wrote : | #76 |
Jeremy: Hemanth (in comment#72) seems to have mixed up this bug (which sets limits for memory / CPU usage for `qemu-img` calls) with *another* bug[x] that is about disk image format guessing.
So, the Nova patches that fix this bug (1449062) are sufficient for the problem it is solving (setting a cap on memory / CPU usage).
[x] https:/
[OSSA 2015-011] Format-guessing and file disclosure
in image convert (CVE-2015-1850, CVE-2015-1851)
Nikhil Komawar (nikhil-komawar) wrote : | #77 |
Thanks Kashyap.
That bug isn't cross listed with Glance and the current one applies to Glance in a very generic way. So, Hemanth did the right thing from our perspective. I will leave the rest of the story to the VMT team.
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 69a9b659fd48aa3
Author: Hemanth Makkapati <email address hidden>
Date: Fri Sep 23 09:29:12 2016 -0500
Adding constraints around qemu-img calls
* All "qemu-img info" calls are now run under resource limitations
that limit CPU time to 2 seconds and address space usage to 1 GB.
This helps avoid any DoS attacks via malicious images.
* All "qemu-img convert" calls now specify the import format so that
it does not have to be inferred by qemu-img.
SecurityImpact
Change-Id: Ib900bbc05cb9cc
Closes-Bug: #1449062
Changed in glance: | |
status: | In Progress → Fix Released |
tags: | added: newton-rc-potential |
Tristan Cacqueray (tristan-cacqueray) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162) | #79 |
Jeremy, the missing bit for oslo-concurrency 3.7.1 is https:/
Fix proposed to branch: stable/newton
Review: https:/
Fix proposed to branch: stable/mitaka
Review: https:/
Jeremy Stanley (fungi) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162) | #82 |
Tristan: I'm still a little confused on the oslo.concurrency recommendation. Are you saying that we should suggest stable/liberty and stable/mitaka deployments to also use oslo.concurrenc
Sean McGinnis (sean-mcginnis) wrote : | #83 |
I had the same question for mitaka, so had to dig a little deeper. oslo.concurrency 3.7.1 was the follow on release to 3.7.0 to include the prlimit option. So based on current requirements for stable/mitaka, it should be safe to backport this fix.
I did not investigate stable/liberty, but I'm assuming 2.6.1 is the same situation.
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/newton
commit 6cba6b18a348e8f
Author: Hemanth Makkapati <email address hidden>
Date: Fri Sep 23 09:29:12 2016 -0500
Adding constraints around qemu-img calls
* All "qemu-img info" calls are now run under resource limitations
that limit CPU time to 2 seconds and address space usage to 1 GB.
This helps avoid any DoS attacks via malicious images.
* All "qemu-img convert" calls now specify the import format so that
it does not have to be inferred by qemu-img.
SecurityImpact
Change-Id: Ib900bbc05cb9cc
Closes-Bug: #1449062
(cherry picked from commit 69a9b659fd48aa3
Tristan Cacqueray (tristan-cacqueray) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162) | #85 |
Indeed my bad, oslo-concurrency 2.6.1, 3.7.1 and 3.8.0 are all good and referenced in the upper-constraint of supported stable branches. Thus I agree we could omit that note on the advisory.
Changed in glance: | |
status: | Fix Released → Fix Committed |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/mitaka
commit c90830d71969f68
Author: Hemanth Makkapati <email address hidden>
Date: Fri Sep 23 09:29:12 2016 -0500
Adding constraints around qemu-img calls
* All "qemu-img info" calls are now run under resource limitations
that limit CPU time to 2 seconds and address space usage to 1 GB.
This helps avoid any DoS attacks via malicious images.
* All "qemu-img convert" calls now specify the import format so that
it does not have to be inferred by qemu-img.
SecurityImpact
Change-Id: Ib900bbc05cb9cc
Closes-Bug: #1449062
(cherry picked from commit 69a9b659fd48aa3
This issue was fixed in the openstack/glance 13.0.0.0rc2 release candidate.
This issue was fixed in the openstack/cinder 9.0.0.0rc2 release candidate.
This issue was fixed in the openstack/nova 12.0.5 release.
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/liberty
commit 58311904a73f931
Author: Brian Rosmaita <email address hidden>
Date: Tue Sep 27 16:11:17 2016 -0400
Adding constraints around qemu-img calls
* All "qemu-img info" calls are now run under resource limitations that
limit CPU time to 2 seconds and address space usage to 1 GB. This
helps avoid any DoS attacks via malicious images.
* All "qemu-img convert" calls now specify the import format so that it
does not have to be inferred by qemu-img.
SecurityImpact
(Hemanth did all the work on this, I'm just doing the backport.)
Co-authored-by: Hemanth Makkapati <email address hidden>
Closes-Bug: #1449062
(cherry picked from commit 69a9b659fd48aa3
Change-Id: I65f30b85439a88
Jeremy Stanley (fungi) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162) | #91 |
Status update: it looks like all Glance and Nova fixes have merged; so too have the master and stable/newton changes for Cinder. At this point we're waiting for https:/
Sean: were you planning to work on a liberty change for this? Or is stable/liberty of Cinder unaffected (in which case I need to adjust the impact description accordingly)?
Sean McGinnis (sean-mcginnis) wrote : | #92 |
I think the impact description is good. I will push through those backports.
Fix proposed to branch: stable/liberty
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/mitaka
commit c6adc020a67ae77
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500
Limit memory & CPU when running qemu-img info
It was found that a modified or corrupted image file can cause a DoS
on the host when getting image info with qemu-img.
This uses the newer 'prlimit' parameter for oslo.concurrency execute
to set an address space limit of 1GB and CPU time limit of 2 seconds
when running the qemu-img info command.
Change-Id: If5b7129b266ef0
Closes-bug: #1449062
(cherry picked from commit 8547444775e406a
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/liberty
commit 455b318ced717fb
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500
Limit memory & CPU when running qemu-img info
It was found that a modified or corrupted image file can cause a DoS
on the host when getting image info with qemu-img.
This uses the newer 'prlimit' parameter for oslo.concurrency execute
to set an address space limit of 1GB and CPU time limit of 2 seconds
when running the qemu-img info command.
Change-Id: If5b7129b266ef0
Closes-bug: #1449062
(cherry picked from commit 8547444775e406a
summary: |
- qemu-img calls need to be restricted by ulimit (CVE-2015-5162) + [OSSA 2016-012] qemu-img calls need to be restricted by ulimit + (CVE-2015-5162) |
Changed in ossa: | |
status: | In Progress → Fix Released |
Brian Rosmaita (brian-rosmaita) wrote : | #96 |
See comment #90 for glance fix to stable/liberty
Brian Rosmaita (brian-rosmaita) wrote : | #97 |
Fix released to all stable branches
Changed in glance: | |
status: | Fix Committed → Fix Released |
This issue was fixed in the openstack/nova 12.0.5 release.
This issue was fixed in the openstack/cinder 9.0.0 release.
This issue was fixed in the openstack/glance 13.0.0 release.
This issue was fixed in the openstack/cinder 7.0.3 release.
This issue was fixed in the openstack/cinder 10.0.0.0b1 development milestone.
This issue was fixed in the openstack/glance 14.0.0.0b1 development milestone.
This issue was fixed in the openstack/glance 13.0.0 release.
This issue was fixed in the openstack/glance 11.0.2 release.
FYI, I tested a recentish git master to see if I can trigger it and was succesful:
What I did was
$ cd /usr/bin
$ mv qemu-img qemu-img.real
$ cat > qemu-img <<EOF
#!/bin/sh
echo "$@" >> /tmp/usage.log qemu-img. real "$@" 2>> /tmp/usage.log
/usr/bin/time /usr/bin/
echo >> /tmp/usage.log
EOF
$ chmod +x qemu-img
Then
$ glance image-create --name afl3 --disk-format vmdk --container-format bare --file /home/berrange/ afl3.img --is-public true
Glance did not appear to run qemu-img info at this stage, but when I then boot the image iwth Nova:
$ nova boot --image afl3 --flavor m1.tiny afl3
Nova correctly refuses to boot the guest, because the disk image is larger
than the maximum it permits:
2015-04-27 14:39:00.381 TRACE nova.compute. manager [instance: aab2db9f- cbcc-4858- 98e8-95e82b1d3b 3b] return f(*args, **kwargs) manager [instance: aab2db9f- cbcc-4858- 98e8-95e82b1d3b 3b] File berrange/ src/cloud/ nova/nova/ virt/libvirt/ imagebackend. py", line 221, in fetch_func_sync manager [instance: aab2db9f- cbcc-4858- 98e8-95e82b1d3b 3b] fetch_func( target= target, *args, **kwargs) manager [instance: aab2db9f- cbcc-4858- 98e8-95e82b1d3b 3b] File berrange/ src/cloud/ nova/nova/ virt/libvirt/ utils.py" , line 501, in fetch_image manager [instance: aab2db9f- cbcc-4858- 98e8-95e82b1d3b 3b] max_size=max_size) manager [instance: aab2db9f- cbcc-4858- 98e8-95e82b1d3b 3b] File berrange/ src/cloud/ nova/nova/ virt/images. py", line 119, in fetch_to_raw manager [instance: aab2db9f- cbcc-4858- 98e8-95e82b1d3b 3b] raise exception. FlavorDiskTooSm all() manager [instance: aab2db9f- cbcc-4858- 98e8-95e82b1d3b 3b] FlavorDiskTooSmall: Flavor's disk is too small for manager [instance: aab2db9f- cbcc-4858- 98e8-95e82b1d3b 3b]
2015-04-27 14:39:00.381 TRACE nova.compute.
+"/home/
2015-04-27 14:39:00.381 TRACE nova.compute.
2015-04-27 14:39:00.381 TRACE nova.compute.
+"/home/
2015-04-27 14:39:00.381 TRACE nova.compute.
2015-04-27 14:39:00.381 TRACE nova.compute.
+"/home/
2015-04-27 14:39:00.381 TRACE nova.compute.
2015-04-27 14:39:00.381 TRACE nova.compute.
+requested image.
2015-04-27 14:39:00.381 TRACE nova.compute.
However in checking that maximum limit, it has indeed run qemu-img info
and suffered the memory usage DOS.
$ cat /tmp/usage.log
info /home/berrange/ src/cloud/ data/nova/ instances/ _base/30854c868 15b92c21c8af45c f8ff5757c04046a a.part ent)k 311876minor) pagefaults 0swaps
0.08user 0.94system 0:01.24elapsed 82%CPU (0avgtext+0avgdata 1260680maxresid
0inputs+0outputs (0major+
So someone needs to see just what code paths in glance might trigger the qemu-img info call. I also wonder if there is any risk to cinder, because it too has various invokations of qemu-img