[OSSA 2013-029] Unchecked qcow2 root disk sizes DoS

Bug #1206081 reported by Bernhard M. Wiedemann
272
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Pádraig Brady
Folsom
Won't Fix
High
Pádraig Brady
Grizzly
Fix Released
High
Pádraig Brady
Havana
Fix Released
High
Pádraig Brady
OpenStack Security Advisory
Fix Released
Medium
Thierry Carrez

Bug Description

When doing QA for SUSE on bug 1177830
I found that the fix is incomplete,
because it assumed that the cached image would be mostly sparse.

However, I can easily create non-sparse small compressed qcow2 images with

perl -e 'for(1..11000){print "x" x 1024000}' > img
qemu-img convert -c -O qcow2 img img.qcow2
glance image-create --name=11gb --is-public=True --disk-format=qcow2 --container-format=bare < img.qcow2
nova boot --image 11gb --flavor m1.small testvm

which (in Grizzly and Essex) results in one (or two in Essex) 11GB large files being created in /var/lib/nova/instances/_base/
still allowing attackers to fill up disk space of compute nodes
because the size check is only done after the uncompressing / caching

Tags: libvirt

CVE References

Revision history for this message
Bernhard M. Wiedemann (ubuntubmw) wrote :
Revision history for this message
Thierry Carrez (ttx) wrote :

Russell, Mikal: could you confirm the vulnerability ?

Changed in ossa:
status: New → Incomplete
Revision history for this message
Russell Bryant (russellb) wrote :

Added Pádraig Brady to help review this.

Revision history for this message
Pádraig Brady (p-draigbrady) wrote :

Refreshing my memory for the flow of these images ... http://www.pixelbeat.org/docs/openstack_libvirt_images/

Yes this does seem like a valid issue, though not as serious as the related bug 1177830
as there is only an allocation per image rather than per instance.

Also any sizes over the quote size etc. for an instance wouldn't be thus referenced and so
could be auto deleted at some stage.

But yes this is a valid DOS vector given the difference between input and output sizes.

Now if we did limit sizes, I suppose we could limit to the max flavor size?
But that doesn't help much. One could still upload many small images up to the max flavor size to fill up the _base/

Maybe we need quotas glance side for max actual disk usage per user for all their images?

Revision history for this message
Thierry Carrez (ttx) wrote :

Looks like we'll need an OSSA on this then.

Changed in ossa:
importance: Undecided → Medium
status: Incomplete → Confirmed
Revision history for this message
Thierry Carrez (ttx) wrote :

Anyone on the current subscription list up for a patch ?

Revision history for this message
Thierry Carrez (ttx) wrote :

Padraig, Mikal: any chance you could produce a patch to fix this ?

Changed in nova:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Pádraig Brady (p-draigbrady) wrote :

Ok I'll take this (was away for a couple of weeks). I might get to it in the next couple of days

Changed in nova:
assignee: nobody → Pádraig Brady (p-draigbrady)
Revision history for this message
Thierry Carrez (ttx) wrote :

@Pádraig: any progress ?

Revision history for this message
Pádraig Brady (p-draigbrady) wrote :

Lets summarize _this_ bug as "per _image_ qcow2 root disk sizes DoS"
and bug 1177830 as "per _instance_ qcow2 root disk sizes DoS".

Unfortunately I think that even bug 1177830 mightn't be addressed in the case
where use_cow_images=False, i.e. in nova.virt.libvirt.imagebackend.Raw.create_image().
On initial inspection it seems like the large qcow2 files could be copied to each instance here too.

So thoughts on addressing this.

1. Always set min_disk = virt_size for an image in glance.
This is already checked in nova before we download anything.
That has the advantage of saving on network bandwidth and is
honored by cinder too for volumes.

2. Pass/Inspect size into nova.virt.images.fetch{,_to_raw}()
and disallow disk.get_disk_size('tmp_download') > size

Doing either should address both bugs I think,
but I'm leaning towards 1 as a more general solution.

Revision history for this message
Pádraig Brady (p-draigbrady) wrote :

Actually we probably should do both.
Better to have nova not rely on a particular glance implementation or version or glance at all for that matter.
I'll do the nova change now...

Thierry Carrez (ttx)
Changed in ossa:
assignee: nobody → Thierry Carrez (ttx)
Revision history for this message
Thierry Carrez (ttx) wrote :

@Padraig: we'll take the Nova patch as the vulnerability fix. We'll consider the Glance as a hardening improvement.

Revision history for this message
Thierry Carrez (ttx) wrote :

First attempt at an impact description, not sure I got this right:

----------------------
Title: Potential Nova denial of service through compressed disk images
Reporter: Bernhard M. Wiedemann (SUSE)
Products: Nova
Affects: All versions

Description:
Bernhard M. Wiedemann from SUSE reported a vulnerability in Nova's control of the size of disk images. By using malicious compressed qcow2 disk images, an authenticated user may consume large amounts of disk space, potentially resulting in a Denial of Service attack on Nova compute nodes. This issue is slightly different from CVE-2013-2096 which was addressed in OSSA 2013-012.
---------------------------

Changed in ossa:
status: Confirmed → Triaged
Revision history for this message
Pádraig Brady (p-draigbrady) wrote :

Re impact description, I'd mention that CVE-2013-2096 wasn't fully addressed either
in the non default case where use_cow_images=False, and malicious qcow images
are being transferred from glance. If mentioning both it's probably worth mentioning
the original CVE-2013-2096 was a per instance issue, while this new one is per image.

Changed in nova:
status: Confirmed → In Progress
Changed in nova:
milestone: none → havana-rc1
tags: added: havana-rc-potential libvirt
Revision history for this message
Thierry Carrez (ttx) wrote :

RC1 will probably be out before we can properly embargo and release this as an OSSA. This should stay as havana-rc-potential just in case it can make it prerelease.

Would be good to get some core pre-approvals on the proposed patches. Feel free to subscribe fellow core developers to accelerate that.

Changed in nova:
milestone: havana-rc1 → none
Revision history for this message
Russell Bryant (russellb) wrote :

Added a few nova developers that could help review these patches

Revision history for this message
Thierry Carrez (ttx) wrote :

Hey guys, please review proposed patches !

Revision history for this message
Russell Bryant (russellb) wrote :

+2 on the patch

Changed in nova:
milestone: none → icehouse-1
Revision history for this message
Pádraig Brady (p-draigbrady) wrote :

respinning with just improved commit messages

Revision history for this message
Pádraig Brady (p-draigbrady) wrote :
Revision history for this message
Pádraig Brady (p-draigbrady) wrote :
Revision history for this message
Pádraig Brady (p-draigbrady) wrote :
Revision history for this message
Thierry Carrez (ttx) wrote :

Nova core: Please review proposed patches

Thierry Carrez (ttx)
tags: added: havana-backport-potential
removed: havana-rc-potential
Revision history for this message
Thierry Carrez (ttx) wrote :

New attempt at impact description:

----------------------
Title: Potential Nova denial of service through compressed disk images
Reporter: Bernhard M. Wiedemann (SUSE)
Products: Nova
Affects: All versions

Description:
Bernhard M. Wiedemann from SUSE reported a vulnerability in Nova's control of the size of disk images. By using malicious compressed qcow2 disk images, an authenticated user may consume large amounts of disk space for each image, potentially resulting in a Denial of Service attack on Nova compute nodes. While fixing this issue, Pádraig Brady from Red Hat additionally discovered that OSSA 2013-012 did not fully address CVE-2013-2096 in the non-default case where use_cow_images=False, and malicious qcow images are being transferred from Glance. In that specific case, an authenticated user could still consume large amounts of disk space for each instance using the malicious image, potentially also resulting in a Denial of Service attack on Nova compute nodes. The provided fixes address both issues.
---------------------------

Revision history for this message
Vish Ishaya (vishvananda) wrote :

+2 on patches

Revision history for this message
Pádraig Brady (p-draigbrady) wrote :

+1 on impact description

Revision history for this message
Thierry Carrez (ttx) wrote :

CVE requested

Changed in ossa:
status: Triaged → In Progress
Revision history for this message
Thierry Carrez (ttx) wrote :

CVE-2013-4463

Revision history for this message
Thierry Carrez (ttx) wrote :

Proposed public disclosure date/time: Thursday, October 31, 1500UTC.

Changed in ossa:
status: In Progress → Fix Committed
Revision history for this message
Thierry Carrez (ttx) wrote :

CVE SPLIT w/ CVE-2013-4469:

Title: Potential Nova denial of service through compressed disk images
Reporter: Bernhard M. Wiedemann (SUSE) & Pádraig Brady (Red Hat)
Products: Nova
Affects: All versions

Description:
Bernhard M. Wiedemann from SUSE reported a vulnerability in Nova's
control of the size of disk images. By using malicious compressed qcow2
disk images, an authenticated user may consume large amounts of disk
space for each image, potentially resulting in a Denial of Service
attack on Nova compute nodes (CVE-2013-4463). While fixing this issue, Pádraig Brady from Red Hat additionally discovered that OSSA 2013-012 did not fully address CVE-2013-2096 in the non-default case where use_cow_images=False, and malicious qcow images are being transferred from Glance. In that specific case, an authenticated user could still consume large amounts of disk space for each instance using the malicious image, potentially also resulting in a Denial of Service attack on Nova compute nodes (CVE-2013-4469). The provided fixes address both issues.

Revision history for this message
Thierry Carrez (ttx) wrote :

Patches test runs:

master: tox, smoke PASS
havana: tox, neutron PASS
grizzly: tox, full PASS

Thierry Carrez (ttx)
information type: Private Security → Public Security
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/54767

Revision history for this message
Thierry Carrez (ttx) wrote : Re: Unchecked qcow2 root disk sizes DoS
Revision history for this message
Thierry Carrez (ttx) wrote :

[OSSA 2013-029]

summary: - Unchecked qcow2 root disk sizes DoS
+ [OSSA 2013-029] Unchecked qcow2 root disk sizes DoS
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/54765
Committed: http://github.com/openstack/nova/commit/f6810be4ae1a6c93e7d8017ee67d5344dfdf4a30
Submitter: Jenkins
Branch: master

commit f6810be4ae1a6c93e7d8017ee67d5344dfdf4a30
Author: Pádraig Brady <email address hidden>
Date: Fri Sep 27 04:07:14 2013 +0100

    ensure we don't boot oversized images

    Since we can't generally shrink incoming images, add extra checks
    to ensure oversized images are not allowed through.
    All cases when populating the libvirt image cache are now handled,
    including the initial download from glance, where we avoid
    converting to raw, as that could generate non sparse images
    much larger than the downloaded image.

    * nova/virt/libvirt/utils.py (fetch_image): Allow passing through
    of the max_size parameter.
    * nova/virt/images.py (fetch_to_raw): Accept the max_size parameter,
    and use it to discard images with larger (virtual) sizes.
    * nova/virt/libvirt/imagebackend.py (verify_base_size): A new
    refactored function to identify and raise exception to oversized images.
    (Raw.create_image): Pass the max_size to the fetch function.
    Also enforce virtual image size checking for already fetched images,
    as this class (despite the name) can be handling qcow files.
    (Qcow2.create_image): Pass the max_size to the fetch function,
    or verify the virtual size for the instance as done previously.
    (Lvm.create_image): Pass the max_size to the fetch function.
    Also check the size before transferring to the volume to improve
    efficiency by not even attempting the transfer of oversized images.
    (Rbd.create_image): Likewise.
    * nova/tests/virt/libvirt/fake_libvirt_utils.py: Support max_size arg.
    * nova/tests/virt/libvirt/test_libvirt.py (test_fetch_raw_image):
    Add a case to check oversized images are discarded.
    * nova/tests/virt/libvirt/test_imagebackend.py
    (test_create_image_too_small): Adjust to avoid the fetch size check.

    Fixes bug: 1177830
    Fixes bug: 1206081
    Change-Id: I3d47adaa2ad07434853f447feb27d7aae0e2e717

Changed in nova:
status: In Progress → Fix Committed
tags: added: in-stable-havana
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/havana)

Reviewed: https://review.openstack.org/54767
Committed: http://github.com/openstack/nova/commit/3cdfe894ab58f7b91bf7fb690fc5bc724e44066f
Submitter: Jenkins
Branch: stable/havana

commit 3cdfe894ab58f7b91bf7fb690fc5bc724e44066f
Author: Pádraig Brady <email address hidden>
Date: Fri Sep 27 04:07:14 2013 +0100

    ensure we don't boot oversized images

    Since we can't generally shrink incoming images, add extra checks
    to ensure oversized images are not allowed through.
    All cases when populating the libvirt image cache are now handled,
    including the initial download from glance, where we avoid
    converting to raw, as that could generate non sparse images
    much larger than the downloaded image.

    * nova/virt/libvirt/utils.py (fetch_image): Allow passing through
    of the max_size parameter.
    * nova/virt/images.py (fetch_to_raw): Accept the max_size parameter,
    and use it to discard images with larger (virtual) sizes.
    * nova/virt/libvirt/imagebackend.py (verify_base_size): A new
    refactored function to identify and raise exception to oversized images.
    (Raw.create_image): Pass the max_size to the fetch function.
    Also enforce virtual image size checking for already fetched images,
    as this class (despite the name) can be handling qcow files.
    (Qcow2.create_image): Pass the max_size to the fetch function,
    or verify the virtual size for the instance as done previously.
    (Lvm.create_image): Pass the max_size to the fetch function.
    Also check the size before transferring to the volume to improve
    efficiency by not even attempting the transfer of oversized images.
    (Rbd.create_image): Likewise.
    * nova/tests/virt/libvirt/fake_libvirt_utils.py: Support max_size arg.
    * nova/tests/virt/libvirt/test_libvirt.py (test_fetch_raw_image):
    Add a case to check oversized images are discarded.
    * nova/tests/virt/libvirt/test_imagebackend.py
    (test_create_image_too_small): Adjust to avoid the fetch size check.

    Fixes bug: 1177830
    Fixes bug: 1206081
    Change-Id: I3d47adaa2ad07434853f447feb27d7aae0e2e717

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/grizzly)

Reviewed: https://review.openstack.org/54768
Committed: http://github.com/openstack/nova/commit/135faa7b5d9855312bedc19e5e1ecebae34d3d18
Submitter: Jenkins
Branch: stable/grizzly

commit 135faa7b5d9855312bedc19e5e1ecebae34d3d18
Author: Pádraig Brady <email address hidden>
Date: Fri Sep 27 04:07:14 2013 +0100

    ensure we don't boot oversized images

    Since we can't generally shrink incoming images, add extra checks
    to ensure oversized images are not allowed through.
    All cases when populating the libvirt image cache are now handled,
    including the initial download from glance, where we avoid
    converting to raw, as that could generate non sparse images
    much larger than the downloaded image.

    * nova/virt/libvirt/utils.py (fetch_image): Allow passing through
    of the max_size parameter.
    * nova/virt/images.py (fetch_to_raw): Accept the max_size parameter,
    and use it to discard images with larger (virtual) sizes.
    * nova/virt/libvirt/imagebackend.py (verify_base_size): A new
    refactored function to identify and raise exception to oversized images.
    (Raw.create_image): Pass the max_size to the fetch function.
    Also enforce virtual image size checking for already fetched images,
    as this class (despite the name) can be handling qcow files.
    (Qcow2.create_image): Pass the max_size to the fetch function,
    or verify the virtual size for the instance as done previously.
    (Lvm.create_image): Pass the max_size to the fetch function.
    Also check the size before transferring to the volume to improve
    efficiency by not even attempting the transfer of oversized images.
    (Rbd.create_image): Likewise.
    * nova/tests/fake_libvirt_utils.py: Support max_size arg.
    * nova/tests/test_libvirt.py (test_fetch_raw_image):
    Add a case to check oversized images are discarded.
    * nova/tests/test_imagebackend.py (test_create_image_too_small):
    Adjust to avoid the fetch size check.

    Fixes bug: 1177830
    Fixes bug: 1206081

    Conflicts:

     nova/tests/test_imagebackend.py
     nova/virt/libvirt/imagebackend.py

    Change-Id: Idc35fce580be4f74e23883d1b4bea6475c3f6e30

Thierry Carrez (ttx)
Changed in ossa:
status: Fix Committed → Fix Released
Gary Kotton (garyk)
tags: removed: havana-backport-potential
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Alan Pevec (apevec)
tags: removed: in-stable-havana
Thierry Carrez (ttx)
Changed in nova:
milestone: icehouse-1 → 2014.1
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.