[OSSA 2013-012] Unchecked qcow2 root disk sizes

Bug #1177830 reported by Loganathan Parthipan on 2013-05-08
272
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Critical
Chet Burgess
Grizzly
Critical
Vish Ishaya
Havana
Critical
Pádraig Brady
OpenStack Security Advisory
Undecided
Michael Still

Bug Description

Currently there's no check on the root disk raw sizes. A user can create qcow2 images with any size and upload it to glance and spawn instances off this file. The raw backing file created in the compute node will be small at first due to it being a sparse file, but will grow as data is written to it. This can cause the following issues.

1. Bypass storage quota restrictions
2. Overrun compute host disk space

This was reproduced in Devstack using recent trunk d7e4692.

CVE References

I'm not sure if this should be categorized as security or not. But I'll leave it for someone else to review and move it to public bug.

information type: Public → Private Security
Russell Bryant (russellb) wrote :

Since you originally filed this as Public, there's no reason to make it private.

information type: Private Security → Public
Chris Behrens (cbehrens) wrote :

FWIW, it looks like xenapi does check the size properly. (Just making sure this doesn't apply to xenapi also)

Russell Bryant (russellb) wrote :

Can you provide some more detail on how you reproduced this? From a discussion on IRC, this *shouldn't* be the case. cburgess is looking into reproducing it, though.

Changed in nova:
status: New → Incomplete
importance: Undecided → Critical
Chet Burgess (cfb-n) wrote :

Confirmed with the libvirt driver. We set the virtual size of the qcow2 to be the size of the backing file. We never check the flavor disk size vs. the virtual size of the backing file.

Changed in nova:
status: Incomplete → Confirmed
milestone: none → havana-1
information type: Public → Public Security
Changed in nova:
assignee: nobody → Chet Burgess (cfb-n)
Phil Day (philip-day) wrote :

Nova does have a check for the min_disk metadata property against the root_gb size of the flavor (providing min_disk is not 0).

That check in itself seems slightly flawed at the moment as the root_gb is really the minimum root volume size, and as shown here the virt layer will if required expand the root disk.

So a possible fix might be to:
 - Enforce in Glance that min_disk is always checked and set to the full size the image will expand to
 - Change the checks in nova to compare min_disk against the sum of root_gb + ephemeral_gb in the flavor

Phil Day (philip-day) wrote :

If we can't rely on min_disk being accurate (Its not clear to me to what extend Glance enforces accuracy here) then Nova will need to get this info direct from the image I guess, and use that for the checks instead of the glance metadata

Glance updating a virtual disk size has the following benefits:

1. Cinder can check this value outright and fail fast for a volume creation if the volume size is less than the virtual size instead of having to download the image.
2. Nova can check this value and do the same against flavor quota without having to download the image.

Fix proposed to branch: master
Review: https://review.openstack.org/28717

Changed in nova:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/28717
Committed: http://github.com/openstack/nova/commit/44a8aba1d5da87d54db48079103fdef946666d80
Submitter: Jenkins
Branch: master

commit 44a8aba1d5da87d54db48079103fdef946666d80
Author: Chet Burgess <email address hidden>
Date: Thu May 9 09:57:28 2013 +0000

    Check QCOW2 image size during root disk creation

    glance can only tell us the size of the file, not the virtual
    size of the QCOW2. As such we need to check the virtual size of
    the image once its cached and ensure it's <= to the flavor's
    root disk size.

    Change-Id: I833467284126557eb598b8350a84e10c06292fa9
    Fixes: bug 1177830

Changed in nova:
status: In Progress → Fix Committed

Hi. So, it seems that we don't have a Folsom patch yet? Is anyone working on that? I also propose the following advisory. What do people think?

*****

OpenStack Security Advisory: 2013-XXX
CVE: Not yet assigned
Date: May 14, 2013
Title: Nova fails to verify image virtual size
Reporter: Loganathan Parthipan
Products: Nova
Affects: All versions

Loganathan Parthipan publicly reported a vulnerability in Nova. Nova did not
implement checking for the virtual size of a qcow2 image used as ephemeral
storage for instances. It is therefore possible for a user to create an image
which has a large virtual size, but little data. Once the instance is created,
the user can then proceed to fill the virtual disk, and consume all available
disk on the host node filesystem.

Havana (development branch) fix:
https://review.openstack.org/28717

Grizzly fix:
https://review.openstack.org/28901

Folsom fix:
No patch yet

References:
https://bugs.launchpad.net/keystone/+bug/1177830

Russell Bryant (russellb) wrote :

The bug URL says keystone. Other than that it seems fine.

Reviewed: https://review.openstack.org/28901
Committed: http://github.com/openstack/nova/commit/a4fc0c800502338e4530cad910efb64a5483e1ea
Submitter: Jenkins
Branch: stable/grizzly

commit a4fc0c800502338e4530cad910efb64a5483e1ea
Author: Chet Burgess <email address hidden>
Date: Thu May 9 09:57:28 2013 +0000

    Check QCOW2 image size during root disk creation

    glance can only tell us the size of the file, not the virtual
    size of the QCOW2. As such we need to check the virtual size of
    the image once its cached and ensure it's <= to the flavor's
    root disk size.

    Change-Id: I833467284126557eb598b8350a84e10c06292fa9
    Fixes: bug 1177830
    (cherry picked from commit 44a8aba1d5da87d54db48079103fdef946666d80)

Advisory looks good (except the bug reference that Russell noticed). You can drop the first three lines in the stakeholders warning (since you don't have the CVE, final date or final OSSA number yet)

Thierry Carrez (ttx) wrote :

Also wait for the Folsom patch to be proposed for review before sending the stakeholders/CVE email.

CVE requested, tentative release date for OSSA set to Thursday, May 16, 0900 UTC

Jamie Strandboge (jdstrand) wrote :

FYI, Essex is affected too. I have a very preliminary patch, but probably won't have it ready before tomorrow.

Thierry Carrez (ttx) wrote :

OSSA-2013-012

Jamie Strandboge (jdstrand) wrote :

Here is a patch for Essex which fixes the bug. It does not add a test case, but it has been manually tested with both use_cow_images=true and use_cow_images=false with flavors with unspecified root size, too small root size and big enough root size.

However, it introduces a regression in that if exception.ImageTooLarge() is raised, the flavor cannot be deleted:
$ nova flavor-list
...
| 8 |small1 | 256 | 1 | 1 | | 1 | 1.0 |
$ nova flavor-delete 8
ERROR: The resource could not be found. (HTTP 404)

Jamie Strandboge (jdstrand) wrote :

Oh, I should also mention that the above patch passes the testsuite.

Jamie Strandboge (jdstrand) wrote :

This may be bug #1048678, checking...

Jamie Strandboge (jdstrand) wrote :

Yes, that fixes it. People deploying my essex patch should also backport https://review.openstack.org/#/c/12729/.

Jamie Strandboge (jdstrand) wrote :

Sorry, while bug #1048678 does affect essex, it is not related to the essex patch. In other words, I'm able to trigger the bug in unpatched essex. Sorry for the noise.

Reviewed: https://review.openstack.org/29192
Committed: http://github.com/openstack/nova/commit/6740c4141ea1152529b82cbf6e5b808eaba912e7
Submitter: Jenkins
Branch: stable/folsom

commit 6740c4141ea1152529b82cbf6e5b808eaba912e7
Author: Chet Burgess <email address hidden>
Date: Thu May 9 09:57:28 2013 +0000

    Check QCOW2 image size during root disk creation

    glance can only tell us the size of the file, not the virtual
    size of the QCOW2. As such we need to check the virtual size of
    the image once its cached and ensure it's <= to the flavor's
    root disk size.

    Change-Id: I833467284126557eb598b8350a84e10c06292fa9
    Fixes: bug 1177830
    (cherry picked from commit 44a8aba1d5da87d54db48079103fdef946666d80)

Thierry Carrez (ttx) on 2013-05-24
summary: - Unchecked qcow2 root disk sizes
+ [OSSA 2013-012] Unchecked qcow2 root disk sizes
Changed in ossa:
assignee: nobody → Michael Still (mikalstill)
status: New → Fix Released
Thierry Carrez (ttx) on 2013-05-29
Changed in nova:
status: Fix Committed → Fix Released
David Ingram (dingram) wrote :

In my folsom deployment, commit 6740c4141ea1152529b82cbf6e5b808eaba912e7 prevents new images being downloaded from Glance. Commenting the size check fixes the behaviour. As far as I can tell, this is because it is trying to get the size of an image that doesn't yet exist on disk, so the call to disk.get_disk_size(base) fails.

Tracing it through, the calls are:

 - image('disk').cache(.....) in nova.virt.libvirt.driver.LibVirtDriver._create_image() is called to start the process
 - this calls nova.virt.libvirt.imagebackend.Image.cache() with filename=<intended filename>
 - this then calls create_image() on Qcow2, with prepare_template=create_if_missing
 - create_image() then tries to check the base disk size *before* calling prepare_template (i.e. fetching the image from Glance)

Do you need any more information to help debug this? I believe the change should simply be to call prepare_template() before the disk size checks.

Chet Burgess (cfb-n) wrote :

David,

You are correct. We need to call prepare_template() before checking the size. The original fix for havanna and the grizzly backport called prepare_template() first, so I think this might be the case of a bad merge.

Thierry Carrez (ttx) wrote :

@David: could you post a separate bug so that we handle the Folsom regression ? Post the bug number here once you're done.

Looks like this should be tested somewhere, but apparently it's not :/

Thierry Carrez (ttx) on 2013-10-17
Changed in nova:
milestone: havana-1 → 2013.2

Reviewed: https://review.openstack.org/54765
Committed: http://github.com/openstack/nova/commit/f6810be4ae1a6c93e7d8017ee67d5344dfdf4a30
Submitter: Jenkins
Branch: master

commit f6810be4ae1a6c93e7d8017ee67d5344dfdf4a30
Author: Pádraig Brady <email address hidden>
Date: Fri Sep 27 04:07:14 2013 +0100

    ensure we don't boot oversized images

    Since we can't generally shrink incoming images, add extra checks
    to ensure oversized images are not allowed through.
    All cases when populating the libvirt image cache are now handled,
    including the initial download from glance, where we avoid
    converting to raw, as that could generate non sparse images
    much larger than the downloaded image.

    * nova/virt/libvirt/utils.py (fetch_image): Allow passing through
    of the max_size parameter.
    * nova/virt/images.py (fetch_to_raw): Accept the max_size parameter,
    and use it to discard images with larger (virtual) sizes.
    * nova/virt/libvirt/imagebackend.py (verify_base_size): A new
    refactored function to identify and raise exception to oversized images.
    (Raw.create_image): Pass the max_size to the fetch function.
    Also enforce virtual image size checking for already fetched images,
    as this class (despite the name) can be handling qcow files.
    (Qcow2.create_image): Pass the max_size to the fetch function,
    or verify the virtual size for the instance as done previously.
    (Lvm.create_image): Pass the max_size to the fetch function.
    Also check the size before transferring to the volume to improve
    efficiency by not even attempting the transfer of oversized images.
    (Rbd.create_image): Likewise.
    * nova/tests/virt/libvirt/fake_libvirt_utils.py: Support max_size arg.
    * nova/tests/virt/libvirt/test_libvirt.py (test_fetch_raw_image):
    Add a case to check oversized images are discarded.
    * nova/tests/virt/libvirt/test_imagebackend.py
    (test_create_image_too_small): Adjust to avoid the fetch size check.

    Fixes bug: 1177830
    Fixes bug: 1206081
    Change-Id: I3d47adaa2ad07434853f447feb27d7aae0e2e717

tags: added: in-stable-havana

Reviewed: https://review.openstack.org/54767
Committed: http://github.com/openstack/nova/commit/3cdfe894ab58f7b91bf7fb690fc5bc724e44066f
Submitter: Jenkins
Branch: stable/havana

commit 3cdfe894ab58f7b91bf7fb690fc5bc724e44066f
Author: Pádraig Brady <email address hidden>
Date: Fri Sep 27 04:07:14 2013 +0100

    ensure we don't boot oversized images

    Since we can't generally shrink incoming images, add extra checks
    to ensure oversized images are not allowed through.
    All cases when populating the libvirt image cache are now handled,
    including the initial download from glance, where we avoid
    converting to raw, as that could generate non sparse images
    much larger than the downloaded image.

    * nova/virt/libvirt/utils.py (fetch_image): Allow passing through
    of the max_size parameter.
    * nova/virt/images.py (fetch_to_raw): Accept the max_size parameter,
    and use it to discard images with larger (virtual) sizes.
    * nova/virt/libvirt/imagebackend.py (verify_base_size): A new
    refactored function to identify and raise exception to oversized images.
    (Raw.create_image): Pass the max_size to the fetch function.
    Also enforce virtual image size checking for already fetched images,
    as this class (despite the name) can be handling qcow files.
    (Qcow2.create_image): Pass the max_size to the fetch function,
    or verify the virtual size for the instance as done previously.
    (Lvm.create_image): Pass the max_size to the fetch function.
    Also check the size before transferring to the volume to improve
    efficiency by not even attempting the transfer of oversized images.
    (Rbd.create_image): Likewise.
    * nova/tests/virt/libvirt/fake_libvirt_utils.py: Support max_size arg.
    * nova/tests/virt/libvirt/test_libvirt.py (test_fetch_raw_image):
    Add a case to check oversized images are discarded.
    * nova/tests/virt/libvirt/test_imagebackend.py
    (test_create_image_too_small): Adjust to avoid the fetch size check.

    Fixes bug: 1177830
    Fixes bug: 1206081
    Change-Id: I3d47adaa2ad07434853f447feb27d7aae0e2e717

tags: added: in-stable-grizzly

Reviewed: https://review.openstack.org/54768
Committed: http://github.com/openstack/nova/commit/135faa7b5d9855312bedc19e5e1ecebae34d3d18
Submitter: Jenkins
Branch: stable/grizzly

commit 135faa7b5d9855312bedc19e5e1ecebae34d3d18
Author: Pádraig Brady <email address hidden>
Date: Fri Sep 27 04:07:14 2013 +0100

    ensure we don't boot oversized images

    Since we can't generally shrink incoming images, add extra checks
    to ensure oversized images are not allowed through.
    All cases when populating the libvirt image cache are now handled,
    including the initial download from glance, where we avoid
    converting to raw, as that could generate non sparse images
    much larger than the downloaded image.

    * nova/virt/libvirt/utils.py (fetch_image): Allow passing through
    of the max_size parameter.
    * nova/virt/images.py (fetch_to_raw): Accept the max_size parameter,
    and use it to discard images with larger (virtual) sizes.
    * nova/virt/libvirt/imagebackend.py (verify_base_size): A new
    refactored function to identify and raise exception to oversized images.
    (Raw.create_image): Pass the max_size to the fetch function.
    Also enforce virtual image size checking for already fetched images,
    as this class (despite the name) can be handling qcow files.
    (Qcow2.create_image): Pass the max_size to the fetch function,
    or verify the virtual size for the instance as done previously.
    (Lvm.create_image): Pass the max_size to the fetch function.
    Also check the size before transferring to the volume to improve
    efficiency by not even attempting the transfer of oversized images.
    (Rbd.create_image): Likewise.
    * nova/tests/fake_libvirt_utils.py: Support max_size arg.
    * nova/tests/test_libvirt.py (test_fetch_raw_image):
    Add a case to check oversized images are discarded.
    * nova/tests/test_imagebackend.py (test_create_image_too_small):
    Adjust to avoid the fetch size check.

    Fixes bug: 1177830
    Fixes bug: 1206081

    Conflicts:

     nova/tests/test_imagebackend.py
     nova/virt/libvirt/imagebackend.py

    Change-Id: Idc35fce580be4f74e23883d1b4bea6475c3f6e30

Alan Pevec (apevec) on 2013-12-14
tags: removed: in-stable-grizzly in-stable-havana
Sean Dague (sdague) on 2014-09-19
no longer affects: nova/folsom
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Duplicates of this bug

Other bug subscribers