[OSSA 2013-012] Unchecked qcow2 root disk sizes

Bug #1177830 reported by Loganathan Parthipan
272
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Critical
Chet Burgess
Grizzly
Fix Released
Critical
Vish Ishaya
Havana
Fix Released
Critical
Pádraig Brady
OpenStack Security Advisory
Fix Released
Undecided
Michael Still

Bug Description

Currently there's no check on the root disk raw sizes. A user can create qcow2 images with any size and upload it to glance and spawn instances off this file. The raw backing file created in the compute node will be small at first due to it being a sparse file, but will grow as data is written to it. This can cause the following issues.

1. Bypass storage quota restrictions
2. Overrun compute host disk space

This was reproduced in Devstack using recent trunk d7e4692.

CVE References

Revision history for this message
Loganathan Parthipan (parthipan) wrote :

I'm not sure if this should be categorized as security or not. But I'll leave it for someone else to review and move it to public bug.

information type: Public → Private Security
Revision history for this message
Russell Bryant (russellb) wrote :

Since you originally filed this as Public, there's no reason to make it private.

information type: Private Security → Public
Revision history for this message
Chris Behrens (cbehrens) wrote :

FWIW, it looks like xenapi does check the size properly. (Just making sure this doesn't apply to xenapi also)

Revision history for this message
Russell Bryant (russellb) wrote :

Can you provide some more detail on how you reproduced this? From a discussion on IRC, this *shouldn't* be the case. cburgess is looking into reproducing it, though.

Changed in nova:
status: New → Incomplete
importance: Undecided → Critical
Revision history for this message
Chet Burgess (cfb-n) wrote :

Confirmed with the libvirt driver. We set the virtual size of the qcow2 to be the size of the backing file. We never check the flavor disk size vs. the virtual size of the backing file.

Changed in nova:
status: Incomplete → Confirmed
milestone: none → havana-1
information type: Public → Public Security
Changed in nova:
assignee: nobody → Chet Burgess (cfb-n)
Revision history for this message
Phil Day (philip-day) wrote :

Nova does have a check for the min_disk metadata property against the root_gb size of the flavor (providing min_disk is not 0).

That check in itself seems slightly flawed at the moment as the root_gb is really the minimum root volume size, and as shown here the virt layer will if required expand the root disk.

So a possible fix might be to:
 - Enforce in Glance that min_disk is always checked and set to the full size the image will expand to
 - Change the checks in nova to compare min_disk against the sum of root_gb + ephemeral_gb in the flavor

Revision history for this message
Phil Day (philip-day) wrote :

If we can't rely on min_disk being accurate (Its not clear to me to what extend Glance enforces accuracy here) then Nova will need to get this info direct from the image I guess, and use that for the checks instead of the glance metadata

Revision history for this message
Loganathan Parthipan (parthipan) wrote :

Glance updating a virtual disk size has the following benefits:

1. Cinder can check this value outright and fail fast for a volume creation if the volume size is less than the virtual size instead of having to download the image.
2. Nova can check this value and do the same against flavor quota without having to download the image.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/28717

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/28717
Committed: http://github.com/openstack/nova/commit/44a8aba1d5da87d54db48079103fdef946666d80
Submitter: Jenkins
Branch: master

commit 44a8aba1d5da87d54db48079103fdef946666d80
Author: Chet Burgess <email address hidden>
Date: Thu May 9 09:57:28 2013 +0000

    Check QCOW2 image size during root disk creation

    glance can only tell us the size of the file, not the virtual
    size of the QCOW2. As such we need to check the virtual size of
    the image once its cached and ensure it's <= to the flavor's
    root disk size.

    Change-Id: I833467284126557eb598b8350a84e10c06292fa9
    Fixes: bug 1177830

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/grizzly)

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/28901

Revision history for this message
Michael Still (mikal) wrote : Re: Unchecked qcow2 root disk sizes

Hi. So, it seems that we don't have a Folsom patch yet? Is anyone working on that? I also propose the following advisory. What do people think?

*****

OpenStack Security Advisory: 2013-XXX
CVE: Not yet assigned
Date: May 14, 2013
Title: Nova fails to verify image virtual size
Reporter: Loganathan Parthipan
Products: Nova
Affects: All versions

Loganathan Parthipan publicly reported a vulnerability in Nova. Nova did not
implement checking for the virtual size of a qcow2 image used as ephemeral
storage for instances. It is therefore possible for a user to create an image
which has a large virtual size, but little data. Once the instance is created,
the user can then proceed to fill the virtual disk, and consume all available
disk on the host node filesystem.

Havana (development branch) fix:
https://review.openstack.org/28717

Grizzly fix:
https://review.openstack.org/28901

Folsom fix:
No patch yet

References:
https://bugs.launchpad.net/keystone/+bug/1177830

Revision history for this message
Russell Bryant (russellb) wrote :

The bug URL says keystone. Other than that it seems fine.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/grizzly)

Reviewed: https://review.openstack.org/28901
Committed: http://github.com/openstack/nova/commit/a4fc0c800502338e4530cad910efb64a5483e1ea
Submitter: Jenkins
Branch: stable/grizzly

commit a4fc0c800502338e4530cad910efb64a5483e1ea
Author: Chet Burgess <email address hidden>
Date: Thu May 9 09:57:28 2013 +0000

    Check QCOW2 image size during root disk creation

    glance can only tell us the size of the file, not the virtual
    size of the QCOW2. As such we need to check the virtual size of
    the image once its cached and ensure it's <= to the flavor's
    root disk size.

    Change-Id: I833467284126557eb598b8350a84e10c06292fa9
    Fixes: bug 1177830
    (cherry picked from commit 44a8aba1d5da87d54db48079103fdef946666d80)

Revision history for this message
Thierry Carrez (ttx) wrote : Re: Unchecked qcow2 root disk sizes

Advisory looks good (except the bug reference that Russell noticed). You can drop the first three lines in the stakeholders warning (since you don't have the CVE, final date or final OSSA number yet)

Revision history for this message
Thierry Carrez (ttx) wrote :

Also wait for the Folsom patch to be proposed for review before sending the stakeholders/CVE email.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/29192

Revision history for this message
Thierry Carrez (ttx) wrote : Re: Unchecked qcow2 root disk sizes

CVE requested, tentative release date for OSSA set to Thursday, May 16, 0900 UTC

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

FYI, Essex is affected too. I have a very preliminary patch, but probably won't have it ready before tomorrow.

Revision history for this message
Thierry Carrez (ttx) wrote :

OSSA-2013-012

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Here is a patch for Essex which fixes the bug. It does not add a test case, but it has been manually tested with both use_cow_images=true and use_cow_images=false with flavors with unspecified root size, too small root size and big enough root size.

However, it introduces a regression in that if exception.ImageTooLarge() is raised, the flavor cannot be deleted:
$ nova flavor-list
...
| 8 |small1 | 256 | 1 | 1 | | 1 | 1.0 |
$ nova flavor-delete 8
ERROR: The resource could not be found. (HTTP 404)

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Oh, I should also mention that the above patch passes the testsuite.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

This may be bug #1048678, checking...

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Yes, that fixes it. People deploying my essex patch should also backport https://review.openstack.org/#/c/12729/.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Sorry, while bug #1048678 does affect essex, it is not related to the essex patch. In other words, I'm able to trigger the bug in unpatched essex. Sorry for the noise.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/folsom)

Reviewed: https://review.openstack.org/29192
Committed: http://github.com/openstack/nova/commit/6740c4141ea1152529b82cbf6e5b808eaba912e7
Submitter: Jenkins
Branch: stable/folsom

commit 6740c4141ea1152529b82cbf6e5b808eaba912e7
Author: Chet Burgess <email address hidden>
Date: Thu May 9 09:57:28 2013 +0000

    Check QCOW2 image size during root disk creation

    glance can only tell us the size of the file, not the virtual
    size of the QCOW2. As such we need to check the virtual size of
    the image once its cached and ensure it's <= to the flavor's
    root disk size.

    Change-Id: I833467284126557eb598b8350a84e10c06292fa9
    Fixes: bug 1177830
    (cherry picked from commit 44a8aba1d5da87d54db48079103fdef946666d80)

Thierry Carrez (ttx)
summary: - Unchecked qcow2 root disk sizes
+ [OSSA 2013-012] Unchecked qcow2 root disk sizes
Changed in ossa:
assignee: nobody → Michael Still (mikalstill)
status: New → Fix Released
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Revision history for this message
David Ingram (dingram) wrote :

In my folsom deployment, commit 6740c4141ea1152529b82cbf6e5b808eaba912e7 prevents new images being downloaded from Glance. Commenting the size check fixes the behaviour. As far as I can tell, this is because it is trying to get the size of an image that doesn't yet exist on disk, so the call to disk.get_disk_size(base) fails.

Tracing it through, the calls are:

 - image('disk').cache(.....) in nova.virt.libvirt.driver.LibVirtDriver._create_image() is called to start the process
 - this calls nova.virt.libvirt.imagebackend.Image.cache() with filename=<intended filename>
 - this then calls create_image() on Qcow2, with prepare_template=create_if_missing
 - create_image() then tries to check the base disk size *before* calling prepare_template (i.e. fetching the image from Glance)

Do you need any more information to help debug this? I believe the change should simply be to call prepare_template() before the disk size checks.

Revision history for this message
Chet Burgess (cfb-n) wrote :

David,

You are correct. We need to call prepare_template() before checking the size. The original fix for havanna and the grizzly backport called prepare_template() first, so I think this might be the case of a bad merge.

Revision history for this message
Thierry Carrez (ttx) wrote :

@David: could you post a separate bug so that we handle the Folsom regression ? Post the bug number here once you're done.

Looks like this should be tested somewhere, but apparently it's not :/

Thierry Carrez (ttx)
Changed in nova:
milestone: havana-1 → 2013.2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/54765

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/54767

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/grizzly)

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/54768

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/54765
Committed: http://github.com/openstack/nova/commit/f6810be4ae1a6c93e7d8017ee67d5344dfdf4a30
Submitter: Jenkins
Branch: master

commit f6810be4ae1a6c93e7d8017ee67d5344dfdf4a30
Author: Pádraig Brady <email address hidden>
Date: Fri Sep 27 04:07:14 2013 +0100

    ensure we don't boot oversized images

    Since we can't generally shrink incoming images, add extra checks
    to ensure oversized images are not allowed through.
    All cases when populating the libvirt image cache are now handled,
    including the initial download from glance, where we avoid
    converting to raw, as that could generate non sparse images
    much larger than the downloaded image.

    * nova/virt/libvirt/utils.py (fetch_image): Allow passing through
    of the max_size parameter.
    * nova/virt/images.py (fetch_to_raw): Accept the max_size parameter,
    and use it to discard images with larger (virtual) sizes.
    * nova/virt/libvirt/imagebackend.py (verify_base_size): A new
    refactored function to identify and raise exception to oversized images.
    (Raw.create_image): Pass the max_size to the fetch function.
    Also enforce virtual image size checking for already fetched images,
    as this class (despite the name) can be handling qcow files.
    (Qcow2.create_image): Pass the max_size to the fetch function,
    or verify the virtual size for the instance as done previously.
    (Lvm.create_image): Pass the max_size to the fetch function.
    Also check the size before transferring to the volume to improve
    efficiency by not even attempting the transfer of oversized images.
    (Rbd.create_image): Likewise.
    * nova/tests/virt/libvirt/fake_libvirt_utils.py: Support max_size arg.
    * nova/tests/virt/libvirt/test_libvirt.py (test_fetch_raw_image):
    Add a case to check oversized images are discarded.
    * nova/tests/virt/libvirt/test_imagebackend.py
    (test_create_image_too_small): Adjust to avoid the fetch size check.

    Fixes bug: 1177830
    Fixes bug: 1206081
    Change-Id: I3d47adaa2ad07434853f447feb27d7aae0e2e717

tags: added: in-stable-havana
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/havana)

Reviewed: https://review.openstack.org/54767
Committed: http://github.com/openstack/nova/commit/3cdfe894ab58f7b91bf7fb690fc5bc724e44066f
Submitter: Jenkins
Branch: stable/havana

commit 3cdfe894ab58f7b91bf7fb690fc5bc724e44066f
Author: Pádraig Brady <email address hidden>
Date: Fri Sep 27 04:07:14 2013 +0100

    ensure we don't boot oversized images

    Since we can't generally shrink incoming images, add extra checks
    to ensure oversized images are not allowed through.
    All cases when populating the libvirt image cache are now handled,
    including the initial download from glance, where we avoid
    converting to raw, as that could generate non sparse images
    much larger than the downloaded image.

    * nova/virt/libvirt/utils.py (fetch_image): Allow passing through
    of the max_size parameter.
    * nova/virt/images.py (fetch_to_raw): Accept the max_size parameter,
    and use it to discard images with larger (virtual) sizes.
    * nova/virt/libvirt/imagebackend.py (verify_base_size): A new
    refactored function to identify and raise exception to oversized images.
    (Raw.create_image): Pass the max_size to the fetch function.
    Also enforce virtual image size checking for already fetched images,
    as this class (despite the name) can be handling qcow files.
    (Qcow2.create_image): Pass the max_size to the fetch function,
    or verify the virtual size for the instance as done previously.
    (Lvm.create_image): Pass the max_size to the fetch function.
    Also check the size before transferring to the volume to improve
    efficiency by not even attempting the transfer of oversized images.
    (Rbd.create_image): Likewise.
    * nova/tests/virt/libvirt/fake_libvirt_utils.py: Support max_size arg.
    * nova/tests/virt/libvirt/test_libvirt.py (test_fetch_raw_image):
    Add a case to check oversized images are discarded.
    * nova/tests/virt/libvirt/test_imagebackend.py
    (test_create_image_too_small): Adjust to avoid the fetch size check.

    Fixes bug: 1177830
    Fixes bug: 1206081
    Change-Id: I3d47adaa2ad07434853f447feb27d7aae0e2e717

tags: added: in-stable-grizzly
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/grizzly)

Reviewed: https://review.openstack.org/54768
Committed: http://github.com/openstack/nova/commit/135faa7b5d9855312bedc19e5e1ecebae34d3d18
Submitter: Jenkins
Branch: stable/grizzly

commit 135faa7b5d9855312bedc19e5e1ecebae34d3d18
Author: Pádraig Brady <email address hidden>
Date: Fri Sep 27 04:07:14 2013 +0100

    ensure we don't boot oversized images

    Since we can't generally shrink incoming images, add extra checks
    to ensure oversized images are not allowed through.
    All cases when populating the libvirt image cache are now handled,
    including the initial download from glance, where we avoid
    converting to raw, as that could generate non sparse images
    much larger than the downloaded image.

    * nova/virt/libvirt/utils.py (fetch_image): Allow passing through
    of the max_size parameter.
    * nova/virt/images.py (fetch_to_raw): Accept the max_size parameter,
    and use it to discard images with larger (virtual) sizes.
    * nova/virt/libvirt/imagebackend.py (verify_base_size): A new
    refactored function to identify and raise exception to oversized images.
    (Raw.create_image): Pass the max_size to the fetch function.
    Also enforce virtual image size checking for already fetched images,
    as this class (despite the name) can be handling qcow files.
    (Qcow2.create_image): Pass the max_size to the fetch function,
    or verify the virtual size for the instance as done previously.
    (Lvm.create_image): Pass the max_size to the fetch function.
    Also check the size before transferring to the volume to improve
    efficiency by not even attempting the transfer of oversized images.
    (Rbd.create_image): Likewise.
    * nova/tests/fake_libvirt_utils.py: Support max_size arg.
    * nova/tests/test_libvirt.py (test_fetch_raw_image):
    Add a case to check oversized images are discarded.
    * nova/tests/test_imagebackend.py (test_create_image_too_small):
    Adjust to avoid the fetch size check.

    Fixes bug: 1177830
    Fixes bug: 1206081

    Conflicts:

     nova/tests/test_imagebackend.py
     nova/virt/libvirt/imagebackend.py

    Change-Id: Idc35fce580be4f74e23883d1b4bea6475c3f6e30

Alan Pevec (apevec)
tags: removed: in-stable-grizzly in-stable-havana
Sean Dague (sdague)
no longer affects: nova/folsom
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.