[OSSA 2013-029] Unchecked qcow2 root disk sizes DoS

[Bernhard M. Wiedemann]

When doing QA for SUSE on bug 1177830
I found that the fix is incomplete,
because it assumed that the cached image would be mostly sparse.

However, I can easily create non-sparse small compressed qcow2 images with

perl -e 'for(1..11000){print "x" x 1024000}' > img
qemu-img convert -c -O qcow2 img img.qcow2
glance image-create --name=11gb --is-public=True --disk-format=qcow2 --container-format=bare < img.qcow2
nova boot --image 11gb --flavor m1.small testvm

which (in Grizzly and Essex) results in one (or two in Essex) 11GB large files being created in /var/lib/nova/instances/_base/
still allowing attackers to fill up disk space of compute nodes
because the size check is only done after the uncompressing / caching

[Thierry Carrez (ttx)]

Russell, Mikal: could you confirm the vulnerability ?

[Russell Bryant (russellb)]

Added Pádraig Brady to help review this.

[Pádraig Brady (p-draigbrady)]

Refreshing my memory for the flow of these images ... http://www.pixelbeat.org/docs/openstack_libvirt_images/

Yes this does seem like a valid issue, though not as serious as the related bug 1177830
as there is only an allocation per image rather than per instance.

Also any sizes over the quote size etc. for an instance wouldn't be thus referenced and so
could be auto deleted at some stage.

But yes this is a valid DOS vector given the difference between input and output sizes.

Now if we did limit sizes, I suppose we could limit to the max flavor size?
But that doesn't help much. One could still upload many small images up to the max flavor size to fill up the _base/

Maybe we need quotas glance side for max actual disk usage per user for all their images?

[Thierry Carrez (ttx)]

Looks like we'll need an OSSA on this then.

[Thierry Carrez (ttx)]

Anyone on the current subscription list up for a patch ?

[Thierry Carrez (ttx)]

Padraig, Mikal: any chance you could produce a patch to fix this ?

[Pádraig Brady (p-draigbrady)]

Ok I'll take this (was away for a couple of weeks). I might get to it in the next couple of days

[Thierry Carrez (ttx)]

@Pádraig: any progress ?

[Pádraig Brady (p-draigbrady)]

Lets summarize _this_ bug as "per _image_ qcow2 root disk sizes DoS"
and bug 1177830 as "per _instance_ qcow2 root disk sizes DoS".

Unfortunately I think that even bug 1177830 mightn't be addressed in the case
where use_cow_images=False, i.e. in nova.virt.libvirt.imagebackend.Raw.create_image().
On initial inspection it seems like the large qcow2 files could be copied to each instance here too.

So thoughts on addressing this.

1. Always set min_disk = virt_size for an image in glance.
This is already checked in nova before we download anything.
That has the advantage of saving on network bandwidth and is
honored by cinder too for volumes.

2. Pass/Inspect size into nova.virt.images.fetch{,_to_raw}()
and disallow disk.get_disk_size('tmp_download') > size

Doing either should address both bugs I think,
but I'm leaning towards 1 as a more general solution.

[Pádraig Brady (p-draigbrady)]

Actually we probably should do both.
Better to have nova not rely on a particular glance implementation or version or glance at all for that matter.
I'll do the nova change now...

[Thierry Carrez (ttx)]

@Padraig: we'll take the Nova patch as the vulnerability fix. We'll consider the Glance as a hardening improvement.

[Thierry Carrez (ttx)]

First attempt at an impact description, not sure I got this right:

----------------------
Title: Potential Nova denial of service through compressed disk images
Reporter: Bernhard M. Wiedemann (SUSE)
Products: Nova
Affects: All versions

Description:
Bernhard M. Wiedemann from SUSE reported a vulnerability in Nova's control of the size of disk images. By using malicious compressed qcow2 disk images, an authenticated user may consume large amounts of disk space, potentially resulting in a Denial of Service attack on Nova compute nodes. This issue is slightly different from CVE-2013-2096 which was addressed in OSSA 2013-012.
---------------------------

[Pádraig Brady (p-draigbrady)]

Re impact description, I'd mention that CVE-2013-2096 wasn't fully addressed either
in the non default case where use_cow_images=False, and malicious qcow images
are being transferred from glance. If mentioning both it's probably worth mentioning
the original CVE-2013-2096 was a per instance issue, while this new one is per image.

[Thierry Carrez (ttx)]

RC1 will probably be out before we can properly embargo and release this as an OSSA. This should stay as havana-rc-potential just in case it can make it prerelease.

Would be good to get some core pre-approvals on the proposed patches. Feel free to subscribe fellow core developers to accelerate that.

[Russell Bryant (russellb)]

Added a few nova developers that could help review these patches

[Thierry Carrez (ttx)]

Hey guys, please review proposed patches !

[Russell Bryant (russellb)]

+2 on the patch

[Pádraig Brady (p-draigbrady)]

respinning with just improved commit messages

[Thierry Carrez (ttx)]

Nova core: Please review proposed patches

[Thierry Carrez (ttx)]

New attempt at impact description:

----------------------
Title: Potential Nova denial of service through compressed disk images
Reporter: Bernhard M. Wiedemann (SUSE)
Products: Nova
Affects: All versions

Description:
Bernhard M. Wiedemann from SUSE reported a vulnerability in Nova's control of the size of disk images. By using malicious compressed qcow2 disk images, an authenticated user may consume large amounts of disk space for each image, potentially resulting in a Denial of Service attack on Nova compute nodes. While fixing this issue, Pádraig Brady from Red Hat additionally discovered that OSSA 2013-012 did not fully address CVE-2013-2096 in the non-default case where use_cow_images=False, and malicious qcow images are being transferred from Glance. In that specific case, an authenticated user could still consume large amounts of disk space for each instance using the malicious image, potentially also resulting in a Denial of Service attack on Nova compute nodes. The provided fixes address both issues.
---------------------------

[Vish Ishaya (vishvananda)]

+2 on patches

[Pádraig Brady (p-draigbrady)]

+1 on impact description

[Thierry Carrez (ttx)]

CVE requested

[Thierry Carrez (ttx)]

CVE-2013-4463

[Thierry Carrez (ttx)]

Proposed public disclosure date/time: Thursday, October 31, 1500UTC.

[Thierry Carrez (ttx)]

CVE SPLIT w/ CVE-2013-4469:

Title: Potential Nova denial of service through compressed disk images
Reporter: Bernhard M. Wiedemann (SUSE) & Pádraig Brady (Red Hat)
Products: Nova
Affects: All versions

Description:
Bernhard M. Wiedemann from SUSE reported a vulnerability in Nova's
control of the size of disk images. By using malicious compressed qcow2
disk images, an authenticated user may consume large amounts of disk
space for each image, potentially resulting in a Denial of Service
attack on Nova compute nodes (CVE-2013-4463). While fixing this issue, Pádraig Brady from Red Hat additionally discovered that OSSA 2013-012 did not fully address CVE-2013-2096 in the non-default case where use_cow_images=False, and malicious qcow images are being transferred from Glance. In that specific case, an authenticated user could still consume large amounts of disk space for each instance using the malicious image, potentially also resulting in a Denial of Service attack on Nova compute nodes (CVE-2013-4469). The provided fixes address both issues.

[Thierry Carrez (ttx)]

Patches test runs:

master: tox, smoke PASS
havana: tox, neutron PASS
grizzly: tox, full PASS

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/54767

[Thierry Carrez (ttx)]

master: https://review.openstack.org/54765
stable/havana: https://review.openstack.org/54767
stable/grizzly: https://review.openstack.org/54768

[Thierry Carrez (ttx)]

[OSSA 2013-029]

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/54765
Committed: http://github.com/openstack/nova/commit/f6810be4ae1a6c93e7d8017ee67d5344dfdf4a30
Submitter: Jenkins
Branch: master

commit f6810be4ae1a6c93e7d8017ee67d5344dfdf4a30
Author: Pádraig Brady <email address hidden>
Date: Fri Sep 27 04:07:14 2013 +0100

    ensure we don't boot oversized images

    Since we can't generally shrink incoming images, add extra checks
    to ensure oversized images are not allowed through.
    All cases when populating the libvirt image cache are now handled,
    including the initial download from glance, where we avoid
    converting to raw, as that could generate non sparse images
    much larger than the downloaded image.

    * nova/virt/libvirt/utils.py (fetch_image): Allow passing through
    of the max_size parameter.
    * nova/virt/images.py (fetch_to_raw): Accept the max_size parameter,
    and use it to discard images with larger (virtual) sizes.
    * nova/virt/libvirt/imagebackend.py (verify_base_size): A new
    refactored function to identify and raise exception to oversized images.
    (Raw.create_image): Pass the max_size to the fetch function.
    Also enforce virtual image size checking for already fetched images,
    as this class (despite the name) can be handling qcow files.
    (Qcow2.create_image): Pass the max_size to the fetch function,
    or verify the virtual size for the instance as done previously.
    (Lvm.create_image): Pass the max_size to the fetch function.
    Also check the size before transferring to the volume to improve
    efficiency by not even attempting the transfer of oversized images.
    (Rbd.create_image): Likewise.
    * nova/tests/virt/libvirt/fake_libvirt_utils.py: Support max_size arg.
    * nova/tests/virt/libvirt/test_libvirt.py (test_fetch_raw_image):
    Add a case to check oversized images are discarded.
    * nova/tests/virt/libvirt/test_imagebackend.py
    (test_create_image_too_small): Adjust to avoid the fetch size check.

    Fixes bug: 1177830
    Fixes bug: 1206081
    Change-Id: I3d47adaa2ad07434853f447feb27d7aae0e2e717

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/54767
Committed: http://github.com/openstack/nova/commit/3cdfe894ab58f7b91bf7fb690fc5bc724e44066f
Submitter: Jenkins
Branch: stable/havana

commit 3cdfe894ab58f7b91bf7fb690fc5bc724e44066f
Author: Pádraig Brady <email address hidden>
Date: Fri Sep 27 04:07:14 2013 +0100

    ensure we don't boot oversized images

    Since we can't generally shrink incoming images, add extra checks
    to ensure oversized images are not allowed through.
    All cases when populating the libvirt image cache are now handled,
    including the initial download from glance, where we avoid
    converting to raw, as that could generate non sparse images
    much larger than the downloaded image.

    * nova/virt/libvirt/utils.py (fetch_image): Allow passing through
    of the max_size parameter.
    * nova/virt/images.py (fetch_to_raw): Accept the max_size parameter,
    and use it to discard images with larger (virtual) sizes.
    * nova/virt/libvirt/imagebackend.py (verify_base_size): A new
    refactored function to identify and raise exception to oversized images.
    (Raw.create_image): Pass the max_size to the fetch function.
    Also enforce virtual image size checking for already fetched images,
    as this class (despite the name) can be handling qcow files.
    (Qcow2.create_image): Pass the max_size to the fetch function,
    or verify the virtual size for the instance as done previously.
    (Lvm.create_image): Pass the max_size to the fetch function.
    Also check the size before transferring to the volume to improve
    efficiency by not even attempting the transfer of oversized images.
    (Rbd.create_image): Likewise.
    * nova/tests/virt/libvirt/fake_libvirt_utils.py: Support max_size arg.
    * nova/tests/virt/libvirt/test_libvirt.py (test_fetch_raw_image):
    Add a case to check oversized images are discarded.
    * nova/tests/virt/libvirt/test_imagebackend.py
    (test_create_image_too_small): Adjust to avoid the fetch size check.

    Fixes bug: 1177830
    Fixes bug: 1206081
    Change-Id: I3d47adaa2ad07434853f447feb27d7aae0e2e717

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/54768
Committed: http://github.com/openstack/nova/commit/135faa7b5d9855312bedc19e5e1ecebae34d3d18
Submitter: Jenkins
Branch: stable/grizzly

commit 135faa7b5d9855312bedc19e5e1ecebae34d3d18
Author: Pádraig Brady <email address hidden>
Date: Fri Sep 27 04:07:14 2013 +0100

    ensure we don't boot oversized images

    Since we can't generally shrink incoming images, add extra checks
    to ensure oversized images are not allowed through.
    All cases when populating the libvirt image cache are now handled,
    including the initial download from glance, where we avoid
    converting to raw, as that could generate non sparse images
    much larger than the downloaded image.

    * nova/virt/libvirt/utils.py (fetch_image): Allow passing through
    of the max_size parameter.
    * nova/virt/images.py (fetch_to_raw): Accept the max_size parameter,
    and use it to discard images with larger (virtual) sizes.
    * nova/virt/libvirt/imagebackend.py (verify_base_size): A new
    refactored function to identify and raise exception to oversized images.
    (Raw.create_image): Pass the max_size to the fetch function.
    Also enforce virtual image size checking for already fetched images,
    as this class (despite the name) can be handling qcow files.
    (Qcow2.create_image): Pass the max_size to the fetch function,
    or verify the virtual size for the instance as done previously.
    (Lvm.create_image): Pass the max_size to the fetch function.
    Also check the size before transferring to the volume to improve
    efficiency by not even attempting the transfer of oversized images.
    (Rbd.create_image): Likewise.
    * nova/tests/fake_libvirt_utils.py: Support max_size arg.
    * nova/tests/test_libvirt.py (test_fetch_raw_image):
    Add a case to check oversized images are discarded.
    * nova/tests/test_imagebackend.py (test_create_image_too_small):
    Adjust to avoid the fetch size check.

    Fixes bug: 1177830
    Fixes bug: 1206081

    Conflicts:

     nova/tests/test_imagebackend.py
     nova/virt/libvirt/imagebackend.py

    Change-Id: Idc35fce580be4f74e23883d1b4bea6475c3f6e30