[OSSA 2016-001] Unprivileged api user can access host data using instance snapshot (CVE-2015-7548)

[Matthew Booth]

There is a qcow2 format vulnerability in LibvirtDriver.snapshot. The impact is that on an affected system, an unprivileged api user can retrieve any file on the host readable by the nova user. This includes guest data of other instances on the same host, and credentials used by nova to access other services externally.

LibvirtDriver.snapshot does:

    source_format = libvirt_utils.get_disk_type(disk_path)
    ...
    snapshot_backend = self.image_backend.snapshot(instance,
        disk_path,
        image_type=source_format)
    ...
    snapshot_backend.snapshot_extract(out_path, image_format)

libvirt_utils.get_disk_type falls back to image inspection for disks which are not lvm, rbd or ploop, which means raw and qcow2 images.

The vulnerability only exists when a user can write to a raw volume which is later erroneously detected as qcow2. This means that the vulnerability is only present on systems using the libvirt driver which have defined use_cow_images=False in nova.conf. This is not the default, so by default nova is not vulnerable.

libvirt.utils.extract_snapshot() expects to be reading from an instance disk and writing to a temporary directory created by nova for storing snapshots before transferring them to glance. As nova directly creates this directory and its contents, the 'qemu-img convert' process does not need to run privileged. This means that the exposure is limited to files directly readable by the nova user.

Unfortunately, as is clear from the context this includes all instance data which, despite being owned by the qemu user, is world readable. Additionally, because the qemu-img process is executed by nova directly, it does not benefit from any confinement by libvirt. Specifically, SELinux is not likely to be a defence on a typical deployment.

I have tested this exploit on a Fedora 23 system running devstack as of 8th Dec 2015:

Ensure nova.conf contains use_cow_images = False in the DEFAULT section.

As an unprivileged api user, do:
$ nova boot --image cirros --flavor m1.tiny foo

Somewhere, run:
$ qemu-img create -f qcow2 -o backing_file=/etc/passwd bad.qcow2
Ensure bad.qcow2 is available in the foo instance.

Log into foo, and execute as root:
# dd if=bad.qcow2 of=/dev/vda conv=fsync

As an unprivileged api user, do:
$ nova image-create foo passwd
$ glance image-download <newly created image id> --file passwd

The unprivileged api now has the contents of /etc/passwd from the host locally.

Mitigations:

Nova is not vulnerable by default. The user must have configured use_cow_images=False.
Nova configurations using ceph or lvm for instance storage are not vulnerable.
An attacker must know the uuid of another user's instance in order to be able to access its data.

[John Garbutt (johngarbutt)]

Added ndipanov to help with this one

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

In the past, we've considered bugs requiring UUID guessing to be security hardening opportunities rather than reasonably exploitable vulnerabilities. If it's confirmed that you need to guess or otherwise obtain the UUID of another tenant's instance to take advantage of this, then the VMT likely won't issue a security advisory nor request a CVE assignment. Also, the reduced risk implies that we'd be better off working this bug through our normal public process rather than incurring the expense of a private embargo.

[Tristan Cacqueray (tristan-cacqueray)]

This seems like a similar issue of https://security.openstack.org/ossa/OSSA-2014-009.html

We did issue an advisory because nova user is able to read /etc/nova/nova.conf which usually contains sensitive information like services token.

Anyway, is this a regression or snapshot never used the disk info file to inspect file format ?

[Nikola Đipanov (ndipanov)]

This is very similar to https://bugs.launchpad.net/nova/+bug/1221190

The problematic line

https://github.com/openstack/nova/blob/b5890b3c36613919338f83c4f59225f424c99cb1/nova/virt/libvirt/driver.py#L1357

was introduced by this commit: https://review.openstack.org/#/c/13684/ which was merged more than 3 years ago, but looking at the surrounding code at the time, the problem was there even before the patch, since we'd just grab the image format from glance which is completely user controlled as well.

So this does not seem like a regression.

[Matthew Booth (mbooth-9)]

The bug is worse that I originally realised, as it also affects installations running on ceph, lvm and ploop. The only setup which is not affected is qcow2 on filesystem.

The issue with ceph, lvm and ploop is that their snapshot_extract does convert_image, which calls qemu-img convert. It specifies an output format but not an input format, which means that the input format is implictly auto-detected. It is worse still on lvm, because in that case qemu-img convert runs as root, so it can read any file or device on the host without restriction.

I have successfully exploited this on rbd.

[Matthew Booth (mbooth-9)]

Here's a patch which addresses the originally reported issue with raw on filesystem. However, it does not fix the problem. With this patch, the disk is correctly handled as raw, but then it calls Raw.snapshot_extract, which is also vulnerable as described above, as it doesn't explicitly specify input format.

So, I believe this patch is required, but another patch is also required. I will work on this second patch tomorrow.

[Nikola Đipanov (ndipanov)]

Just for clarity the problematic line for RBD is

https://github.com/openstack/nova/blob/b5890b3c36613919338f83c4f59225f424c99cb1/nova/virt/libvirt/imagebackend.py#L818

if you look at convert_image method it calls:

https://github.com/openstack/nova/blob/b5890b3c36613919338f83c4f59225f424c99cb1/nova/virt/images.py#L71

the call to qemu-image convert never gets passed an input file format (-f) which means it tries to guess it from the image.

Other formats Matt mentions call the same method and are exploitable for the same reason.

[Tristan Cacqueray (tristan-cacqueray)]

Thanks for the clarification, I've confirmed the OSSA task. Matthew, about ceph, lvm, ... does it also triggers even with use_cow_images = True ?

Here is a first draft for the impact description:

Title: Nova host data leak through snapshot
Reporter: Matthew Booth (Red Hat)
Products: Nova
Versions: <=2015.1.2, ==12.0.0

Description:
Matthew Booth from Red Hat reported a vulnerability in Nova instance snapshot. By overwriting the disk inside an instance with a malicious image and requesting a snapshot, an authenticated user would be able to read an arbitrary file from the compute host. Note that the host file needs to be readable by the libvirt/kvm context to be exposed, lvm image backend run as root user, other backends run as nova user. Only setups using libvirt to spawn instance, and having "use_cow_images = False" in Nova configuration are affected.

[Matthew Booth (mbooth-9)]

use_cow_images = True is only used by filesystem storage. Systems which set use_cow_images = True and use filesystem storage are not vulnerable. Everything else is vulnerable. For clarity:

nfs or local files, use_cow_images = True: not vulnerable
nfs or local files, use_cow_images = False: vulnerable
ceph, lvm, ploop: vulnerable

Essentially, anything which stores raw data is vulnerable.

[Matthew Booth (mbooth-9)]

The impact description isn't quite right, yet. The process which reads host data is executed directly by nova. It runs as the nova user, except for the lvm backend, which runs as root. libvirt/kvm is not involved, which means libvirt confinement isn't a protection here. As discussed above, use_cow_images = False is only relevant to filesystem storage (local/nfs).

How about:

Matthew Booth from Red Hat reported a vulnerability in Nova instance snapshot. By overwriting the disk inside an instance with a malicious image and requesting a snapshot, an authenticated user would be able to read an arbitrary file from the compute host. Note that the host file needs to be readable by the nova user to be exposed except when using lvm for instance storage, when all files readable by root are exposed. Only setups using libvirt to spawn instances are vulnerable. Of these, setups which use filesystem storage, and do not set "use_cow_images = False" in Nova configuration are not affected. Setups which use ceph, lvm, or ploop for instance storage, and setups which use filesystem storage with "use_cow_images = False" are all affected.

[Daniel Berrange (berrange)]

I've looked at the patch from comment #6 and I agree with the way Matt has fixed the find_disk/get_disk_type methods to be safe.

[Nikola Đipanov (ndipanov)]

Some comments on the patch from comment #6

@@ -1356,10 +1357,23 @@ class LibvirtDriver(driver.ComputeDriver):

         snapshot = self._image_api.get(context, image_id)

- disk_path = libvirt_utils.find_disk(virt_dom)
- source_format = libvirt_utils.get_disk_type(disk_path)
+ # source_format is an on-disk format
+ # source_type is a backend type
+ disk_path, source_format = libvirt_utils.find_disk(virt_dom)
+ source_type = libvirt_utils.get_disk_type_from_path(disk_path)

So the problem with using source_type derived from the file we find in disk_path is actually that we use it to instantiate the image backend class further down

https://github.com/openstack/nova/blob/b5890b3c36613919338f83c4f59225f424c99cb1/nova/virt/libvirt/driver.py#L1421-L1422

But we really do not need this information at all here, we should really just be calling

  snapshot_backend = self.image_backend.snapshot(instance, disk_path)

as the format will be taken care of by the resolve_driver_format() method of the backend which then looks at the stored format in the disk_info_path file.

I agree that this patch probably fixes the issue, but does it in a way that is slightly different from the mechanism for doing this that we already have in place - the resolve_driver_format method (and seems to require more code). We should go for the simplest/most confined solution if possible.

[Nikola Đipanov (ndipanov)]

As a side note: I wonder if we should have a separate bug and CVE for the convert_image issue. It seems that under the circumstances and considering the severity - it requires a proper analysis, which it may not get if we keep it all under an umbrella of a single bug.

[Daniel Berrange (berrange)]

IMHO it is only appropriate to use the resolve_drive_format() method in cases where we don't have access to the information from libvirt guest config. ie it if the guest does not already exist. If there is a libvirt guest XML present, we should consider that the canonical source of information about the disk format, because that reflects what format QEMU is actually running with. The Nova look-aside cache is just a backup data source IMHO, not the canonical source, so should be used as a last resort

[Matthew Booth (mbooth-9)]

Nikola,

I figured that while the 2 issues are technically separate, though, the exploit and impact is essentially identical. i.e. from a user and patching pov, they're the same thing. In particular, I'd worry about producing 2 very similar CVEs with the same impact and exploit, but with different sets of affected users. The potential outcome of that is that somebody might mix them up and not patch, whereas in practice almost everybody has to patch.

However, I'm not sufficiently familiar with the process to know whether 2 bugs or 1 handles this better. I'm happy to create a second bug for the convert_image issue if that's more useful. I'll continue treating them as 1 issue until somebody better informed tells me otherwise.

[Sahid Orentino (sahid-ferdjaoui)]

- return disk_path
+ if driver is not None:
+ format = driver.get('type')
+ # This is a legacy quirk of libvirt/xen. Everything else should
+ # report the on-disk format in type.
+ if format == 'aio':
+ format = 'raw'
+ else:
+ format = None
+ return (disk_path, format)

QEMU is running guest so I'm not sure we can be on case where format is None or if we are we should to stop process.

What about:

- return disk_path
+ if driver is None:
+ raise RuntimeError("Can't retrieve disk driver from guest's domain XML")
+ format = driver.get('type')
+ if format == 'aio':
+ # This is a legacy quirk of libvirt/xen. Everything else should
+ # report the on-disk format in type.
+ format = 'raw'
+ return (disk_path, format)

[Matthew Booth (mbooth-9)]

Sahid,

I wrote it the way I did because I don't know what's going on with lxc guests, and I wanted to minimise the chances of breaking them (I'm not going to test them). We're passing a mounted filesystem to libvirt for lxc guests, so libvirt doesn't know what the file format is of the underlying storage. That's not ideal, but it's a separate issue.

[Nikola Đipanov (ndipanov)]

So I think we have one more path that allows for a problem but affects only raw backend (as qcow is not affected and _live_snapshot is enabled only for file backed instances):

https://github.com/openstack/nova/blob/240df4285964d341239376d8ea355c1b3763007a/nova/virt/libvirt/driver.py#L1567-L1571

this creates an overlay on top of whatever is read from the image by calling libvirt_utils.get_disk_backing_file, and then image_create (in case of raw it could have been tampered with). The code then proceeds to call blockRebase and finally qemu-image convert to whatever was detected.

It would be good to try to reproduce this path as well as we'll want to fix the issue with the

[Matthew Booth (mbooth-9)]

So the live snapshot thing is definitely a bug. However, it *doesn't* result in leaking data from the host. I'm not 100% sure why, but the resulting file is still a qcow2 with a backing file. As we don't have this backing file, there's no data leak.

As I don't understand why it's not broken, I'd still be inclined to fix it, tbh. It's definitely not safe.

[Matthew Booth (mbooth-9)]

Apologies for the lack of commit messages. I have to run again and I don't have time to do them justice right now. The second patch addresses the convert_image bug. I have tested it against both filesystem/raw and ceph. The exploit no longer works on either setup with both patches in place.

[Tristan Cacqueray (tristan-cacqueray)]

The migration case is also worrisome, the exploit may be prevented by another bug or something... We should fix it in another patch (and maybe another bug-report) only if we are confident this is not exploitable. Otherwise I would suggest we fix all qemu-img convert trickery in a single advisory.

Here is another draft that only covers the snapshot case with the description proposed by Matthew:

Title: Nova host data leak through snapshot
Reporter: Matthew Booth (Red Hat)
Products: Nova
Versions: <=2015.1.2, ==12.0.0

Description:
Matthew Booth from Red Hat reported a vulnerability in Nova instance snapshot. By overwriting the disk inside an instance with a malicious image and requesting a snapshot, an authenticated user would be able to read an arbitrary file from the compute host. Note that the host file needs to be readable by the nova user to be exposed except when using lvm for instance storage, when all files readable by root are exposed. Only setups using libvirt to spawn instances are vulnerable. Of these, setups which use filesystem storage, and do not set "use_cow_images = False" in Nova configuration are not affected. Setups which use ceph, lvm, or ploop for instance storage, and setups which use filesystem storage with "use_cow_images = False" are all affected.

[Tristan Cacqueray (tristan-cacqueray)]

I meant live-snapshot, not migration. my bad!

[Matthew Booth (mbooth-9)]

I've uploaded 3 patches to address the 3 issues discussed in this bug, including the live-snapshot issue mentioned by nikola which we aren't 100% sure is usefully exploitable. They all now have proper commit messages. I have tested them against filesystem/raw and rbd locally, but I would be very grateful for review.

[Tristan Cacqueray (tristan-cacqueray)]

Thanks Matthew, nova-coresec, please review proposed patch.

I'd like to request a CVE now with the proposed impact description in #21 since it's good enough for both issues (snapshot and live-snapshot).

If patches are approved by Monday morning (before 1500UTC), this could be disclosed the following Thursday:
2015-12-17, 1500 UTC

If patches are approved by Thursday, this would need to wait another week and could be disclosed the following Tuesday:
2015-12-22, 1500 UTC

This second date is not really ideal since it's dangerously close to Christmas holidays...

[Daniel Berrange (berrange)]

The 3 new patches get my +2.

[Nikola Đipanov (ndipanov)]

The only improvement I'd make in patche #2 would be to make sure that we never call an unsafe convert_image outside of that one case where we have to do it, by making that a separate method we name _unsafe

Otherwise lgtm - I think sahid was working on some backports. How do the tests look?

[Jeremy Stanley (fungi)]

Tristan's impact description in comment #21 looks good to me.

The rules to determine whether your deployment is impacted are somewhat convoluted and it took me a few times reading through to grok. Nevertheless I can't think of any easier way to state that, and hopefully downstream consumers will just apply the patch regardless.

[Matthew Booth (mbooth-9)]

I would make 1 change to the impact statement: I no longer believe ploop is affected, although I still haven't tested it and I'm not going to. So affected setups are:

* filesystem + use_cow_images = False
* LVM
* RBD

I would remove mention of ploop from the impact statement.

[Sahid Orentino (sahid-ferdjaoui)]

> I think sahid was working on some backports. How do the tests look?
All patches are passing tests.

If patches are acked by monday morning I could to propose them backported for liberty, kilo, juno, icehouse by end of day.

[Nikola Đipanov (ndipanov)]

Actually come to think of it we could use the same trick for patch #3 qemu-img info call - but seeing that there are more call sites, and it's not as critical, we can do that in a separate patch(es)

[Matthew Booth (mbooth-9)]

This is an updated patch #2, as per nikola's comments.

[Sahid Orentino (sahid-ferdjaoui)]

stable/liberty (1, 2, 3) are backports, please take time to review them. Thanks.

[Sahid Orentino (sahid-ferdjaoui)]

stable/kilo (1, 2, 3) are backports please take time to review them. Thanks

[Matthew Booth (mbooth-9)]

Backports look good to me. I've inspected them, and run pep8 and unit tests against all of them individually (kilo and liberty). I haven't tested them in a complete working system, though, or run tempest against them.

The only comment I would make is that ideally the cherry-pick line would either refer to an actual upstream commit, or be omitted. Right now it refers to something in your local tree. It might be simpler just to omit it, or to frig the commit messages after it actually lands on upstream master. However, this is a minor point, and shouldn't hold anything up.

[Matthew Booth (mbooth-9)]

Taking into account the upcoming holiday period, we would like to propose an embargoed disclosure date of Monday 4th January, with a public disclosure date of Thursday 7th January. Does that suit?

[Jeremy Stanley (fungi)]

I'm okay with the bug reporter requesting a delay in disclosure. Under the current calendar circumstances, this seems like a reasonable timeline.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/264812

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/264813

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/264814

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/264815

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/264816

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/264817

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/264819

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/264820

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/264821

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264812
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2bb6635a21e4f7abe386c01ec7d3af5480498f14
Submitter: Jenkins
Branch: master

commit 2bb6635a21e4f7abe386c01ec7d3af5480498f14
Author: Matthew Booth <email address hidden>
Date: Wed Dec 9 15:36:32 2015 +0000

    Fix format detection in libvirt snapshot

    The libvirt driver was using automatic format detection during
    snapshot for disks stored on the local filesystem. This opened an
    exploit if nova was configured to use local file storage, and
    additionally to store those files in raw format by specifying
    use_cow_images = False in nova.conf. An authenticated user could write
    a qcow2 header to their guest image with a backing file on the host.
    libvirt.utils.get_disk_type() would then misdetect the type of this
    image as qcow2 and pass this to the Qcow2 image backend, whose
    snapshot_extract method interprets the image as qcow2 and writes the
    backing file to glance. The authenticated user can then download the
    host file from glance.

    This patch makes 2 principal changes. libvirt.utils.get_disk_type,
    which ought to be removed entirely as soon as possible, is updated to
    no longer do format detection if the format can't be determined from
    the path. Its name is changed to get_disk_type_from_path to reflect
    its actual function.

    libvirt.utils.find_disk is updated to return both the path and format
    of the root disk, rather than just the path. This is the most reliable
    source of this information, as it reflects the actual format in use.
    The previous format detection function of get_disk_type is replaced by
    the format taken from libvirt.

    We replace a call to get_disk_type in _rebase_with_qemu_img with an
    explicit call to qemu_img_info, as the other behaviour of
    get_disk_type was not relevant in this context. qemu_img_info is safe
    from the backing file exploit when called on a file known to be a
    qcow2 image. As the file in this context is a volume snapshot, this is
    a safe use.

    Partial-Bug: #1524274

    Change-Id: I94c1c0d26215c061f71c3f95e1a6bf3a58fa19ea

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264813
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=820430b62aa5e87a8f7750f1b9873f56c4822452
Submitter: Jenkins
Branch: master

commit 820430b62aa5e87a8f7750f1b9873f56c4822452
Author: Matthew Booth <email address hidden>
Date: Thu Dec 10 16:34:19 2015 +0000

    Fix format conversion in libvirt snapshot

    The libvirt driver was calling images.convert_image during snapshot to
    convert snapshots to the intended output format. However, this
    function does not take the input format as an argument, meaning it
    implicitly does format detection. This opened an exploit for setups
    using raw storage on the backend, including raw on filesystem, LVM,
    and RBD (Ceph). An authenticated user could write a qcow2 header to
    their instance's disk which specified an arbitrary backing file on the
    host. When convert_image ran during snapshot, this would then write
    the contents of the backing file to glance, which is then available to
    the user. If the setup uses an LVM backend this conversion runs as
    root, meaning the user can exfiltrate any file on the host, including
    raw disks.

    This change adds an input format to convert_image.

    Partial-Bug: #1524274

    Change-Id: If73e73718ecd5db262ed9904091024238f98dbc0

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/267178

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264815
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cd4eeead3d70719f04d940af7da608eec81ab4e8
Submitter: Jenkins
Branch: stable/liberty

commit cd4eeead3d70719f04d940af7da608eec81ab4e8
Author: Matthew Booth <email address hidden>
Date: Wed Dec 9 15:36:32 2015 +0000

    Fix format detection in libvirt snapshot

    The libvirt driver was using automatic format detection during
    snapshot for disks stored on the local filesystem. This opened an
    exploit if nova was configured to use local file storage, and
    additionally to store those files in raw format by specifying
    use_cow_images = False in nova.conf. An authenticated user could write
    a qcow2 header to their guest image with a backing file on the host.
    libvirt.utils.get_disk_type() would then misdetect the type of this
    image as qcow2 and pass this to the Qcow2 image backend, whose
    snapshot_extract method interprets the image as qcow2 and writes the
    backing file to glance. The authenticated user can then download the
    host file from glance.

    This patch makes 2 principal changes. libvirt.utils.get_disk_type,
    which ought to be removed entirely as soon as possible, is updated to
    no longer do format detection if the format can't be determined from
    the path. Its name is changed to get_disk_type_from_path to reflect
    its actual function.

    libvirt.utils.find_disk is updated to return both the path and format
    of the root disk, rather than just the path. This is the most reliable
    source of this information, as it reflects the actual format in use.
    The previous format detection function of get_disk_type is replaced by
    the format taken from libvirt.

    We replace a call to get_disk_type in _rebase_with_qemu_img with an
    explicit call to qemu_img_info, as the other behaviour of
    get_disk_type was not relevant in this context. qemu_img_info is safe
    from the backing file exploit when called on a file known to be a
    qcow2 image. As the file in this context is a volume snapshot, this is
    a safe use.

    Partial-Bug: #1524274

    (cherry picked from commit 2bb6635a21e4f7abe386c01ec7d3af5480498f14)

    Conflicts:
        nova/tests/unit/virt/libvirt/test_utils.py
        nova/virt/libvirt/utils.py

    Minor conflicts as ploop was added in Mitaka.

    Change-Id: I94c1c0d26215c061f71c3f95e1a6bf3a58fa19ea

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264816
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a9c8dee7ec81d6331c3699a2dda3a57b48ecfe5c
Submitter: Jenkins
Branch: stable/liberty

commit a9c8dee7ec81d6331c3699a2dda3a57b48ecfe5c
Author: Matthew Booth <email address hidden>
Date: Thu Dec 10 16:34:19 2015 +0000

    Fix format conversion in libvirt snapshot

    The libvirt driver was calling images.convert_image during snapshot to
    convert snapshots to the intended output format. However, this
    function does not take the input format as an argument, meaning it
    implicitly does format detection. This opened an exploit for setups
    using raw storage on the backend, including raw on filesystem, LVM,
    and RBD (Ceph). An authenticated user could write a qcow2 header to
    their instance's disk which specified an arbitrary backing file on the
    host. When convert_image ran during snapshot, this would then write
    the contents of the backing file to glance, which is then available to
    the user. If the setup uses an LVM backend this conversion runs as
    root, meaning the user can exfiltrate any file on the host, including
    raw disks.

    This change adds an input format to convert_image.

    Partial-Bug: #1524274

    (cherry picked from commit 820430b62aa5e87a8f7750f1b9873f56c4822452)

    Change-Id: If73e73718ecd5db262ed9904091024238f98dbc0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264814
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=915fdbbfb82272b87cd80210943372b09351cf88
Submitter: Jenkins
Branch: master

commit 915fdbbfb82272b87cd80210943372b09351cf88
Author: Matthew Booth <email address hidden>
Date: Fri Dec 11 13:40:54 2015 +0000

    Fix backing file detection in libvirt live snapshot

    When doing a live snapshot, the libvirt driver creates an intermediate
    qcow2 file with the same backing file as the original disk. However,
    it calls qemu-img info without specifying the input format explicitly.
    An authenticated user can write data to a raw disk which will cause
    this code to misinterpret the disk as a qcow2 file with a
    user-specified backing file on the host, and return an arbitrary host
    file as the backing file.

    This bug does not appear to result in a data leak in this case, but
    this is hard to verify. It certainly results in corrupt output.

    Closes-Bug: #1524274

    Change-Id: I11485f077d28f4e97529a691e55e3e3c0bea8872

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264817
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c9c0b1b4b5acc1ea3c2bbb694aa79d7e4a7087e3
Submitter: Jenkins
Branch: stable/liberty

commit c9c0b1b4b5acc1ea3c2bbb694aa79d7e4a7087e3
Author: Matthew Booth <email address hidden>
Date: Fri Dec 11 13:40:54 2015 +0000

    Fix backing file detection in libvirt live snapshot

    When doing a live snapshot, the libvirt driver creates an intermediate
    qcow2 file with the same backing file as the original disk. However,
    it calls qemu-img info without specifying the input format explicitly.
    An authenticated user can write data to a raw disk which will cause
    this code to misinterpret the disk as a qcow2 file with a
    user-specified backing file on the host, and return an arbitrary host
    file as the backing file.

    This bug does not appear to result in a data leak in this case, but
    this is hard to verify. It certainly results in corrupt output.

    Closes-Bug: #1524274

    (cherry picked from commit 915fdbbfb82272b87cd80210943372b09351cf88)

    Conflicts:
        nova/virt/images.py

    Liberty does not catch processutils.ProcessExecutionError in
    qemu_img_info.

    Change-Id: I11485f077d28f4e97529a691e55e3e3c0bea8872

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/267178
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b2acc9fa864b6fe10bc0c5f3786b976b472b1b27
Submitter: Jenkins
Branch: stable/liberty

commit b2acc9fa864b6fe10bc0c5f3786b976b472b1b27
Author: Matt Riedemann <email address hidden>
Date: Wed Jan 13 09:41:57 2016 -0800

    Add security fixes to the release notes for 12.0.1

    There are two security fixes in the upcoming stable/liberty
    point release (12.0.1) so we should document those in the
    release notes.

    This shouldn't be merged until I11485f077d28f4e97529a691e55e3e3c0bea8872
    is merged. We don't use Depends-On here since we don't want to hold this
    up on the stable/kilo backport of that change.

    Change-Id: Ib2235d6aee540d9010dc86c73aff71179d46e921
    Related-Bug: #1516765
    Related-Bug: #1524274

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264819
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7534182e7f1ae466f96fdb13d3715a458f300604
Submitter: Jenkins
Branch: stable/kilo

commit 7534182e7f1ae466f96fdb13d3715a458f300604
Author: Matthew Booth <email address hidden>
Date: Wed Dec 9 15:36:32 2015 +0000

    Fix format detection in libvirt snapshot

    The libvirt driver was using automatic format detection during
    snapshot for disks stored on the local filesystem. This opened an
    exploit if nova was configured to use local file storage, and
    additionally to store those files in raw format by specifying
    use_cow_images = False in nova.conf. An authenticated user could write
    a qcow2 header to their guest image with a backing file on the host.
    libvirt.utils.get_disk_type() would then misdetect the type of this
    image as qcow2 and pass this to the Qcow2 image backend, whose
    snapshot_extract method interprets the image as qcow2 and writes the
    backing file to glance. The authenticated user can then download the
    host file from glance.

    This patch makes 2 principal changes. libvirt.utils.get_disk_type,
    which ought to be removed entirely as soon as possible, is updated to
    no longer do format detection if the format can't be determined from
    the path. Its name is changed to get_disk_type_from_path to reflect
    its actual function.

    libvirt.utils.find_disk is updated to return both the path and format
    of the root disk, rather than just the path. This is the most reliable
    source of this information, as it reflects the actual format in use.
    The previous format detection function of get_disk_type is replaced by
    the format taken from libvirt.

    We replace a call to get_disk_type in _rebase_with_qemu_img with an
    explicit call to qemu_img_info, as the other behaviour of
    get_disk_type was not relevant in this context. qemu_img_info is safe
    from the backing file exploit when called on a file known to be a
    qcow2 image. As the file in this context is a volume snapshot, this is
    a safe use.

    Partial-Bug: #1524274

    Conflicts:
        nova/tests/unit/virt/libvirt/test_driver.py
        nova/tests/unit/virt/libvirt/test_utils.py
        nova/virt/libvirt/driver.py
        nova/virt/libvirt/utils.py

    Change-Id: I94c1c0d26215c061f71c3f95e1a6bf3a58fa19ea

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264820
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3bc1295e2d5fb1376e3a5c708efe9fbf9435e6a4
Submitter: Jenkins
Branch: stable/kilo

commit 3bc1295e2d5fb1376e3a5c708efe9fbf9435e6a4
Author: Matthew Booth <email address hidden>
Date: Thu Dec 10 16:34:19 2015 +0000

    Fix format conversion in libvirt snapshot

    The libvirt driver was calling images.convert_image during snapshot to
    convert snapshots to the intended output format. However, this
    function does not take the input format as an argument, meaning it
    implicitly does format detection. This opened an exploit for setups
    using raw storage on the backend, including raw on filesystem, LVM,
    and RBD (Ceph). An authenticated user could write a qcow2 header to
    their instance's disk which specified an arbitrary backing file on the
    host. When convert_image ran during snapshot, this would then write
    the contents of the backing file to glance, which is then available to
    the user. If the setup uses an LVM backend this conversion runs as
    root, meaning the user can exfiltrate any file on the host, including
    raw disks.

    This change adds an input format to convert_image.

    Partial-Bug: #1524274

    Change-Id: If73e73718ecd5db262ed9904091024238f98dbc0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264821
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b974c6d1d5753f333d1d71f8190ddf3b4f8fbbf1
Submitter: Jenkins
Branch: stable/kilo

commit b974c6d1d5753f333d1d71f8190ddf3b4f8fbbf1
Author: Matthew Booth <email address hidden>
Date: Fri Dec 11 13:40:54 2015 +0000

    Fix backing file detection in libvirt live snapshot

    When doing a live snapshot, the libvirt driver creates an intermediate
    qcow2 file with the same backing file as the original disk. However,
    it calls qemu-img info without specifying the input format explicitly.
    An authenticated user can write data to a raw disk which will cause
    this code to misinterpret the disk as a qcow2 file with a
    user-specified backing file on the host, and return an arbitrary host
    file as the backing file.

    This bug does not appear to result in a data leak in this case, but
    this is hard to verify. It certainly results in corrupt output.

    Closes-Bug: #1524274

    Change-Id: I11485f077d28f4e97529a691e55e3e3c0bea8872

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 12.0.1 release.

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/nova 13.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/291208

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/291208
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9c0bbda07fdcf134308371644d09becbb18c62b1
Submitter: Jenkins
Branch: master

commit 9c0bbda07fdcf134308371644d09becbb18c62b1
Author: Matt Riedemann <email address hidden>
Date: Thu Mar 10 09:35:00 2016 -0500

    Add release notes for security fixes in 13.0.0 mitaka GA

    There are three security issues fixed in mitaka.

    The first two were documented for liberty 12.0.1 but we
    apparently forgot to doc them for mitaka.

    Related-Bug: #1524274
    Related-Bug: #1516765
    Related-Bug: #1548450

    Change-Id: I3eba75f1fc86c4c9abd258042dfafc6df1f2405c

[Hao Jun Wang (shhjwang)]

I am sorry that I am not clear reproducing process. Anyone can help me? Thanks.

I have set use_cow_images = False in nova.conf and restarted services.
Then I executed the steps in the description and file "passwd" generated.

As the description says, the contents of /etc/passwd should be in the passwd file. Since the passwd file is a binary file, I tried to use "grep" to search the contents of /etc/passwd in passwd file. But I found nothing. And I also tried "grep" command on bad.qcow2. Also nothing was found. In the instance, I executed "sudo strings /dev/vda | head -1" and it shows "/etc/passwd". So the "dd" operation should have worked. But I don't know how can I show the contents of "/etc/passwd" in downloaded passwd file.

Anyone has some advice? Thanks very much.