[OSSA-2023-002] Arbitrary file access through custom VMDK flat descriptor (CVE-2022-47951)

Bug #1996188 reported by Jeremy Stanley
304
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Critical
Unassigned
Glance
Fix Released
Undecided
Dan Smith
OpenStack Compute (nova)
Fix Released
Critical
Unassigned
OpenStack Security Advisory
Fix Released
High
Jeremy Stanley

Bug Description

The vulnerability managers received the following report from Sébastien Meriot with OVH via encrypted E-mail:

Our Openstack team did discover what looks like a security issue in Nova this morning allowing a remote attacker to read any file on the system.
After making a quick CVSS calculation, we got a CVSS of 5.8 (CVSS:3.0/AV:N/AC:H/PR:L/UI:R/S:C/C:H/I:N/A:N).

Here is the details :
By using a VMDK file, you can dump any file on the hypervisor.
1. Create an image: qemu-img create -f vmdk leak.vmdk 1M -o subformat=monolithicFlat
2. Edit the leak.vmdk and change the name this way: RW 2048 FLAT "leak-flat.vmdk" 0 --> RW 2048 FLAT "/etc/nova/nova.conf" 0
3. Upload the image: openstack image create --file leak.vmdk leak.vmdk
4. Start a new instance: openstack server create --image leak.vmdk --net demo --flavor nano leak-instance
5. The instance won't boot of course. You can create an image from this instance: openstack server image create --name leak-instance-image leak-instance
6. Download the image: openstack image save --file leak-instance-image leak-instance-image
7. You get access to the nova.conf file content and you can get access to the openstack admin creds.

We are working on a fix and would be happy to share it with you if needed.
We think it does affect Nova but it could affect Glance as well. We're not sure yet.

[postscript per Arnaud Morin (amorin) in IRC]

cinder seems also affected

CVE References

Revision history for this message
Jeremy Stanley (fungi) wrote :

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

I've started with just Nova even though the reporters mention Glance and Cinder, since it's unclear (at least to me) precisely which project(s) could need fixes. We can widen the discussion to include those teams after initial triage from Nova's security folks indicates it's warranted.

Changed in ossa:
status: New → Incomplete
Revision history for this message
Dan Smith (danms) wrote :

Here's a reproducer of the underlying operation without the need for any nova or glance:

qemu-img create -f vmdk leak.vmdk 1M -o subformat=monolithicFlat
sed -i 's#leak-flat.vmdk#/etc/hosts#' leak.vmdk
qemu-img convert -f vmdk -O raw leak.vmdk leak.raw
head -n1 leak.raw

Note that I can repro the behavior with qemu-img using qcow2 as well:

qemu-img create -f qcow2 -F raw -b /etc/hosts leak.qcow 1M
qemu-img convert -O raw leak.qcow leak2.raw
head -n1 leak2.raw

Which means even people that don't use vmdk can't just ban that format to work around this, I suspect.

This helps quantify what I think is going on here under the covers so we can determine which other projects are affected. Glance does effectively the above commands if image conversion is enabled, so I suspect it is affected without needing nova at all, if and when image conversion is in use. I'll have to check.

Not sure about cinder, but I suspect if it can lay down a COW-based image on a volume in raw format, it too is probably affected.

I'm guessing that maybe we'll need to come up with some difficult rules about backing file locations and inspect/reject based on those. I would say that nova requiring that backing files are in /var/lib/nova would be enough, but that wouldn't prevent me from snooping someone else's image if I could determine their UUID.

Revision history for this message
Guillaume Espanel (guillaume-espanel) wrote :

About qcow2:

It seems that nova refuses to convert qcow2 images that have a backing file, at least here:
https://opendev.org/openstack/nova/src/commit/c97507dfcd57cce9d76670d3b0d48538900c00e9/nova/virt/images.py#L127

Same for cinder: https://opendev.org/openstack/cinder/src/commit/dcec2f6f01ffc63dc058f641370e9f5a0bad07e4/cinder/image/image_utils.py#L659

I think glance also rejects qcow2 that have backing files:
https://opendev.org/openstack/glance/src/branch/master/glance/async_/flows/base_import.py#L178
but I am not sure that's the case everywhere.

Revision history for this message
Dan Smith (danms) wrote :

Cool, on nova and cinder for qcow2. Glance would need to check that during/before the image format conversion, because the code you linked there would happen after we've already converted it to raw, I think. I don't see such a check here:

https://opendev.org/openstack/glance/src/branch/master/glance/async_/flows/plugins/image_conversion.py

That is an optional plugin, requires being enabled, and only impacts images being imported (not uploaded). So, a much smaller surface.

Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Could we clarify the attacking surface, please ?
From what I see, the nova libvirt driver calls fetch_image() which does the backing file check *before* converting the image, so could we confirm that all environments having compute services configured to use the libvirt driver *aren't* impacted ?

For other convert operations that could be run by Glance, Cinder or nova-computes not using the libvirt driver, I tho totally agree : yes, we can dump any file.

Revision history for this message
Dan Smith (danms) wrote :

The backing file check in fetch_image() only applies to qcow2, not vmdk. If we make it to there with a VMDK file, the backing_file check won't apply and we'll do the bad thing during the raw conversion.

That said, I just realized that our image type support for libvirt does not declare that we support VMDK. That means *if* you have that feature enabled, we won't ever send instances to libvirt computes if a VMDK image is used. That further limits the scope of impacted people (and versions), but I think if you have that disabled, you'll still get there.

I assume the reporter was using the libvirt driver but I suppose it's worth checking.

Either way, I'm testing a patch to detect/reject this in nova now.

Revision history for this message
Dan Smith (danms) wrote :

Okay, I confirmed that I can hit the libvirt compute node with a vmdk file, using the default config, and specifically the bad format image specified in the bug description. I'm attaching a proposed patch for nova which restricts the allowed VMDK types to just the two we think are usable with nova/glance anyway. However, it adds a config option (as discussed) to allow overriding or eliminating this check to avoid breaking people that are successfully using another subtype without a way to work around it.

With this patch applied and the above-mentioned maliciously-crafted image, I get a failed server build and this log message:

Nov 10 10:35:18 ubuntu nova-compute[119048]: WARNING nova.virt.images [None req-1cd34d51-ed13-488b-90eb-27c135e8bf0f demo admin] Refusing to process VMDK file with create-type of 'monolithicFlat' which is not in allowed set of: streamOptimized,monolithicSparse

Revision history for this message
Arnaud Morin (arnaud-morin) wrote :

Hello,
I confirme that de are using the libvirt driver.

Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

Code LGTM. I do have a concern about the config opt, namely, that the default is

  vmdk_allowed_types = ['streamOptimized', 'monolithicSparse']

and if you set it like this:

  vmdk_allowed_types = [ ]

The behavior is "allow all" ... which the help text mentions, but I think this is very easy to mis-configure, because if you only want 'streamOptimized' you remove 'monolithicSparse' from the list, and following that logic, if you don't want any, you'll remove 'streamOptimized' too ... but the behavior will be the exact opposite of what you expect. So I'd prefer an empty list mean "don't allow any".

The downside is that if someone does want to allow all possible formats, they'll have to list them all. Or we could introduce a sentinel value like 'vmdk_allowed_types_ANY'

  vmdk_allowed_types = ['vmdk_allowed_types_ANY']

and hope that vmware doesn't introduce a 'vmdk_allowed_types_ANY' subformat.

But the other upside of [] == "none" is that use of vmdk can be turned off in Nova without having to depend on the disk_formats setting in Glance. It would also handle the problem of existing vmdks in deployments that don't actually use vmdk, but are using the default disk_formats value in Glance (which includes vmdk).

Revision history for this message
Dan Smith (danms) wrote :

Yep, I was going for the minimal amount of work for someone to get back to the existing behavior if this ends up cutting off a workflow for them.

I don't love the magic sentinel approach, and would prefer to just go with "empty means nothing is enabled" and make people enable what they need for now. If there's a big push-back we can work on a sentinel to disable it entirely, but in reality I expect not many people actually want to boot from VMDKs with the libvirt driver.

If there are other sane formats to add to the default list, then hopefully they'll report them too.

I'll let other people comment on the patch with the assumption that we'll flip the meaning of an empty list and will update it here after that (or after nobody has other comments).

Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

This patch looks good to me as well at least for the nova libvirt driver.

Revision history for this message
Arnaud Morin (arnaud-morin) wrote :

The patch looks good. Are we sure we are always going through the fetch_to_raw function?

I'll try to take some time to test it on our infra and let you know.

Thanks

Revision history for this message
Guillaume Espanel (guillaume-espanel) wrote :

Looks good to me too, for nova. I looked a bit yesterday that the two variants we plan to allow cannot be tweaked to leak data through qemu-img convert and they seem fine, at least with the simple path change on the extent entry.

Speaking of fetch_to_raw, what I understand is that we let qemu-img info tell us what format the image is in before converting:
https://opendev.org/openstack/nova/src/commit/2a73a1db84da15a24231f7219f6c4a4ea574bae6/nova/virt/images.py#L120

I am not sure we should be doing that (see the comment in convert_image_unsafe).

Revision history for this message
Dan Smith (danms) wrote :

You know, I really thought we did assert the source type of the image when we fetch it. But looking through all of that, at least as it exists today, I see that we're not. Since the previous requirement for doing that was to protect against the qcow2 backing_file attack, the comments referencing fetch_to_raw() as some sort of safety measure must be referencing the backing_file check itself.

Unless we're aware of other attacks that can be made by claiming an image is one thing when probing disagrees, we're probably okay for the scope of this bug. My code checks the vmdk rules if we probe it as such, so we won't leak there, even if someone claims it is something else.

I can imagine glance having an enforcement mode that requires images to detect as what they claim to be (it has the infrastructure to make such an enforcement today). Nova could also have a flag that verifies the image's disk_format agrees with the probe in fetch_to_raw().

Revision history for this message
Guillaume Espanel (guillaume-espanel) wrote :

I agree, I think we're good for VMDK and I might be worrying too early:

I haven't tested it yet but reading the code it looks like we could convert other less-used formats to raw (qed, or qcow for example). My worry is that qed, for example, is no longer developed and I imagine hasn't been as scrutinized as qcow2 (or vmdk, for that matter).

To be on the safe side, I would further restrict the formats we accept to convert from, but I am not sure to what extend that is feasible.

Revision history for this message
Dan Smith (danms) wrote :

Yes, I think a list of acceptable formats in nova would be a good idea, and a flag in glance to require that an image detects as the format claimed in disk_format would be good as well.

Revision history for this message
Dan Smith (danms) wrote :

Okay, here's the updated nova patch which considers the empty list to mean "none allowed".

Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

This affects cinder, too. Attaching a patch based on Dan's nova fix.

1 comments hidden view all 189 comments
Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

I didn't add Glance because the situation isn't so dire. I believe that an operator must configure the optional image conversion plugin for image import in order for this exploit to affect Glance. That said, I think we should fix it in Glance, too, but waiting to hear what others think before adding Glance to this bug.

Revision history for this message
Arnaud Morin (arnaud-morin) wrote :

Hello, thanks for the patch, I think we need to include glance in that bug as well.

Revision history for this message
Guillaume Espanel (guillaume-espanel) wrote :

About Cinder, I think we should put the check_vmdk_format as close as possible the call to convert.

First, I think we may be missing some convert_images. I cannot test it, but I have come across this where we seem to convert an image into a volume directly: https://opendev.org/openstack/cinder/src/branch/master/cinder/volume/drivers/netapp/dataontap/nfs_base.py#L767

Second, I noted that, in the case of AMI image format, we let qemu-img convert detect the real format of the image:
https://opendev.org/openstack/cinder/src/branch/master/cinder/image/image_utils.py#L277
I think this is caught as part of the proposed patch, but there could be similar constructions elsewhere.

Third, in fetch_to_volume_format, we first check the VMDK createType if qemu-img info detects a VMDK, but we call the convert with the format declared in the image metadata.

Importantly, I was able to trick qemu-img info into detecting a "raw" image than can actually be converted from VMDK to raw by simply adding an empty line at the beginning of the file, thus (if I understand correctly) bypassing the check entirely.

Here's a short reproducer, note the empty line before # Disk DescriptorFile :

cat > test.raw << EOF

# Disk DescriptorFile
version=1
CID=86cc8022
parentCID=ffffffff
createType="monolithicFlat"

# Extent description
RW 2048 FLAT "/etc/hosts" 0

# The Disk Data Base
#DDB

ddb.virtualHWVersion = "4"
ddb.geometry.cylinders = "2"
ddb.geometry.heads = "16"
ddb.geometry.sectors = "63"
ddb.adapterType = "ide"
EOF

qemu-img info test.raw
image: test.raw
file format: raw
virtual size: 512 B (512 bytes)
disk size: 4 KiB

qemu-img convert -f vmdk -O raw test.raw hacked.raw

I'll look again at the nova patch to make sure we account for that funny discovery there too.

About glance, I think we should also patch it as part of this effort, even though it's not as bad given the fact the exploit is unavailable by default.

Revision history for this message
Dan Smith (danms) wrote :

It's not available by default in glance - you have to enable the image_conversion feature (which enforces that all images get converted to a single specific format). That said, I do also think we should include glance here, especially since I found that the same code is also unpatched against the long-fixed-in-nova/cinder qcow backing_file attack. Host file exposure to unprivileged users seems like a large enough impact to justify fixing it ASAP.

This patch fixes the vmdk thing just like cinder and nova, and also fixes the similar qemu vulnerability as well.

Perhaps we could loop in another glance person (say Abhi) for his opinion on the patch and including glance specifically. If so, then would just be a formality.

Revision history for this message
Jeremy Stanley (fungi) wrote :

Yes, a number of public clouds I interact with apply the image conversion task to uploads, so while it may be optional I expect the impact is still fairly widespread.

Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

Latest version of cinder patch hardens the check function a bit and adds another test; still looking into what Guillaume mentioned on his earlier comment

Revision history for this message
Guillaume Espanel (guillaume-espanel) wrote :

I keep thinking we should move the check_vmdk_image (and possibly duplicate the image inspection) as close as possible to the actual convert.

Hopefully exhaustive list of places where we I think we should perform the check:

- For nova, in virt.images._convert_image()
- For glance, in async_.flows.convert._Convert.execute() and async_.flows.plugins.image_conversion._ConvertImage._execute()
- For cinder, in image.image_utils._convert_image()

In addition, I think we should never run qemu-img convert without a "-f something" argument. We are doing this at least for AMIs (they can be either qcow2 or raw IIRC) and LVM. In such cases I suggest we run qemu-img info to determine the format of the source and pass the inspected format to convert (after running the checks), instead of letting qemu-img convert detect it, thus bypassing the checks.

Revision history for this message
Dan Smith (danms) wrote :

Why do you think that? Doing checks upfront in fetch_image_to_raw (for the nova case) makes the most sense to me, and it matches where we do the check for the qcow backing file. IMHO, it's best to just check and reject immediately, before we do any other inspection or potential work.

IMHO, convert_image() is an "internal" method that just does what we ask it to, and it already requires source and destination formats. We need to be suspicious of images when we pull the from glance, but after we've internalized them I don't think we need to do that check every time we're handling an image.

Revision history for this message
Guillaume Espanel (guillaume-espanel) wrote :

Focusing on nova, we seem to be going through fetch_to_raw most of the time.
I am a bit suspicious of the convert_image_unsafe because it is not entirely clear to me how we end-up calling it, but if we believe the comments, it should either run on a fetch'd_to_raw image, or on a locally generated image. If we are confident this is true and is likely to remain true, we're probably good.

Regarding glance, I think we are already following the right approach, but unless I am missing something, we have to add the same patch you did in async_.flows.plugins.image_conversion._ConvertImage._execute() in async_.flows.convert._Convert.execute().

Finally, in cinder, convert_image has callers in many different places (in the drivers for example), and it could be easy to miss an unsafe call there.

Overall, I agree it would be nice to check and reject early, but for me it'd be nicer to provide convert_image functions that are safe (at least from this bug) by default. I could imagine, sometime in the future, someone using one of these convert_image functions without checking their VMDK is of this or that createType.

Revision history for this message
Dan Smith (danms) wrote :

Cool on Nova. On Glance, TBH, I'm not sure how users access the other flow. The one I patched is used on import, but I'm not sure how those legacy plugins ever get called. Perhaps Brian can comment. Certainly it's possible to just patch that the same way, but we should be deliberate about it.

Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

@Dan: that's a good question about the glance legacy plugins. I don't remember why Erno needed to introduce new ones for the Interoperable Image Import workflow. They're left over from the old tasks API, which I guess can still be activated by directly creating a task, so they're not dead code. We'll need to check with Abhishek or Erno about whether the same technique can be used to patch it (I think it can). I'm not sure what the status of the Tasks API is with respect to being completely removed.

@Guillaume: I'm looking at moving the checks in cinder to convert_image() without causing regressions. Can probably do it for this vmdk issue, but not the backing_file.

Revision history for this message
Dan Smith (danms) wrote :

Yeah, task creation is locked down to admin-only now (and have been for a while, AFAIK) so I don't think this is likely exploitable by a regular user. That, and I'm not even sure how you configure (or request) conversion via that mechanism. Anyway, I can update the patch to just do the same thing for it, but I won't be able to test it, other than unit tests in the tree that I find (which may not exist). But, if you think it's worth updating that one too, I will.

Revision history for this message
Jeremy Stanley (fungi) wrote :

If nothing else, it may serve as a stronger reminder to anyone backporting to very old releases that they need to take care of fixing it there.

Revision history for this message
Dan Smith (danms) wrote :

Okay, rev2 of the glance patch with the legacy import task patched and those tests updated as well. I'm not sure if/when this was ever used, so I'm not sure how important of an indicator it will be to people backporting to kilo or something ancient, but here it is.

Revision history for this message
Arnaud Morin (arnaud-morin) wrote :

Hello,

FYI, we mitigated this on our infra by completely disabling the vmdk convertion. Like guillaume explained, we did that directly in the _convert method.

Anyway, the glance patch looks good.

What are the next steps?

Revision history for this message
Arnaud Morin (arnaud-morin) wrote :

Just to let you know that glance is affected by the bug with the import plugin.
A user can trigger the conversion if the operator enabled this in the config:

[image_import_opts]
image_import_plugins = ['image_conversion']

Then, as a regular user:
openstack image create --file leak.qcow --disk-format qcow2 --import leak9.qcow

Will test the patch to make sure this is mitigating the issue now

Revision history for this message
Arnaud Morin (arnaud-morin) wrote :

I did test the glance patch. It works (mitigate both qcow and vmdk, with import taskflow).

One note, during the convert process, the qemu-img info is executed.
Something like this:
qemu-img info --output=json /tmp/staging/7b7b34b1-332c-4257-8a19-b9ff71bfe2c5

If the backing file does not exists, it fails with:
"qemu-img: Could not open '/tmp/staging/7b7b34b1-332c-4257-8a19-b9ff71bfe2c5': Could not open '/etc/nova/nova.conf': No such file or directory\n"

Leaving the image in "importing" state.
This is wrong IMO and should be catched as an invalid type, what do you think?

Revision history for this message
Arnaud Morin (arnaud-morin) wrote :

Another thing,
are we sure VMDK and QCOW formats are the only formats allowing backing files?

Revision history for this message
Jeremy Stanley (fungi) wrote :

Assuming we have answers to your other questions, and Cinder/Glance/Nova security reviewers confirm they're comfortable with the proposed master branch patches and expect them to be safely backportable to maintained stable branches of their respective projects, I'll start drafting an impact statement to use in requesting a private CVE assignment and for inclusion in embargoed downstream stakeholder notifications (and eventually in a corresponding security advisory publication).

Once we have a CVE assignment and backports attached for each project's patch, we should be ready to schedule the advisory publication and provide advance copies of those patches to the downstream stakeholders.

Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

Revised patch posted in case anyone has time to look over Thanksgiving. This is not the final version. I had to move some code around because we weren't checking the format info for a image with container_format==compressed correctly (qemu-img info was being run on the gzip'd file, which would always report itself as 'raw' format. Also added a check to make sure the disk_format and what cinder detects as the image format matches, which I think is correct, but which may make some previously successful requests to create a volume from a source image to now fail.

Jeremy Stanley (fungi)
Changed in ossa:
status: Incomplete → Confirmed
importance: Undecided → High
assignee: nobody → Jeremy Stanley (fungi)
Jeremy Stanley (fungi)
Changed in ossa:
status: Confirmed → In Progress
Jeremy Stanley (fungi)
summary: Arbitrary file access through custom VMDK flat descriptor
+ (CVE-2022-47951)
Jeremy Stanley (fungi)
Changed in ossa:
status: In Progress → Fix Committed
Jeremy Stanley (fungi)
description: updated
information type: Private Security → Public Security
Changed in glance:
status: New → In Progress
Changed in cinder:
status: New → In Progress
Jeremy Stanley (fungi)
summary: - Arbitrary file access through custom VMDK flat descriptor
- (CVE-2022-47951)
+ [OSSA-2023-002] Arbitrary file access through custom VMDK flat
+ descriptor (CVE-2022-47951)
Changed in ossa:
status: Fix Committed → Fix Released
Changed in glance:
status: In Progress → Fix Released
tags: added: in-stable-zed
tags: added: in-stable-yoga
110 comments hidden view all 189 comments
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/871702

Revision history for this message
Christian Rohmann (christian-rohmann) wrote :

Jeremy there still seem to be no updated packages on UCA (Ubuntu Cloud Archive) - https://openstack-ci-reports.ubuntu.com/reports/cloud-archive/xena_versions.html, https://openstack-ci-reports.ubuntu.com/reports/cloud-archive/yoga_versions.html, ...

I highly doubt there will be any without an SRU. Take this (my) recent bug as an example:
https://bugs.launchpad.net/cloud-archive/+bug/1995861

stable/xyz already has the fix, there was a release made, but no package updates were triggerd.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/glance/+/871614
Committed: https://opendev.org/openstack/glance/commit/4967ab6935cfd0274ae801ac943d01909a236a0a
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 4967ab6935cfd0274ae801ac943d01909a236a0a
Author: Dan Smith <email address hidden>
Date: Mon Dec 19 15:00:35 2022 +0000

    Enforce image safety during image_conversion

    This does two things:

    1. It makes us check that the QCOW backing_file is unset on those
    types of images. Nova and Cinder do this already to prevent an
    arbitrary (and trivial to accomplish) host file exposure exploit.
    2. It makes us restrict VMDK files to only allowed subtypes. These
    files can name arbitrary files on disk as extents, providing the
    same sort of attack. Default that list to just the types we believe
    are actually useful for openstack, and which are monolithic.

    The configuration option to specify allowed subtypes is added in
    glance's config and not in the import options so that we can extend
    this check later to image ingest. The format_inspector can tell us
    what the type and subtype is, and we could reject those images early
    and even in the case where image_conversion is not enabled.

    Closes-Bug: #1996188
    Change-Id: Idf561f6306cebf756c787d8eefdc452ce44bd5e0
    (cherry picked from commit 0d6282a01691cecc2798f7858b181c4bb30f850c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/glance/+/871617
Committed: https://opendev.org/openstack/glance/commit/dc8e5a5cc7f5e9d1b697e520a7533cc90516db1b
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit dc8e5a5cc7f5e9d1b697e520a7533cc90516db1b
Author: Dan Smith <email address hidden>
Date: Mon Dec 19 15:00:35 2022 +0000

    Enforce image safety during image_conversion

    This does two things:

    1. It makes us check that the QCOW backing_file is unset on those
    types of images. Nova and Cinder do this already to prevent an
    arbitrary (and trivial to accomplish) host file exposure exploit.
    2. It makes us restrict VMDK files to only allowed subtypes. These
    files can name arbitrary files on disk as extents, providing the
    same sort of attack. Default that list to just the types we believe
    are actually useful for openstack, and which are monolithic.

    The configuration option to specify allowed subtypes is added in
    glance's config and not in the import options so that we can extend
    this check later to image ingest. The format_inspector can tell us
    what the type and subtype is, and we could reject those images early
    and even in the case where image_conversion is not enabled.

    Closes-Bug: #1996188
    Change-Id: Idf561f6306cebf756c787d8eefdc452ce44bd5e0
    (cherry picked from commit 0d6282a01691cecc2798f7858b181c4bb30f850c)
    (cherry picked from commit 4967ab6935cfd0274ae801ac943d01909a236a0a)

Revision history for this message
Jeremy Stanley (fungi) wrote :

Christian: OpenStack's vulnerability coordinators provide advance copies of embargoed security fixes to representatives of distributions including Ubuntu. We also notify the private linux-distros mailing list where they have representatives subscribed. At time of publication we circulate our advisories not only to our community's openstack-announce and openstack-discuss mailing lists, but also the broader <email address hidden> discussion list where they participate.

OpenStack is packaged and included in numerous distributions. The available time for our vulnerability coordinators is already quite minimal, and we simply cannot follow every single distribution's custom workflow for requesting that they act on security advisories. If a distribution you use isn't providing updated packages with the fixes we've supplied in a timely enough manner to suit your tastes, please follow up with them directly. It will mean more coming from their users than from me even if I did have the time to do so.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/nova/+/871616
Committed: https://opendev.org/openstack/nova/commit/6e8ed78470edbae7b58d75e9e9f4f62bdb30a170
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 6e8ed78470edbae7b58d75e9e9f4f62bdb30a170
Author: Dan Smith <email address hidden>
Date: Thu Nov 10 09:55:48 2022 -0800

    [stable-only][cve] Check VMDK create-type against an allowed list

    NOTE(sbauza): Stable policy allows us to proactively merge a backport without waiting for the parent patch to be merged (exception to rule #4 in [1]. Marking [stable-only] in order to silence nova-tox-validate-backport

    [1] https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes

    Related-Bug: #1996188
    Change-Id: I5a399f1d3d702bfb76c067893e9c924904c8c360

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/871625
Committed: https://opendev.org/openstack/cinder/commit/e96415409daa158cc503c1adec1ca1ca4f940082
Submitter: "Zuul (22348)"
Branch: stable/xena

commit e96415409daa158cc503c1adec1ca1ca4f940082
Author: Brian Rosmaita <email address hidden>
Date: Sat Dec 10 17:09:36 2022 -0500

    Check VMDK subformat against an allowed list

    Also add a more general check to convert_image that the image format
    reported by qemu-img matches what the caller says it is.

    Change-Id: I3c60ee4c0795aadf03108ed9b5a46ecd116894af
    Partial-bug: #1996188
    (cherry picked from commit 930fc93e9fda82a4aa4568ae149c3c80af7379d0)
    (cherry picked from commit ba37dc2ead69c08d7ede242295ff997086e6121d)
    Conflicts:
      cinder/image/image_utils.py
       - changed type annotations to use implicit Optional to be
         consistent with cinder yoga mypy usage
       - removed refs to image_conversion_disable in tests
    (cherry picked from commit 2ae5d53526e2b224d81b3259140c59aba97d72c3)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/glance/+/871619
Committed: https://opendev.org/openstack/glance/commit/f45b5f024e765f0000884dfec5ac222124cfbc6d
Submitter: "Zuul (22348)"
Branch: stable/xena

commit f45b5f024e765f0000884dfec5ac222124cfbc6d
Author: Dan Smith <email address hidden>
Date: Mon Dec 19 15:00:35 2022 +0000

    Enforce image safety during image_conversion

    This does two things:

    1. It makes us check that the QCOW backing_file is unset on those
    types of images. Nova and Cinder do this already to prevent an
    arbitrary (and trivial to accomplish) host file exposure exploit.
    2. It makes us restrict VMDK files to only allowed subtypes. These
    files can name arbitrary files on disk as extents, providing the
    same sort of attack. Default that list to just the types we believe
    are actually useful for openstack, and which are monolithic.

    The configuration option to specify allowed subtypes is added in
    glance's config and not in the import options so that we can extend
    this check later to image ingest. The format_inspector can tell us
    what the type and subtype is, and we could reject those images early
    and even in the case where image_conversion is not enabled.

    Closes-Bug: #1996188
    Change-Id: Idf561f6306cebf756c787d8eefdc452ce44bd5e0
    (cherry picked from commit 0d6282a01691cecc2798f7858b181c4bb30f850c)
    (cherry picked from commit 4967ab6935cfd0274ae801ac943d01909a236a0a)
    (cherry picked from commit dc8e5a5cc7f5e9d1b697e520a7533cc90516db1b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/nova/+/871622
Committed: https://opendev.org/openstack/nova/commit/867c4dd893ea7211e89b78b22b8da920a74622ff
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 867c4dd893ea7211e89b78b22b8da920a74622ff
Author: Dan Smith <email address hidden>
Date: Thu Nov 10 09:55:48 2022 -0800

    [stable-only][cve] Check VMDK create-type against an allowed list

    Trivial conflicts on xena only in:
            nova/conf/compute.py

    NOTE(sbauza): Stable policy allows us to proactively merge a backport without waiting for the parent patch to be merged (exception to rule #4 in [1]. Marking [stable-only] in order to silence nova-tox-validate-backport

    [1] https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes

    Related-Bug: #1996188
    Change-Id: I5a399f1d3d702bfb76c067893e9c924904c8c360

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/nova/+/871624
Committed: https://opendev.org/openstack/nova/commit/516f0de1f6a54cd24d8ebc906c1e3fd3bab0d32e
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 516f0de1f6a54cd24d8ebc906c1e3fd3bab0d32e
Author: Dan Smith <email address hidden>
Date: Thu Nov 10 09:55:48 2022 -0800

    [stable-only][cve] Check VMDK create-type against an allowed list

    NOTE(sbauza): Stable policy allows us to proactively merge a backport without waiting for the parent patch to be merged (exception to rule #4 in [1]. Marking [stable-only] in order to silence nova-tox-validate-backport

    [1] https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes

    Related-Bug: #1996188
    Change-Id: I5a399f1d3d702bfb76c067893e9c924904c8c360

Revision history for this message
Christian Rohmann (christian-rohmann) wrote (last edit ):

Jeremy: Thanks for your response. I did not mean to imply you did not inform to the best of your abilities. I rather wanted to mention recent experience with the SRU process of the Ubuntu Cloud Archive, which apparently does mean there are no new packages built and released for stable branch point releases per se.

I know opened an SRU bug (https://bugs.launchpad.net/cloud-archive/+bug/2003899) pointing to this bug here asking for new packages.

FWIW, I totally agree that the issue of the UCA not automatically providing new packages to fix critical (security) bugs is not something the OpenStack team is responsible for having sent out notifications on all the usual channels. But since UCA is a quite popular source for OpenStack packages and it's also used and promoted by deployment tooling like openstack-ansible I believe it's worth ask why the notification of incoming security fixes seems to not work as smooth here as with other distros.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/cinder/+/871817

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/glance/+/871623
Committed: https://opendev.org/openstack/glance/commit/06e6be579156d8bf2ce8e021d07d6f351fb88c07
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 06e6be579156d8bf2ce8e021d07d6f351fb88c07
Author: Dan Smith <email address hidden>
Date: Mon Dec 19 15:00:35 2022 +0000

    Enforce image safety during image_conversion

    This does two things:

    1. It makes us check that the QCOW backing_file is unset on those
    types of images. Nova and Cinder do this already to prevent an
    arbitrary (and trivial to accomplish) host file exposure exploit.
    2. It makes us restrict VMDK files to only allowed subtypes. These
    files can name arbitrary files on disk as extents, providing the
    same sort of attack. Default that list to just the types we believe
    are actually useful for openstack, and which are monolithic.

    The configuration option to specify allowed subtypes is added in
    glance's config and not in the import options so that we can extend
    this check later to image ingest. The format_inspector can tell us
    what the type and subtype is, and we could reject those images early
    and even in the case where image_conversion is not enabled.

    Closes-Bug: #1996188
    Change-Id: Idf561f6306cebf756c787d8eefdc452ce44bd5e0
    (cherry picked from commit 0d6282a01691cecc2798f7858b181c4bb30f850c)
    (cherry picked from commit 4967ab6935cfd0274ae801ac943d01909a236a0a)
    (cherry picked from commit dc8e5a5cc7f5e9d1b697e520a7533cc90516db1b)
    (cherry picked from commit f45b5f024e765f0000884dfec5ac222124cfbc6d)
    (cherry picked from commit 9a98c4a7d1358cdae009cc8fb6377160a126ea7b)
    Conflicts: glance/tests/unit/async_/flows/plugins/test_image_conversion.py
       - removed code related to missing tests - 050802dd67b9135e04a65d340531157c94248c51

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to cinder (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/871976

Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

https://review.opendev.org/c/openstack/nova/+/871612 is now merged, putting the bug report to Fix Released.

Changed in nova:
importance: Undecided → Critical
status: New → Confirmed
status: Confirmed → Fix Released
Changed in cinder:
importance: Undecided → Critical
status: In Progress → Fix Released
Revision history for this message
Giuseppe Petralia (peppepetra) wrote :

I am trying to reproduce the bug with a focal/ussuri openstack and step 4 (creating the VM) is failing for me with error:

2023-01-30 11:45:24.823 62889 ERROR nova.compute.manager [req-54de4256-c3b0-46d9-9fa3-e1b17a81e5cb 7e70ed49536144699c3b2202b699b2a9 39b49452b4184ba4a07df3f50e600de8 - 1a40c665d79e45e6ba
d525d061cdb548 1a40c665d79e45e6bad525d061cdb548] [instance: e5814346-68e2-4ab0-b6ed-31c3ddf627ef] Failed to build and run instance: libvirt.libvirtError: internal error: process exited
while connecting to monitor: 2023-01-30T11:45:23.807934Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-4-format","read-only":true,"discard":"unmap","cache":{"direct":true,"no-flus
h":false},"driver":"vmdk","file":"libvirt-4-storage","backing":null}: Could not open '/etc/nova/nova.conf': Permission denied

My nova version is:

root@juju-c1f3a4-peppepetra-8:~# dpkg -l | grep nova
ii nova-api-metadata 2:21.2.4-0ubuntu2 all OpenStack Compute - metadata API frontend
ii nova-common 2:21.2.4-0ubuntu2 all OpenStack Compute - common files
ii nova-compute 2:21.2.4-0ubuntu2 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:21.2.4-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:21.2.4-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python3-nova 2:21.2.4-0ubuntu2 all OpenStack Compute Python 3 libraries
ii python3-novaclient 2:17.0.0-0ubuntu1 all client library for OpenStack Compute API - 3.x

Can you please tell me on which nova version are you seeing this issue?

Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Honestly, now this bug report is public, I don't really want to discuss about the attack vectors and how to reproduce the security issue.

At least what you need to know is that all the Nova releases versions had the bug until we fixed it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 23.1.0

This issue was fixed in the openstack/glance 23.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 24.2.0

This issue was fixed in the openstack/glance 24.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 25.1.0

This issue was fixed in the openstack/glance 25.1.0 release.

Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

@Guiseppe at comment #165:

This line in your output shows that the "backing file" exploit was attempted:

   Could not open '/etc/nova/nova.conf': Permission denied

Luckily it didn't succeed due to a permissions issue, but it might succeed for other files in your installation, so you should definitely upgrade/patch it.

Revision history for this message
Bjoern (bjoern-t) wrote :

Hello,

are we expecting patches for Rocky as it is still in EM?
But it seems based on oslo_utils.imageutils changes it can't easily be back ported it seems.
Could you add a clarification for Rocky ?

Revision history for this message
Jeremy Stanley (fungi) wrote :

Bjoern: There should be no expectations where extended maintenance[*] branches are concerned. Projects can voluntarily leave their branches open when they reach the extended maintenance phase, in order to facilitate collaboration between interested community members (typically package maintainers for distributions providing downstream support for those versions). If you are interested in supplying a backport to stable/rocky, please push it into Gerrit for reviewing.

Also, be aware that a proposal[**] has already been floated to close stable/rocky on all projects, so there may not be much time left to get patches merged there if you have them.

[*] https://docs.openstack.org/project-team-guide/stable-branches.html#extended-maintenance

[**] https://lists.openstack.org/pipermail/openstack-discuss/2023-January/031922.html

Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

I backported the fix as far as stable/train so the Cinder project would at least make an attempt at a fix that supported python 2.7. Nova EOL'd queens, rocky, and stein in November 2022, which in my opinion effectively EOL's those branches for openstack in general (plus, the cinder gates are currently nonfunctional for stein and rocky); the last release of openstack to support python 2.7 is the train series.

The issue with oslo.utils.imageutils is that support for the 'format-specific' attribute of the json qemu-img info output was introduced in 4.1.0, which is python-3-only. However, support for reading the json output of qemu-img info was introduced in 3.14.0. Not sure how that matches up branch-wise, but oslo.utils rocky-em tag is the same as 3.36.5. So being able to read json isn't an issue.

If you want to backport the CVE-2022-47951 fix to rocky for Cinder, you'll need to do two things (as discussed in comment #86 above):

1. backport change Ia0353204abf8, which is commit 9f9194d804c in stable/train, to rocky. This will make cinder request json output from qemu-img-info

2. backport the cinder-1996188-train.patch attached to this bug (it's not in stable/train yet ... the stein and rocky gates aren't the only ones having issues!) which you can see in gerrit as https://review.opendev.org/c/openstack/cinder/+/871631. It includes code that will get the format-specific stuff out of the qemu-img info response if oslo.utils isn't able to do it.

The code is so old at that point that both those patches will produce conflicts (because they'll refer to stuff in train that isn't present in rocky), but hopefully the conflicts won't be too bad to resolve.

I'm not sure exactly what you'll need to do for nova. I suspect that json support was added to oslo.utils.imageutils specifically for nova, so you probably won't have to worry about part 1 of the cinder backport (though definitely verify that i'm correct about that). You'll need to make the same adjustment for oslo.utils <4.1.0 that we made in cinder; you can see it on the train patch here:

https://review.opendev.org/c/openstack/cinder/+/871631/1/cinder/image/image_utils.py#137

Hopefully it's obvious where to do that in the nova patch. Good luck!

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/871627
Committed: https://opendev.org/openstack/cinder/commit/be11d54ac420e0ebc0da6917d7ffc3af59b40f24
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit be11d54ac420e0ebc0da6917d7ffc3af59b40f24
Author: Brian Rosmaita <email address hidden>
Date: Wed Jan 11 09:50:29 2023 -0500

    Check VMDK subformat against an allowed list

    Also add a more general check to convert_image that the image format
    reported by qemu-img matches what the caller says it is.

    Change-Id: I3c60ee4c0795aadf03108ed9b5a46ecd116894af
    Partial-bug: #1996188
    (cherry picked from commit 930fc93e9fda82a4aa4568ae149c3c80af7379d0)
    (cherry picked from commit ba37dc2ead69c08d7ede242295ff997086e6121d)
    Conflicts:
      cinder/image/image_utils.py
       - changed type annotations to use implicit Optional to be
         consistent with cinder yoga mypy usage
       - removed refs to image_conversion_disable in tests
    (cherry picked from commit 2ae5d53526e2b224d81b3259140c59aba97d72c3)
    (cherry picked from commit e96415409daa158cc503c1adec1ca1ca4f940082)
    Conflicts:
      cinder/image/image_utils.py
       - removed type annotations
       - restored wallaby-era fetch_verify_image() function signature

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/871628
Committed: https://opendev.org/openstack/cinder/commit/17565262dafb5beeaec8e9e249e760fc64f3df93
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 17565262dafb5beeaec8e9e249e760fc64f3df93
Author: Brian Rosmaita <email address hidden>
Date: Sat Jan 14 07:06:27 2023 -0500

    Check VMDK subformat against an allowed list

    Also add a more general check to convert_image that the image format
    reported by qemu-img matches what the caller says it is.

    Change-Id: I3c60ee4c0795aadf03108ed9b5a46ecd116894af
    Partial-bug: #1996188
    (cherry picked from commit 930fc93e9fda82a4aa4568ae149c3c80af7379d0)
    (cherry picked from commit ba37dc2ead69c08d7ede242295ff997086e6121d)
    Conflicts:
      cinder/image/image_utils.py
       - changed type annotations to use implicit Optional to be
         consistent with cinder yoga mypy usage
       - removed refs to image_conversion_disable in tests
    (cherry picked from commit 2ae5d53526e2b224d81b3259140c59aba97d72c3)
    (cherry picked from commit e96415409daa158cc503c1adec1ca1ca4f940082)
    Conflicts:
      cinder/image/image_utils.py
       - removed type annotations
       - restored wallaby-era fetch_verify_image() function signature
    (cherry picked from commit be11d54ac420e0ebc0da6917d7ffc3af59b40f24)
    Conflicts:
      cinder/tests/unit/test_image_utils.py
       - did not include extraneous test from be11d54ac's parent commit
    Additions:
      cinder/image/image_utils.py
       - added code to handle oslo.utils<4.1.0,>=3.14.0
      cinder/tests/unit/test_image_utils.py
       - added a test for ^^

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/nova/+/871557
Committed: https://opendev.org/openstack/nova/commit/719a2a6089ec240d23caeb93ac25ed5259d8d9bc
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 719a2a6089ec240d23caeb93ac25ed5259d8d9bc
Author: Dan Smith <email address hidden>
Date: Thu Nov 10 09:55:48 2022 -0800

    [stable-only][cve] Check VMDK create-type against an allowed list

    NOTE(sbauza): Stable policy allows us to proactively merge a backport without waiting for the parent patch to be merged (exception to rule #4 in [1]. Marking [stable-only] in order to silence nova-tox-validate-backport

    [1] https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes

    Related-Bug: #1996188
    Change-Id: I5a399f1d3d702bfb76c067893e9c924904c8c360

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/871629
Committed: https://opendev.org/openstack/cinder/commit/9902c17927b69e0b2379dfe8d2a2b89244162526
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 9902c17927b69e0b2379dfe8d2a2b89244162526
Author: Brian Rosmaita <email address hidden>
Date: Mon Jan 16 14:44:04 2023 -0500

    Check VMDK subformat against an allowed list

    Also add a more general check to convert_image that the image format
    reported by qemu-img matches what the caller says it is.

    Change-Id: I3c60ee4c0795aadf03108ed9b5a46ecd116894af
    Partial-bug: #1996188
    (cherry picked from commit 930fc93e9fda82a4aa4568ae149c3c80af7379d0)
    (cherry picked from commit ba37dc2ead69c08d7ede242295ff997086e6121d)
    Conflicts:
      cinder/image/image_utils.py
       - changed type annotations to use implicit Optional to be
         consistent with cinder yoga mypy usage
       - removed refs to image_conversion_disable in tests
    (cherry picked from commit 2ae5d53526e2b224d81b3259140c59aba97d72c3)
    (cherry picked from commit e96415409daa158cc503c1adec1ca1ca4f940082)
    Conflicts:
      cinder/image/image_utils.py
       - removed type annotations
       - restored wallaby-era fetch_verify_image() function signature
    (cherry picked from commit be11d54ac420e0ebc0da6917d7ffc3af59b40f24)
    Conflicts:
      cinder/tests/unit/test_image_utils.py
       - did not include extraneous test from be11d54ac's parent commit
    Additions:
      cinder/image/image_utils.py
       - added code to handle oslo.utils<4.1.0,>=3.14.0
      cinder/tests/unit/test_image_utils.py
       - added a test for ^^
    (cherry picked from commit 17565262dafb5beeaec8e9e249e760fc64f3df93)
    Conflicts:
      cinder/image/image_utils.py
       - removed src_passphrase_file parameter to convert_image() that was
         introduced in victoria by change I896f70d204ad103e
      cinder/tests/unit/test_image_utils.py
       - removed references to ddt.TestNameFormat (not present in this
         version of ddt)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/glance/+/871626
Committed: https://opendev.org/openstack/glance/commit/b60fb70c9faf3b73eb4d8a73cd5cb09f2fa70a61
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit b60fb70c9faf3b73eb4d8a73cd5cb09f2fa70a61
Author: Dan Smith <email address hidden>
Date: Mon Dec 19 15:00:35 2022 +0000

    Enforce image safety during image_conversion

    This does two things:

    1. It makes us check that the QCOW backing_file is unset on those
    types of images. Nova and Cinder do this already to prevent an
    arbitrary (and trivial to accomplish) host file exposure exploit.
    2. It makes us restrict VMDK files to only allowed subtypes. These
    files can name arbitrary files on disk as extents, providing the
    same sort of attack. Default that list to just the types we believe
    are actually useful for openstack, and which are monolithic.

    The configuration option to specify allowed subtypes is added in
    glance's config and not in the import options so that we can extend
    this check later to image ingest. The format_inspector can tell us
    what the type and subtype is, and we could reject those images early
    and even in the case where image_conversion is not enabled.

    Closes-Bug: #1996188
    Change-Id: Idf561f6306cebf756c787d8eefdc452ce44bd5e0
    (cherry picked from commit 0d6282a01691cecc2798f7858b181c4bb30f850c)
    (cherry picked from commit 4967ab6935cfd0274ae801ac943d01909a236a0a)
    (cherry picked from commit dc8e5a5cc7f5e9d1b697e520a7533cc90516db1b)
    (cherry picked from commit f45b5f024e765f0000884dfec5ac222124cfbc6d)
    (cherry picked from commit 9a98c4a7d1358cdae009cc8fb6377160a126ea7b)
    Conflicts: glance/tests/unit/async_/flows/plugins/test_image_conversion.py
       - removed code related to missing tests - 050802dd67b9135e04a65d340531157c94248c51
    (cherry picked from commit 06e6be579156d8bf2ce8e021d07d6f351fb88c07)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/nova/+/871702
Committed: https://opendev.org/openstack/nova/commit/3fe8880d3759cbd7b19d75dcf235dfd5c511be13
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 3fe8880d3759cbd7b19d75dcf235dfd5c511be13
Author: Dan Smith <email address hidden>
Date: Thu Nov 10 09:55:48 2022 -0800

    [stable-only][cve] Check VMDK create-type against an allowed list

    NOTE(sbauza): Stable policy allows us to proactively merge a backport without waiting for the parent patch to be merged (exception to rule #4 in [1]. Marking [stable-only] in order to silence nova-tox-validate-backport

    [1] https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes

    Conflicts vs victoria in:
            nova/conf/compute.py

    Related-Bug: #1996188
    Change-Id: I5a399f1d3d702bfb76c067893e9c924904c8c360

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 26.0.0.0b3

This issue was fixed in the openstack/glance 26.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/train)

Reviewed: https://review.opendev.org/c/openstack/glance/+/871630
Committed: https://opendev.org/openstack/glance/commit/7756a90b7dacf44061d473eb03b92d2a40a307f6
Submitter: "Zuul (22348)"
Branch: stable/train

commit 7756a90b7dacf44061d473eb03b92d2a40a307f6
Author: Dan Smith <email address hidden>
Date: Mon Dec 19 15:00:35 2022 +0000

    Enforce image safety during image_conversion

    This does two things:

    1. It makes us check that the QCOW backing_file is unset on those
    types of images. Nova and Cinder do this already to prevent an
    arbitrary (and trivial to accomplish) host file exposure exploit.
    2. It makes us restrict VMDK files to only allowed subtypes. These
    files can name arbitrary files on disk as extents, providing the
    same sort of attack. Default that list to just the types we believe
    are actually useful for openstack, and which are monolithic.

    The configuration option to specify allowed subtypes is added in
    glance's config and not in the import options so that we can extend
    this check later to image ingest. The format_inspector can tell us
    what the type and subtype is, and we could reject those images early
    and even in the case where image_conversion is not enabled.

    Some changes were required in the backport in order to pass the CI:
    Pep8 fix: ignore H703 errors.
    Py27 fix: FileNotFoundError did not exist in Python2.7.

    Closes-Bug: #1996188
    Change-Id: Idf561f6306cebf756c787d8eefdc452ce44bd5e0
    (cherry picked from commit 0d6282a01691cecc2798f7858b181c4bb30f850c)
    (cherry picked from commit 4967ab6935cfd0274ae801ac943d01909a236a0a)
    (cherry picked from commit dc8e5a5cc7f5e9d1b697e520a7533cc90516db1b)
    (cherry picked from commit f45b5f024e765f0000884dfec5ac222124cfbc6d)
    (cherry picked from commit 9a98c4a7d1358cdae009cc8fb6377160a126ea7b)
    Conflicts: glance/tests/unit/async_/flows/plugins/test_image_conversion.py
       - removed code related to missing tests - 050802dd67b9135e04a65d340531157c94248c51
    (cherry picked from commit 06e6be579156d8bf2ce8e021d07d6f351fb88c07)
    (cherry picked from commit b60fb70c9faf3b73eb4d8a73cd5cb09f2fa70a61)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/glance/+/871621
Committed: https://opendev.org/openstack/glance/commit/9a98c4a7d1358cdae009cc8fb6377160a126ea7b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 9a98c4a7d1358cdae009cc8fb6377160a126ea7b
Author: Dan Smith <email address hidden>
Date: Mon Dec 19 15:00:35 2022 +0000

    Enforce image safety during image_conversion

    This does two things:

    1. It makes us check that the QCOW backing_file is unset on those
    types of images. Nova and Cinder do this already to prevent an
    arbitrary (and trivial to accomplish) host file exposure exploit.
    2. It makes us restrict VMDK files to only allowed subtypes. These
    files can name arbitrary files on disk as extents, providing the
    same sort of attack. Default that list to just the types we believe
    are actually useful for openstack, and which are monolithic.

    The configuration option to specify allowed subtypes is added in
    glance's config and not in the import options so that we can extend
    this check later to image ingest. The format_inspector can tell us
    what the type and subtype is, and we could reject those images early
    and even in the case where image_conversion is not enabled.

    Closes-Bug: #1996188
    Change-Id: Idf561f6306cebf756c787d8eefdc452ce44bd5e0
    (cherry picked from commit 0d6282a01691cecc2798f7858b181c4bb30f850c)
    (cherry picked from commit 4967ab6935cfd0274ae801ac943d01909a236a0a)
    (cherry picked from commit dc8e5a5cc7f5e9d1b697e520a7533cc90516db1b)
    (cherry picked from commit f45b5f024e765f0000884dfec5ac222124cfbc6d)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/train)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/871631
Committed: https://opendev.org/openstack/cinder/commit/e40a0bea89d4e5c99711f23faf3e18ff420ad100
Submitter: "Zuul (22348)"
Branch: stable/train

commit e40a0bea89d4e5c99711f23faf3e18ff420ad100
Author: Brian Rosmaita <email address hidden>
Date: Mon Jan 16 14:44:04 2023 -0500

    Check VMDK subformat against an allowed list

    Also add a more general check to convert_image that the image format
    reported by qemu-img matches what the caller says it is.

    Change-Id: I3c60ee4c0795aadf03108ed9b5a46ecd116894af
    Partial-bug: #1996188
    (cherry picked from commit 930fc93e9fda82a4aa4568ae149c3c80af7379d0)
    (cherry picked from commit ba37dc2ead69c08d7ede242295ff997086e6121d)
    Conflicts:
      cinder/image/image_utils.py
       - changed type annotations to use implicit Optional to be
         consistent with cinder yoga mypy usage
       - removed refs to image_conversion_disable in tests
    (cherry picked from commit 2ae5d53526e2b224d81b3259140c59aba97d72c3)
    (cherry picked from commit e96415409daa158cc503c1adec1ca1ca4f940082)
    Conflicts:
      cinder/image/image_utils.py
       - removed type annotations
       - restored wallaby-era fetch_verify_image() function signature
    (cherry picked from commit be11d54ac420e0ebc0da6917d7ffc3af59b40f24)
    Conflicts:
      cinder/tests/unit/test_image_utils.py
       - did not include extraneous test from be11d54ac's parent commit
    Additions:
      cinder/image/image_utils.py
       - added code to handle oslo.utils<4.1.0,>=3.14.0
      cinder/tests/unit/test_image_utils.py
       - added a test for ^^
    (cherry picked from commit 17565262dafb5beeaec8e9e249e760fc64f3df93)
    Conflicts:
      cinder/image/image_utils.py
       - removed src_passphrase_file parameter to convert_image() that was
         introduced in victoria by change I896f70d204ad103e
      cinder/tests/unit/test_image_utils.py
       - removed references to ddt.TestNameFormat (not present in this
         version of ddt)
    (cherry picked from commit 9902c17927b69e0b2379dfe8d2a2b89244162526)
    Additions:
      cinder/tests/unit/test_image_utils.py
       - added code to handle oslo.utils<4.1.0 when running unit tests under
         Python 2.7

Changed in glance:
assignee: nobody → Dan Smith (danms)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/nova/+/871699
Committed: https://opendev.org/openstack/nova/commit/eabb16a421326388c8d53a1b6ca47d79a03e0e16
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit eabb16a421326388c8d53a1b6ca47d79a03e0e16
Author: Dan Smith <email address hidden>
Date: Thu Nov 10 09:55:48 2022 -0800

    [stable-only][cve] Check VMDK create-type against an allowed list

    NOTE(sbauza): Stable policy allows us to proactively merge a backport without waiting for the parent patch to be merged (exception to rule #4 in [1]. Marking [stable-only] in order to silence nova-tox-validate-backport

    [1] https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes

    Conflicts vs wallaby in:
            nova/conf/compute.py
            nova/tests/unit/virt/test_images.py

    Related-Bug: #1996188
    Change-Id: I5a399f1d3d702bfb76c067893e9c924904c8c360

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (stable/rocky)

Change abandoned by "Brian Rosmaita <email address hidden>" on branch: stable/rocky
Review: https://review.opendev.org/c/openstack/cinder/+/871817
Reason: Rocky transitioned to End of Life by change I600914dd08e9 and is accepting no more changes.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance train-eol

This issue was fixed in the openstack/glance train-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance ussuri-eol

This issue was fixed in the openstack/glance ussuri-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance victoria-eom

This issue was fixed in the openstack/glance victoria-eom release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance wallaby-eom

This issue was fixed in the openstack/glance wallaby-eom release.

Displaying first 40 and last 40 comments. View all 189 comments or add a comment.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.