[OSSA-2018-001] Swapping encrypted volumes can lead to data loss and a possible compute host DOS attack (CVE-2017-18191)

Bug #1739593 reported by Lee Yarwood on 2017-12-21
264
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Critical
Lee Yarwood
Ocata
High
Lee Yarwood
Pike
High
Lee Yarwood
Queens
High
Lee Yarwood
OpenStack Security Advisory
Undecided
Unassigned

Bug Description

Description
===========

At present when swapping encrypted volumes no attempt is made to attach an encryptor to the target volume. This results in the RAW underlying volume being used during the rebase, where decrypted data is copied from the original volume to the target:

https://github.com/openstack/nova/blob/76dfdfc1ad8c0e5376bd997e45f65bec9ff53d12/nova/virt/libvirt/driver.py#L1338-L1372

Any attempt to detach and then reattach this volume from the instance will lead to the volume being reformatted as the os-brick supplied encryptors do not identify the volume as encrypted:

https://github.com/openstack/os-brick/blob/6835b885dc4144fdc6e9863ca59ae54f76938995/os_brick/encryptors/luks.py#L138-L161

Additionally, while unlikely, a malicious user could easily DOS the compute node hosting the instance by writing a corrupt LUKS header to the RAW volume before detaching and reattaching the volume. For example, setting a keyslot iters (used by PBKDF2) to a large value etc (kudos to mdbooth for suggesting this):

https://gitlab.com/cryptsetup/cryptsetup/wikis/LUKS-standard/on-disk-format.pdf

This method of DOS'ing the compute host was previously discussed in the context of bug 1724573 but dismissed as access to the underlying volume was dependent on a host reboot, outside of a users control. This bug differs as a user has full control of the above volume-update/swap_volume flow that provides access to the underlying volume.

Steps to reproduce
==================

- Create two encrypted volumes

$ cinder type-create LUKS
$ cinder encryption-type-create --cipher aes-xts-plain64 \
                                --key_size 256 \
                                --control_location front-end LUKS luks

$ cinder type-create LUKS_NEW
$ cinder encryption-type-create --cipher aes-xts-plain64 \
                                --key_size 256 \
                                --control_location front-end LUKS_NEW luks

$ cinder create --volume-type LUKS 1
$ cinder create --volume-type LUKS_NEW 1

- Spawn an instance, attaching the first volume before swapping to the second:

$ nova boot --image cirros-0.3.5-x86_64-disk --flavor 1 swap_test
$ nova volume-attach $instance $vol-luks
$ nova volume-update $instance $vol-luks $vol-luks-new

- Review the resulting volume attachment on the compute host:

$ virsh domblklist $instance
Target Source
------------------------------------------------
vda /opt/stack/data/nova/instances/3d4c5842-45ab-4660-bf6e-9459f9a2ff8a/disk
vdb /dev/disk/by-id/scsi-36001405ba072cc9f93e444c9433ead1c

$ ll /dev/disk/by-id/scsi-36001405ba072cc9f93e444c9433ead1c
lrwxrwxrwx. 1 root root 9 Dec 21 05:30 /dev/disk/by-id/scsi-36001405ba072cc9f93e444c9433ead1c -> ../../sdd

$ sudo qemu-img info /dev/disk/by-id/scsi-36001405ba072cc9f93e444c9433ead1c
image: /dev/disk/by-id/scsi-36001405ba072cc9f93e444c9433ead1c
file format: raw
virtual size: 1.0G (1073741824 bytes)
disk size: 0

Expected result
===============

The encrypted volumes are rebased with their associated encryptors attached, leading to encrypted data being written to the underlying volumes.

Actual result
=============

Decrypted data from the source volume is written to the underlying target volume. This data will be lost with a subsequent detach / attach cycle. Access to the underlying volume could also be used by a malicious user to DOS the local compute host.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   Libvirt + KVM

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   LVM

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   N/A

Logs & Configs
==============

N/A

CVE References

Matthew Booth (mbooth-9) wrote :

I think the priority here is that the user loses their data. The DoS potential is real, but relatively inefficient. I think we could use a CVE to communicate the problem to users, but I don't think this particular issue is important enough to jump through secrecy hoops while we work on the data-loss bug.

I would personally be in favour of early disclosure, handling this openly, and getting it done quickly. We could possibly leave disclosure til early January, though, as we're unlikely to even start work on the fix until then. I'll obviously defer to the VMT and the reporter, just my 2c.

Lee Yarwood (lyarwood) wrote :

Agreed, the data loss bug is of much higher importance than the potential DoS, just wanted to ensure VMT members were at least aware of the possibility for this to be used maliciously. If there are no further concerns or objections then I'm fine with early disclosure so we can fix this as a priority, in the open.

Changed in ossa:
status: New → Incomplete
description: updated

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

This sounds similar to OSSA-2016-012, thus I this this is a class A according to VMT taxonomy ( https://security.openstack.org/vmt-process.html#incident-report-taxonomy ).

If this doesn't warrant an embargoed disclosure, or if a fix isn't proposed in timely manner, we should loop ossg-coresec before making this public, as explained in the embargo exception process here: https://security.openstack.org/vmt-process.html#embargo-exceptions

Jeremy Stanley (fungi) wrote :

Given Matthew and Lee are both suggesting we switch to our public workflow in hope of getting the data loss problem solved in a more timely manner (at the expense of disclosing a somewhat inefficient DoS risk), I've gone ahead and subscribed ossg-coresec reviewers to get their input first.

Jeremy Stanley (fungi) wrote :

Have these problems have existed since the introduction of LUKS support? Or did the ability to perform a volume-update/swap_volume come later? Are all currently supported stable branches (let's say back to stable/ocata anyway since stable/newton is likely to finally get EOL'd by the time this is fixed) at risk from this?

Lee Yarwood (lyarwood) wrote :

Yes this has been an issue since the introduction of the LUKS encryptors and is present all the way back to stable/ocata (and stable/newton FWIW).

Lee Yarwood (lyarwood) wrote :

Any additional thoughts on this? I'd like to open this up and get the bug fix in sooner rather than later if possible.

Jeremy Stanley (fungi) wrote :

I know it's been holiday time for a lot of people, but given it's been two weeks at this point and we've got some consensus from the reporter and another core reviewer as well as some of the VMT, I'm going ahead and switching this one to our public workflow ending the embargo.

description: updated
information type: Private Security → Public Security

Fix proposed to branch: master
Review: https://review.openstack.org/531233

Changed in nova:
assignee: nobody → Lee Yarwood (lyarwood)
status: New → In Progress

Change abandoned by Lee Yarwood (<email address hidden>) on branch: master
Review: https://review.openstack.org/531233
Reason: ca323b87fa85a454fca9d46ada3677f18fe50022

Lee Yarwood (lyarwood) wrote :

Yes, the refactor in https://review.openstack.org/460243 will resolve this.

Jay Pipes (jaypipes) wrote :

data loss bugs are critical.

Changed in nova:
importance: Undecided → Critical

Reviewed: https://review.openstack.org/460243
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cd3eb60c2c00bcccfa9ccd4bf9d1a96ae7a5cd88
Submitter: Zuul
Branch: master

commit cd3eb60c2c00bcccfa9ccd4bf9d1a96ae7a5cd88
Author: Lee Yarwood <email address hidden>
Date: Mon Apr 24 19:23:44 2017 +0100

    libvirt: Collocate encryptor and volume driver calls

    This change introduces new utility methods for attaching and detaching
    frontend volume encryptors. These methods centralise the optional
    fetching of encryption metadata associated with a volume, fetching of the
    required encryptor and calls to detach or attach the encryptor.

    These new utility methods are called either after initially connecting
    to or before disconnecting from a volume. This ensures encryptors are
    correctly connected when swapping volumes for example, where previously
    no attempt was made to attach an encryptor to the target volume.

    The request context is provided to swap_volume and various other config
    generation related methods to allow for the lookup of the relevant
    encryption metadata if it is not provided.

    Closes-bug: #1739593
    Change-Id: Ica323b87fa85a454fca9d46ada3677f18fe50022

Changed in nova:
status: In Progress → Fix Released

Related BZ where swap volume won't work with native LUKS encrypted volumes if you're using new enough libvirt/qemu:

https://bugzilla.redhat.com/show_bug.cgi?id=760547

This issue was fixed in the openstack/nova 17.0.0.0b3 development milestone.

Backport to stable branches doesn't seems trivial, is this planed to be fixed in stable/pike and stable/ocata?

Lee Yarwood (lyarwood) wrote :

I wanted to give the master change sometime to bed in and while there are conflicts there don't appear to be any show stoppers AFAICT. I'll try to post something for review in the coming week so other stable cores can review.

Rather than backport these libvirt driver changes to stable branches, which are non-trivial refactors of the code, why can't we just put something in the stable branches that explicitly fails a swap volume of encrypted volumes? If swapping encrypted volumes doesn't work until queens, and only then if you're not using native luks encryption, it seems OK to just deny the operation outright in stable branches as a resolution to this bug for pike and ocata.

Jeremy Stanley (fungi) wrote :

If swapping encrypted volumes is already unusable in stable, then I don't see any significant concerns with making it safely unusable. Sounds far less likely to introduce regressions in stable too.

Change abandoned by Lee Yarwood (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/540861

So at this point I think we're just waiting for review of the stable/pike fix and then a similar fix backported to stable/ocata? Looks like we can probably get moving on CVE assignment from the VMT end in that case. Thanks!

summary: Swapping encrypted volumes can lead to data loss and a possible compute
- host DOS attack
+ host DOS attack (CVE-2017-18191)

I'm +1 now on the stable/pike backport https://review.openstack.org/#/c/543569/ - my +2 is pending discussion about a 'security' release note in the patch that mentions the CVE / OSSA which isn't yet published, and sounds like we might have a chicken-and-egg with that until the stable/ocata backport is proposed?

Jeremy Stanley (fungi) wrote :

CVE-2017-18191 is already reserved for this and can, I suppose, be included in that release note already. You're right that OSSA numbers have historically been assigned at the time of publication so I would consider either amending the release note later and not blocking the patch on it, or we can attempt to work out some sort of coordination mechanism for earlier OSSA assignment.

Reviewed: https://review.openstack.org/543569
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5b64a1936122eeb35f37a09f9d38159e1a224c58
Submitter: Zuul
Branch: stable/pike

commit 5b64a1936122eeb35f37a09f9d38159e1a224c58
Author: Lee Yarwood <email address hidden>
Date: Mon Feb 12 18:07:14 2018 +0000

    libvirt: Block swap volume attempts with encrypted volumes prior to Queens

    Prior to Queens any attempt to swap between encrypted volumes would
    result in unencrypted data being written to the new volume. This
    unencrypted data would then be overwritten the next time the volume was
    attached to an instance as Nova no longer identified the volume as
    encrypted, resulting in the volume being reformatted.

    This stable only change uses limited parts of the following changes to
    block all swap_volume attempts with encrypted volumes prior to Queens
    where this was resolved by Ica323b87fa85a454fca9d46ada3677f18 and also
    blocked when using QEMU to decrypt LUKS volumes by
    Ibfa64f18bbd2fb70db7791330ed1a64fe61c1.

    Ica323b87fa85a454fca9d46ada3677f18fe50022

    The request context is provided to swap_volume in order to look up the
    encryption metadata of a volume.

    Ibfa64f18bbd2fb70db7791330ed1a64fe61c1355

    Attempts to swap from an encrypted volume are blocked with a
    NotImplementedError exception raised.

    I258127fdcd011ccec721d5ff62eb7f128f130336

    Attempts to swap from an unencrypted volume to an encrypted volume are
    also blocked with a NotImplementedError exception raised.

    Ie02d298cd92d5b5ebcbbcd2b0e8be01f197bfafb

    The serial of a volume is used as the id if connection_info for the
    volume doesn't contain the volume_id key. Required to avoid bug #1746609.

    Closes-bug: #1739593
    Change-Id: If12e7860baad2899380f06144a0270784a5466b8

Reviewed: https://review.openstack.org/561604
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0225a61fc4557c1257383a654f0741f7ef2ddeac
Submitter: Zuul
Branch: stable/ocata

commit 0225a61fc4557c1257383a654f0741f7ef2ddeac
Author: Lee Yarwood <email address hidden>
Date: Mon Feb 12 18:07:14 2018 +0000

    libvirt: Block swap volume attempts with encrypted volumes prior to Queens

    Prior to Queens any attempt to swap between encrypted volumes would
    result in unencrypted data being written to the new volume. This
    unencrypted data would then be overwritten the next time the volume was
    attached to an instance as Nova no longer identified the volume as
    encrypted, resulting in the volume being reformatted.

    This stable only change uses limited parts of the following changes to
    block all swap_volume attempts with encrypted volumes prior to Queens
    where this was resolved by Ica323b87fa85a454fca9d46ada3677f18 and also
    blocked when using QEMU to decrypt LUKS volumes by
    Ibfa64f18bbd2fb70db7791330ed1a64fe61c1.

    Ica323b87fa85a454fca9d46ada3677f18fe50022

    The request context is provided to swap_volume in order to look up the
    encryption metadata of a volume.

    Ibfa64f18bbd2fb70db7791330ed1a64fe61c1355

    Attempts to swap from an encrypted volume are blocked with a
    NotImplementedError exception raised.

    I258127fdcd011ccec721d5ff62eb7f128f130336

    Attempts to swap from an unencrypted volume to an encrypted volume are
    also blocked with a NotImplementedError exception raised.

    Ie02d298cd92d5b5ebcbbcd2b0e8be01f197bfafb

    The serial of a volume is used as the id if connection_info for the
    volume doesn't contain the volume_id key. Required to avoid bug #1746609.

    Conflicts:
            nova/tests/unit/compute/test_compute_mgr.py
            nova/tests/unit/virt/libvirt/test_driver.py

    NOTE(lyarwood): Conflict due to cinderv3 support for swap_volume not
    being present in stable/ocata via
    I4b8bd01f1ffe2640fe7313213bf853d2e1bef9dd.

    Closes-bug: #1739593
    Change-Id: If12e7860baad2899380f06144a0270784a5466b8
    (cherry picked from commit 5b64a1936122eeb35f37a09f9d38159e1a224c58)

Reviewed: https://review.openstack.org/539893
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=c5f504bf1d2769191467b6a9749a1bb440750991
Submitter: Zuul
Branch: master

commit c5f504bf1d2769191467b6a9749a1bb440750991
Author: Tristan Cacqueray <email address hidden>
Date: Thu Feb 1 10:24:36 2018 +0000

    Adds OSSA-2018-001 (CVE-2017-18191)

    Change-Id: I43abe5ca3e14010b578a450bf2fa7bc3839b24b1
    Related-Bug: #1739593

summary: - Swapping encrypted volumes can lead to data loss and a possible compute
- host DOS attack (CVE-2017-18191)
+ [OSSA-2018-001] Swapping encrypted volumes can lead to data loss and a
+ possible compute host DOS attack (CVE-2017-18191)
Changed in ossa:
status: Incomplete → Fix Committed

This issue was fixed in the openstack/nova 16.1.2 release.

This issue was fixed in the openstack/nova 15.1.1 release.

To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.