There is no volume encryption support for rbd-backed volumes

[Matt Riedemann]

This came up as a discussion point in the nova IRC channel today because someone was talking about adding encryption support to Ceph in Nova and I pointed out that there is already a ceph job that runs the tempest luks/cryptsetup encrypted volume tests successfully, so why aren't those failing if it's not supported today?

We got looking at the code and logs and found that when nova tries to get volume encryption metadata from cinder for rbd-backed instances, nothing comes back so nova isn't doing anything with volume encryption using it's providers (luks / cryptsetup).

Change https://review.openstack.org/#/c/189799/ in nova adds logging to see this:

Confirmed that for LVM backed Cinder we get something back:

http://logs.openstack.org/99/189799/2/check/check-tempest-dsvm-full/c3ee602/logs/screen-n-cpu.txt.gz#_2015-06-09_18_18_18_078

For Ceph we don't:

http://logs.openstack.org/99/189799/2/check/check-tempest-dsvm-full-ceph/353db23/logs/screen-n-cpu.txt.gz#_2015-06-09_18_21_16_723

This might be working as designed, I'm not sure, but I'm opening the bug to track the effort since if you think you have encrypted volumes when using ceph and nova you're probably not, so there is a false sense of security here which is a bug.

[Matt Riedemann (mriedem)]

I've dug through the cinder API as much as I know how and I'm not seeing anything volume driver-specific going on where the mapping between the encryption table and the volume_type (that tempest is creating) isn't being created properly, and I poked through the logs as much as I could but I'm not seeing where things are breaking down, but apparently cinder is not finding that mapping (or making that connection for the volume) so that when we query the cinder API we get nothing back.

[Matt Riedemann (mriedem)]

Not sure if this is related, but in the ceph job I see that rbd_secret_uuid is set in nova.conf:

http://logs.openstack.org/99/189799/2/check/check-tempest-dsvm-full-ceph/353db23/logs/etc/nova/nova.conf.txt.gz

rbd_secret_uuid = 279b555f-cc0f-4a58-9e1d-6137a66a78df

But it's not set in cinder.conf in the [ceph] group:

http://logs.openstack.org/99/189799/2/check/check-tempest-dsvm-full-ceph/353db23/logs/etc/cinder/cinder.conf.txt.gz

[ceph]
rbd_max_clone_depth = 5
rbd_flatten_volume_from_snapshot = False
rbd_uuid = 279b555f-cc0f-4a58-9e1d-6137a66a78df
rbd_user = cinder
rbd_pool = volumes
rbd_ceph_conf =
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph

[Matt Riedemann (mriedem)]

From the docs here (which are a bit old, goes up to Juno):

http://ceph.com/docs/master/rbd/rbd-openstack/

Sounds like rbd_secret_uuid should also be set in cinder.conf, but I'm not sure if cephx authentication is enabled in this job or not.

[Josh Durgin (jdurgin)]

rbd_secret_uuid is just an identifier for looking up ceph's auth key in libvirt. It is unrelated to encryption.

[Matt Riedemann (mriedem)]

I noticed in the logging of the nova debug change related to this that the volume coming back to nova has encrypted=True but the connection_info['data'] dict doesn't:

With this change I noticed that nothing is being logged anymore, yet the warning above wasn't being logged either.

Apparently that's because the connection_info['data'] dict doesn't contain the 'encrypted' key in the ceph case:

http://logs.openstack.org/99/189799/2/check/check-tempest-dsvm-full-ceph/353db23/logs/screen-n-cpu.txt.gz#_2015-06-09_18_21_16_723

However, the volume does have the encrypted value set to True:
http://logs.openstack.org/99/189799/2/check/check-tempest-dsvm-full-ceph/353db23/logs/screen-n-cpu.txt.gz#_2015-06-09_18_21_16_217

So it seems we have some issues with the connection_info coming back from the cinder API after initializing the connection, cinder doesn't thing the volume is encrypted - maybe in the ceph case it's not, hence the bug.

[Matt Riedemann (mriedem)]

I see the issue in the cinder rbd driver, comparing it to the lvm iscsi module's initialize_connection method, that sets the 'encrypted' value in the connection_info['data'] properties returned:

https://github.com/openstack/cinder/blob/master/cinder/volume/targets/iscsi.py#L153

The rbd volume driver in cinder doesn't do that, so nova doesn't see it and try to use it. I'm not sure why the rbd driver in cinder doesn't do that, but setting it to true will trigger nova to get the encryption metadata and try to use the cryptsetup/luks modules in nova and that will probably blow up.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/189799
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c13588517a270bd8349c084c1f210598917180d4
Submitter: Jenkins
Branch: master

commit c13588517a270bd8349c084c1f210598917180d4
Author: Daniel P. Berrange <email address hidden>
Date: Tue Jun 9 17:08:20 2015 +0100

    volume: log which encryptor class is being used

    After resolving which encryptor class to use with a cinder
    volume, record it in the logs for easier debugging.

    Related-Bug: #1463525

    Change-Id: Ic753b11a48cb2c17c24c6636a456f00966f7c8ab

[Zoltan Arnold Nagy (zoltan)]

I made the rbd driver give back the encryption data and indeed the encryption tests blow up in nova:

http://logs.openstack.org/53/190053/2/check/check-tempest-dsvm-full-ceph/69019dc/console.html#_2015-06-10_10_52_15_110

Will chat on IRC later on how to fix this in nova.

[Daniel Berrange (berrange)]

FWIW, the fact that the tempest encryption job is currently passing on Ceph shows a serious bug in the cinder APIs, and possibly nova too.

If the client has requested encrypted storage, and the volume driver in cinder cannot support that, then it is unacceptable to simply ignore the encryption request - we should be reporting that back as a fatal error and /not/ continuing unencrypted.

Similarly if cinder reports encryption for a volume, and nova is unable to use that, we must ensure Nova reports an error at VM boot time, or volume attach.

[Zoltan Arnold Nagy (zoltan)]

What would be the best way to communicate from Nova's side that for the given hypervisor with the given cinder volume driver encryption support is not implemented?

If we were to make the code path fail if someone requests encrypted volumes but the cinder side doesn't supply the required metadata that would basically block the gate as -ceph is voting?

[Matt Riedemann (mriedem)]

I have a related cinder change here to pass the encrypted flag back in connection_info from the rbd volume driver:

https://review.openstack.org/#/c/193673/

I expect that to make nova fail in the ceph job.

[Matt Riedemann (mriedem)]

This is what it looks like when cinder says the volume is encrypted but nova isn't ready to handle it and doesn't set the 'device_path' key in the connection_info dict:

http://logs.openstack.org/73/193673/1/check/check-tempest-dsvm-full-ceph/f296b16/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-06-19_19_49_34_821

[Matt Riedemann (mriedem)]

2015-06-19 19:49:34.821 ERROR nova.volume.encryptors [req-b49ee6e6-4938-4838-875b-d89c66788619 TestEncryptedCinderVolumes-960229319 TestEncryptedCinderVolumes-401534610] Error instantiating nova.volume.encryptors.cryptsetup.CryptsetupEncryptor: 'device_path'
2015-06-19 19:49:34.821 ERROR nova.virt.libvirt.driver [req-b49ee6e6-4938-4838-875b-d89c66788619 TestEncryptedCinderVolumes-960229319 TestEncryptedCinderVolumes-401534610] [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397] Failed to attach volume at mountpoint: /dev/vdb
2015-06-19 19:49:34.821 1027 ERROR nova.virt.libvirt.driver [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397] Traceback (most recent call last):
2015-06-19 19:49:34.821 1027 ERROR nova.virt.libvirt.driver [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1081, in attach_volume
2015-06-19 19:49:34.821 1027 ERROR nova.virt.libvirt.driver [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397] encryption)
2015-06-19 19:49:34.821 1027 ERROR nova.virt.libvirt.driver [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1032, in _get_volume_encryptor
2015-06-19 19:49:34.821 1027 ERROR nova.virt.libvirt.driver [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397] **encryption)
2015-06-19 19:49:34.821 1027 ERROR nova.virt.libvirt.driver [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397] File "/opt/stack/new/nova/nova/volume/encryptors/__init__.py", line 48, in get_volume_encryptor
2015-06-19 19:49:34.821 1027 ERROR nova.virt.libvirt.driver [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397] **kwargs)
2015-06-19 19:49:34.821 1027 ERROR nova.virt.libvirt.driver [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 38, in import_object
2015-06-19 19:49:34.821 1027 ERROR nova.virt.libvirt.driver [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397] return import_class(import_str)(*args, **kwargs)
2015-06-19 19:49:34.821 1027 ERROR nova.virt.libvirt.driver [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397] File "/opt/stack/new/nova/nova/volume/encryptors/cryptsetup.py", line 38, in __init__
2015-06-19 19:49:34.821 1027 ERROR nova.virt.libvirt.driver [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397] self.symlink_path = connection_info['data']['device_path']
2015-06-19 19:49:34.821 1027 ERROR nova.virt.libvirt.driver [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397] KeyError: 'device_path'
2015-06-19 19:49:34.821 1027 ERROR nova.virt.libvirt.driver [instance: f6474d57-6e1c-425a-b34d-25f7d3baa397]

[Matt Riedemann (mriedem)]

Related nova change which handles the cinder change to set the encrypted key in connection_info for the rbd volume driver:

https://review.openstack.org/#/c/193830/

^ changes the KeyError to a VolumeEncryptionNotSupported exception.

[Matt Riedemann (mriedem)]

Tempest change to make the encrypted cinder volume scenario test configurable (supported or not):

https://review.openstack.org/#/c/193831/

[Matt Riedemann (mriedem)]

Related devstack change to configure tempest: https://review.openstack.org/#/c/193834/

Related devstack-gate change to configure devstack: https://review.openstack.org/#/c/193835/

[Ramy Asselin (ramy-asselin)]

Our team ran into this issue a while back and reported it..I think it was marked invalid. Looking for the number to link it. It affects all cinder drivers.

[Ramy Asselin (ramy-asselin)]

Found the bug: https://bugs.launchpad.net/tempest/+bug/1440227

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/193673
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=ba6f4037ab982b3bcf2de48cb51f919b3f37cbaf
Submitter: Jenkins
Branch: master

commit ba6f4037ab982b3bcf2de48cb51f919b3f37cbaf
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 19 11:25:43 2015 -0700

    Set encrypted key in connection_info during initialize

    There are encrypted volume tests in Tempest which are passing
    in the ceph job but the encryption providers in nova (luks and
    cryptsetup) are not actually being called in nova because the
    'encrypted' key isn't being set in connection_info for the Rbd
    volume driver, and nova is keying off that value to determine
    if it needs to run it's encryption providers.

    So the tests are passing in the ceph job but it's a total false
    positive and the API should actually fail - either the cinder
    API to create the encrypted volumes if the Rbd volume driver in
    Cinder doesn't support encryption, or the volume attach call in
    nova if it can't encrypt the connected volume, either way the
    test should fail until that support is baked into nova.

    I'm not aware of any kind of 'supports_encryption' or 'encryptable'
    capability flag for cinder volume drivers, but that might be
    needed for the volume create API to fail if trying to create a
    volume from an encrypted volume type where the volume driver
    itself doesn't support encryption - but maybe it's only a nova
    problem since nova has the encryption providers.

    This is fixed generically in the VolumeManager's
    initialize_connection method where we check for the encrypted
    flag being set in the volume driver method and if not, the
    manager sets it based on the volume.encryption_key_id value.
    We do this globally since there are other volume drivers besides
    rbd that aren't setting the encrypted key in connection_info,
    but we test rbd in the gate via the ceph job so most of the
    bug tracking is around validation of the ceph job.

    Related Nova change: I8efc2628b09d4e9e59831353daa080b20e17ccde

    Depends-On: I8548d41095513b9e669f773e3f35353e9228ead9

    DocImpact: Attaching 'encrypted' RBD volumes to a Nova server
               instance created from the libvirt virt driver will fail
               since RBD volume encryption is not currently supported in
               Nova's libvirt driver.

    Closes-Bug: #1440227
    Related-Bug: #1463525

    Change-Id: I03f8cae05cc117e14f7482115de685fc9f3fa54a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/206576

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/193830
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ac8a7e91e33abe4089cafb2ffa0509b5ce02039c
Submitter: Jenkins
Branch: master

commit ac8a7e91e33abe4089cafb2ffa0509b5ce02039c
Author: Matt Riedemann <email address hidden>
Date: Sat Jun 20 13:33:06 2015 -0700

    Handle KeyError when volume encryption is not supported

    When attaching a volume, after the connection is initialized to the
    volume in Cinder, the nova.volume.encryptors.get_encryption_metadata
    method is called to get encryption metadata for the volume. That call is
    based on the 'encrypted' key in connection_info['data'] returned from
    the os-initialize_connection Cinder API.

    However, just because the volume has an encryption key in Cinder does
    not mean that the corresponding volume driver in Nova supports
    encrypting the volume, like in the case of RBD volumes.

    Tempest has tests for encrypted volumes which succeed today in the Ceph
    job but they are actually false positives since without Cinder change
    I03f8cae05cc117e14f7482115de685fc9f3fa54a, the 'encrypted' key is not
    set in the connection_info dict and Nova doesn't attempt encryption of
    the volume during attach.

    The Ceph job fails when encrypted=True is in connection_info because
    cryptsetup (and luks which extends cryptsetup) requires the
    'device_path' key in the connection_info dict, which is set when
    connecting the volume during attach via the corresponding Nova volume
    driver. In the case of RBD and libvirt, the LibvirtNetVolumeDriver is
    used and the 'device_path' key isn't set, so a KeyError is raised when
    trying to construct the CryptsetupEncryptor or LuksEncryptor objects.

    This change adds a check in CryptsetupEncryptor such that if the
    device_path is not in connection_info, a VolumeEcnryptionNotSupported
    error is raised rather than KeyError.

    Note that this doesn't fix the encrypted volume tests in Tempest. Those
    tests fail due to a timeout waiting for the volume status to be 'in-use'
    which doesn't happen since the compute manager rolls back the
    reservation on the volume when the error occurs. The Tempest tests will
    have to be skipped in the Ceph job until volume encryption is supported
    for RBD in Nova, which will be a separate set of changes.

    Related-Bug: #1463525

    Change-Id: I8efc2628b09d4e9e59831353daa080b20e17ccde

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/206576
Reason: The spec isn't going to be approved for mitaka, we're waiting for qemu native rbd encryption so might as well abandon this.

[Sean McGinnis (sean-mcginnis)]

This appears to have been fixed on the Cinder side with Matt's Cinder patch referenced above. If there is more work to do on the Cinder side please reopen.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/534811

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/534811
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=fcb45b439ba039fd88c332fd912949d52cfe290f
Submitter: Zuul
Branch: master

commit fcb45b439ba039fd88c332fd912949d52cfe290f
Author: Eric Harney <email address hidden>
Date: Wed Jan 17 20:50:25 2018 -0500

    RBD: Support encrypted volumes

    When creating an encrypted RBD volume, initialize
    LUKS on the volume using the volume's encryption key.

    This is required because os-brick only handles this
    step for volumes that attach via block devices.

    This requires qemu-img 2.10.

    Co-Authored-By: Lee Yarwood <email address hidden>
    Related-Bug: #1463525
    Implements: blueprint libvirt-qemu-native-luks
    Change-Id: Id02130e9af8bdf90a712968916017d05c3213c32