NFS LUKSv1 volume creation from an image incorrectly reformats volume as plain LUKSv1 instead of an encrypted QCOW2 volume

[Lee Yarwood]

I've been playing around with LUKSv1 encrypted NFS volumes today after I was asked if Ia255a25da081e7ac42b4caba7a4cbd8d76ac5848 would also extend encrypted NFS volumes and have stumbled across several major issues with the recently landed implementation.

At present when creating an encrypted LUKSv1 NFS volume we create a LUKSv1 encrypted QCOW2 file:

Jul 23 09:48:19 localhost.localdomain cinder-volume[179023]: DEBUG oslo_concurrency.processutils [None req-378df225-2051-4dbf-ad69-0a6e1b020c38 admin None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img create -f qcow2 -o encrypt.format=luks,encrypt.key-secret=*** --object secret,id=sec1,format=raw,file=/opt/stack/data/cinder/conversion/tmpjar1igr5 /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-9a9d8993-5594-4618-a259-78fa0f920633 1073741824" returned: 0 in 5.251s {{(pid=179042) execute /usr/local/lib/python3.7/site-packages/oslo_concurrency/processutils.py:416}}

This is fine however when creating the volume from an image we then go on to attach the os-brick provided cryptsetup encryptors that attempt to open the device as if it were a plain LUKSv1 encrypted device:

Jul 23 09:48:21 localhost.localdomain cinder-volume[179023]: Command: cryptsetup luksOpen --key-file=- /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-9a9d8993-5594-4618-a259-78fa0f920633 crypt-volume-9a9d8993-5594-4618-a259-78fa0f920633
Jul 23 09:48:21 localhost.localdomain cinder-volume[179023]: Exit code: 1

As the file is actually QCOW2 this fails and the encryptors then reformat the file as a plain LUKSv1 file:

Jul 23 09:48:21 localhost.localdomain cinder-volume[179023]: INFO os_brick.encryptors.luks [None req-378df225-2051-4dbf-ad69-0a6e1b020c38 admin None] /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-9a9d8993-5594-4618-a259-78fa0f920633 is not a valid LUKS device; formatting device for first use
[..]
Jul 23 09:48:25 localhost.localdomain cinder-volume[179023]: DEBUG oslo_concurrency.processutils [-] CMD "cryptsetup --batch-mode luksFormat --type luks1 --key-file=- --cipher aes-xts-plain64 --key-size 256 /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-9a9d8993-5594-4618-a259-78fa0f920633" returned: 0 in 4.256s {{(pid=237616) execute /usr/local/lib/python3.7/site-packages/oslo_concurrency/processutils>
Jul 23 09:48:25 localhost.localdomain cinder-volume[179023]: DEBUG oslo.privsep.daemon [-] privsep: reply[139834412181648]: (4, ('', '')) {{(pid=237616) _call_back /usr/local/lib/python3.7/site-packages/oslo_privsep/daemon.py:475}}

At this point the volume is no longer QCOW2 as we've basically done the following:

# qemu-img create -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec1 --object secret,id=sec1,format=raw,data=foo test.img 100M
Formatting 'test.img', fmt=qcow2 size=104857600 encrypt.format=luks encrypt.key-secret=sec1 cluster_size=65536 lazy_refcounts=off refcount_bits=16

# qemu-img info test.img
image: test.img
file format: qcow2
virtual size: 100 MiB (104857600 bytes)
disk size: 480 KiB
encrypted: yes
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    encrypt:
        ivgen alg: plain64
        hash alg: sha256
        cipher alg: aes-256
        uuid: d0363765-0c52-462e-a4e7-c8b3f19b896b
        format: luks
        cipher mode: xts
        slots:
            [0]:
                active: true
                iters: 1881118
                key offset: 4096
                stripes: 4000
            [1]:
                active: false
                key offset: 262144
            [2]:
                active: false
                key offset: 520192
            [3]:
                active: false
                key offset: 778240
            [4]:
                active: false
                key offset: 1036288
            [5]:
                active: false
                key offset: 1294336
            [6]:
                active: false
                key offset: 1552384
            [7]:
                active: false
                key offset: 1810432
        payload offset: 2068480
        master key iters: 474321
    corrupt: false

# cryptsetup --batch-mode luksFormat --type luks1 --key-file=- --cipher aes-xts-plain64 --key-size 256 test.img
Enter passphrase for test.img:
# qemu-img info test.img
image: test.img
file format: luks
virtual size: 256 KiB (262144 bytes)
disk size: 2.03 MiB
encrypted: yes
Format specific information:
    ivgen alg: plain64
    hash alg: sha256
    cipher alg: aes-128
    uuid: d325d8e2-ea2f-45ca-acde-be4512ba7b8a
    cipher mode: xts
    slots:
        [0]:
            active: true
            iters: 4009850
            key offset: 4096
            stripes: 4000
        [1]:
            active: false
            key offset: 135168
        [2]:
            active: false
            key offset: 266240
        [3]:
            active: false
            key offset: 397312
        [4]:
            active: false
            key offset: 528384
        [5]:
            active: false
            key offset: 659456
        [6]:
            active: false
            key offset: 790528
        [7]:
            active: false
            key offset: 921600
    payload offset: 2097152
    master key iters: 250137

Note that the file now reports a format of RAW and an incorrect virtual size. This breaks volume attachment to running instances *and* volume creation from images, both of which appear untested in Tempest at the moment for LUKSv1 volumes.

[Lee Yarwood (lyarwood)]

FWIW we don't need to attach the encryptors for QCOW2 LUKSv1 files, QEMU should be able to convert the provided image into QCOW2 LUKSv1 given the correct secrets etc. I'll work up a change now.

[Lee Yarwood (lyarwood)]

I just attempted to reproduce this and it's even more broken than I originally thought.

While we hit c#0 when creating the image based encrypted NFS volume we then also see os-brick overwriting the entire original file with a symlink to the decrypted dm-crypt device:

$ openstack volume type create --encryption-provider nova.volume.encryptors.luks.LuksEncryptor --encryption-cipher aes-xts-plain64 --encryption-key-size 256 --encryption-control-location front-end LUKS
$ openstack volume create --image cirros-0.5.1-x86_64-disk --type LUKS --size 1 test-encrypted-image
[..]
| id | 9b3a4f89-4194-4972-9292-b60c97c824e2 |
[..]

$ ll /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-*
lrwxrwxrwx. 1 root root 61 Aug 21 08:24
/opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-9b3a4f89-4194-4972-9292-b60c97c824e2
-> /dev/mapper/crypt-volume-9b3a4f89-4194-4972-9292-b60c97c824e2

^ removing the original encrypted file entirely, so before fixing c#0 we also need to resolve this.

[Lee Yarwood (lyarwood)]

Apologies, I forgot to add the following showing the dm-crypt device has been removed:

$ ll /dev/mapper/crypt-volume-9b3a4f89-4194-4972-9292-b60c97c824e2
ls: cannot access '/dev/mapper/crypt-volume-9b3a4f89-4194-4972-9292-b60c97c824e2': No such file or directory

[Eric Harney (eharney)]

This is a deficiency in the NFS driver w.r.t. encrypted volumes, to fix it we should implement a driver copy_image_to_encrypted_volume() method to handle this path. Without it, NFS is currently falling into the base driver generic copy_image_to_encrypted_volume() method which does not work for how NFS managed encrypted volumes.

This method for the NFS driver should mirror how the RBD driver's copy_image_to_encrypted_volume works:
- download the image to a temp file
- call something similar to the RBD driver's _encrypt_image() method which calls image_utils.convert_image() etc to encrypt the image and copy it into the volume file.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/749155