Nova volume encryptors attach volume fails for NFS and FC (rootwrap)

[Tom Barron]

Tempest scenario TestEncryptedCinderVolumes has been silently skipped when run with NFS cinder drivers that did not
set the 'encrypted' key in the connection_info['data'] dict in their initialize_connection methods. Change
https://review.openstack.org/#/c/193673/ - which sets the encrypted flag generically, in the VolumeManager's
initialize_connection, on the basis of the volume.encryption_key_id value - causes this test to actually run its encryption
providers and exposes a problem in LuksEncryptor:attach_volume() for NFS exported volumes.

At https://github.com/openstack/nova/blob/master/nova/volume/encryptors/luks.py#L119 we have:

        # modify the original symbolic link to refer to the decrypted device
        utils.execute('ln', '--symbolic', '--force',
                      '/dev/mapper/%s' % self.dev_name, self.symlink_path,
                      run_as_root=True, check_exit_code=True)

but in TestEncryptedCinderVolumes we get the following exception:

2015-06-29 06:44:06.353 DEBUG oslo_concurrency.processutils [req-35a458fe-8bfc-4570-ac8e-388e8b74d4ea TestEncryptedCinderVolumes-1523565967 TestEncryptedCinderVolumes-1577400956] u'sudo nova-rootwrap /etc/nova/rootwrap.conf ln --symbolic --force /dev/mapper/volume-f5684ecc-959f-4de8-8d62-a8adf4bdb4cc /opt/stack/data/nova/mnt/21dd48babac42ae884d1192b8697a041/volume-f5684ecc-959f-4de8-8d62-a8adf4bdb4cc' failed. Not Retrying. execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:293
2015-06-29 06:44:06.353 ERROR nova.virt.libvirt.driver [req-35a458fe-8bfc-4570-ac8e-388e8b74d4ea TestEncryptedCinderVolumes-1523565967 TestEncryptedCinderVolumes-1577400956] [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] Failed to attach volume at mountpoint: /dev/vdb
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] Traceback (most recent call last):
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1082, in attach_volume
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] encryptor.attach_volume(context, **encryption)
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] File "/opt/stack/new/nova/nova/volume/encryptors/luks.py", line 121, in attach_volume
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] run_as_root=True, check_exit_code=True)
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] File "/opt/stack/new/nova/nova/utils.py", line 229, in execute
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] return processutils.execute(*cmd, **kwargs)
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 260, in execute
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] cmd=sanitized_cmd)
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] ProcessExecutionError: Unexpected error while running command.
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ln --symbolic --force /dev/mapper/volume-f5684ecc-959f-4de8-8d62-a8adf4bdb4cc /opt/stack/data/nova/mnt/21dd48babac42ae884d1192b8697a041/volume-f5684ecc-959f-4de8-8d62-a8adf4bdb4cc
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] Exit code: 99
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] Stdout: u''
2015-06-29 06:44:06.353 13140 ERROR nova.virt.libvirt.driver [instance: b285fed7-6d65-4b57-9ab0-8c17ce0cf6a8] Stderr: u'/usr/local/bin/nova-rootwrap: Unauthorized command: ln --symbolic --force /dev/mapper/volume-f5684ecc-959f-4de8-8d62-a8adf4bdb4cc /opt/stack/data/nova/mnt/21dd48babac42ae884d1192b8697a041/volume-f5684ecc-959f-4de8-8d62-a8adf4bdb4cc (no filter matched)\n'

The cause is evidently the rootwrap filter at
https://github.com/openstack/nova/blob/master/etc/nova/rootwrap.d/compute.filters#L215, namely:

ln: RegExpFilter, ln, root, ln, --symbolic, --force, /dev/mapper/ip-.*-iscsi-iqn.*, /dev/disk/by-path/ip-.*-iscsi-iqn.*

which only allows for iscsi paths.

[Xing Yang (xing-yang)]

The test failed on the FC driver also because Nova was using the nova rootwrap filter for the iSCSI device.

2015-06-29 05:02:18.267 16351 ERROR nova.virt.libvirt.driver [instance: 9d62ca74-31ad-4691-af75-3c2e4f758c72] cmd=sanitized_cmd) 2015-06-29 05:02:18.267 16351 ERROR nova.virt.libvirt.driver [instance: 9d62ca74-31ad-4691-af75-3c2e4f758c72] ProcessExecutionError: Unexpected error while running command. 2015-06-29 05:02:18.267 16351 ERROR nova.virt.libvirt.driver [instance: 9d62ca74-31ad-4691-af75-3c2e4f758c72] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ln --symbolic --force /dev/mapper/pci-0000:06:00.0-fc-0x5006016508603f9f-lun-151 /dev/disk/by-path/pci-0000:06:00.0-fc-0x5006016508603f9f-lun-151 2015-06-29 05:02:18.267 16351 ERROR nova.virt.libvirt.driver [instance: 9d62ca74-31ad-4691-af75-3c2e4f758c72] Exit code: 99 2015-06-29 05:02:18.267 16351 ERROR nova.virt.libvirt.driver [instance: 9d62ca74-31ad-4691-af75-3c2e4f758c72] Stdout: u'' 2015-06-29 05:02:18.267 16351 ERROR nova.virt.libvirt.driver [instance: 9d62ca74-31ad-4691-af75-3c2e4f758c72] Stderr: u'/usr/local/bin/nova-rootwrap: Unauthorized command: ln --symbolic --force /dev/mapper/pci-0000:06:00.0-fc-0x5006016508603f9f-lun-151 /dev/disk/by-path/pci-0000:06:00.0-fc-0x5006016508603f9f-lun-151 (no filter matched)\n' 2015-06-29 05:02:18.267 16351 ERROR nova.virt.libvirt.driver [instance: 9d62ca74-31ad-4691-af75-3c2e4f758c72]

[Tom Barron (tpb)]

Same thing happens for test_encrypted_cinder_volumes_cryptsetup.

[Matt Riedemann (mriedem)]

Possibly related for the FC issue: https://review.openstack.org/#/c/195350/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/197713
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0facd1f89edfd88188662e0cac274a136a424c3d
Submitter: Jenkins
Branch: master

commit 0facd1f89edfd88188662e0cac274a136a424c3d
Author: Matt Riedemann <email address hidden>
Date: Wed Jul 1 12:09:49 2015 -0700

    rootwrap: update ln --symbolic filter for FS and FC type volume drivers

    Cinder change I03f8cae05cc117e14f7482115de685fc9f3fa54a sets the
    'encrypted' key for all cinder volume drivers connection_info. When run
    through the encrypted volume tests in Tempest, this hits the encryption
    providers in Nova that fail for certain types of volume drivers, like
    file system and fibre channel, due to the rootwrap filter not matching.

    This change updates the symbolic link rootwrap filter so it works with
    file system and fibre channel type volume backends rather than just
    iSCSI.

    The /dev/mapper/ prefix is always set in the encryptor modules, so that
    can remain as before.

    The symbolic link path is a complete wildcard, however, because the file
    system volume backends all have a configurable option for the mount path
    prefix, which defaults to $state_path/mnt but may not be that value.

    An example call for NFS:

    ln --symbolic --force \
    /dev/mapper/volume-f5684ecc-959f-4de8-8d62-a8adf4bdb4cc \
    /opt/stack/data/nova/mnt/21dd48babac42ae884d1192b8697a041/\
    volume-f5684ecc-959f-4de8-8d62-a8adf4bdb4cc

    An example call for fibre channel:

    ln --symbolic --force \
    /dev/mapper/pci-0000:06:00.0-fc-0x5006016508603f9f-lun-151 \
    /dev/disk/by-path/pci-0000:06:00.0-fc-0x5006016508603f9f-lun-151

    This change also updates the sg_info and sgscan entries to reference the
    correct module since those are not called from nova.virt.libvirt.volume
    anymore.

    Closes-Bug: #1470142
    Related-Bug: #1440227

    Change-Id: I181b594a3119f7ad74c595fc7059d521079b1d74

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/237582

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lee Yarwood (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/237582
Reason: Missed stable/kilo that is now security fixes only.