encrypted iSCSI volume attach fails when iscsi_use_multipath is enabled

[Anthony Lee]

When attempting to attach an encrypted iSCSI volume to an instance with iscsi_use_multipath set to True in nova.conf an error occurs in n-cpu.

The devstack system being used had the following nova version:

commit ab25f5f34b6ee37e495aa338aeb90b914f622b9d
Merge "instance termination with update_dns_entries set fails"

The following error occurs in n-cpu:

Stack Trace:

2015-04-02 13:46:22.641 ERROR nova.virt.block_device [req-61f49ff8-b814-42c0-8cf8-ffe7b6a3561c admin admin] [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Driver failed to attach volume 4778e71c-a1b5-4d
b5-b677-1d8191468e87 at /dev/vdb
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Traceback (most recent call last):
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/opt/stack/nova/nova/virt/block_device.py", line 251, in attach
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] device_type=self['device_type'], encryption=encryption)
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1064, in attach_volume
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] self._disconnect_volume(connection_info, disk_dev)
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] six.reraise(self.type_, self.value, self.tb)
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1051, in attach_volume
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] encryptor.attach_volume(context, **encryption)
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/opt/stack/nova/nova/volume/encryptors/cryptsetup.py", line 93, in attach_volume
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] self._open_volume(passphrase, **kwargs)
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/opt/stack/nova/nova/volume/encryptors/cryptsetup.py", line 78, in _open_volume
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] check_exit_code=True, run_as_root=True)
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/opt/stack/nova/nova/utils.py", line 206, in execute
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] return processutils.execute(*cmd, **kwargs)
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 233, in execute
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] cmd=sanitized_cmd)
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] ProcessExecutionError: Unexpected error while running command.
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup create --key-file=- 36000eb37601bcf0200
0000000000036c /dev/mapper/36000eb37601bcf02000000000000036c
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Exit code: 1
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Stdout: u''
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Stderr: u''
2015-04-02 13:46:22.641 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]

multipath-tools was installed
iscsi_use_multipath = True was set under the [libvirt] entry in nova.conf

To reproduce:

install multipath-tools
set iscsi_use_multipath to True in nova.conf under the [libvirt] section.
create an encrypted iSCSI volume and attempt to attach it to an instance (I was using CryptsetupEncryptor).

The above error will occur in n-cpu.

Possibly related bug: https://bugs.launchpad.net/nova/+bug/1439855

[Sean Dague (sdague)]

It appears that the local device name is completely bonkers in this case.

sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup create --key-file=- 36000eb37601bcf0200
0000000000036c /dev/mapper/36000eb37601bcf02000000000000036c

Is just the wrong command entirely to be useful.

[ugvddm (271025598-9)]

Crypsetup returns 0 on success or non-zero on error.

       Error codes are: 1 wrong parameters, 2 no permission (bad passphrase),
       3 out of memory, 4 wrong device specified, 5 device already exists or
       device is busy.

http://manpages.ubuntu.com/manpages/precise/man8/cryptsetup.8.html

[Richard Hedlind (richard-hedlind)]

I have a possible fix for this issue. Is it ok if I assign the bug to myself and push up a proposed fix?

[Tomoki Sekiyama (tsekiyama)]

I've proposed for this fix:

https://review.openstack.org/#/c/196482/

I appreciate if you could review this.

[Richard Hedlind (richard-hedlind)]

Tomoki,
I just gave your fix a +1. Hopefully we can get some +2s on this. Such a small fix.

Not sure why launchpad hasn't updated this bug report with a link to your proposed fix?

Richard

[Sivasathurappan Radhakrishnan (siva-radhakrishnan)]

@Tomoki Sekiyama can you please let me know if you are still working on the patch that you proposes as it is in merge conflict right now ?

[Tomoki Sekiyama (tsekiyama)]

Sorry for this left in merge conflict. I have rebased the change to fix the conflict. Thanks.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/196482
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=89a61ab8f4602e018763afb173e1a862f151a222
Submitter: Jenkins
Branch: master

commit 89a61ab8f4602e018763afb173e1a862f151a222
Author: Tomoki Sekiyama <email address hidden>
Date: Thu Jun 25 17:41:47 2015 -0400

    Fix multipath iSCSI encrypted volume attach failure

    Currently iSCSI volume attachment fails if iscsi_use_multipath is
    set to True. This is because the encryptor requests cryptsetup
    to create the symlink to the LUKS device with the same name of
    the device-mapper multipath device. To avoid the name collision,
    this patch adds the 'crypt-' prefix to the symlink.

    Change-Id: I01a3104025a9c479e40933ca24024355df9dfb8b
    Closes-Bug: #1439869

[Lisa Li (lisali)]

As Cinder will use encryptors, nova/encryptors is copied to os-brick. We also need to import this fix into os-brick.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/338017

[Lisa Li (lisali)]

Tom, I introduced the fix to os-brick. Please help to review it: https://review.openstack.org/338017

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/338017
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=9d2bb5e15de92c72cc50bac2ee5d0b3b2704ed1e
Submitter: Jenkins
Branch: master

commit 9d2bb5e15de92c72cc50bac2ee5d0b3b2704ed1e
Author: lisali <email address hidden>
Date: Wed Jul 6 13:48:20 2016 +0800

    Fix multipath iSCSI encrypted volume attach failure

    This is to copy fix of bug 1439869 from Nova to os-brick.

    Currently iSCSI volume attachment fails if iscsi_use_multipath is
    set to True. This is because the encryptor requests cryptsetup
    to create the symlink to the LUKS device with the same name of
    the device-mapper multipath device. To avoid the name collision,
    this patch adds the 'crypt-' prefix to the symlink.

    Change-Id: Ia001204df9b14f635ab998590e8add119c1aec23
    Closes-Bug: #1439869

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 14.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lee Yarwood (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/349898

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/349897
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=82190bdd283dda37f7517fd9a268b5e55183f06c
Submitter: Jenkins
Branch: stable/mitaka

commit 82190bdd283dda37f7517fd9a268b5e55183f06c
Author: Tomoki Sekiyama <email address hidden>
Date: Thu Jun 25 17:41:47 2015 -0400

    Fix multipath iSCSI encrypted volume attach failure

    Currently iSCSI volume attachment fails if iscsi_use_multipath is
    set to True. This is because the encryptor requests cryptsetup
    to create the symlink to the LUKS device with the same name of
    the device-mapper multipath device. To avoid the name collision,
    this patch adds the 'crypt-' prefix to the symlink.

    NOTE(lyarwood): The rootwrap.d portion of this change is dropped for
    stable/mitaka to avoid upgrade issues. The current wildcard based ln
    filter means that the remainder of the change will still correct this
    issue.

    Conflicts:
            etc/nova/rootwrap.d/compute.filters
            nova/tests/unit/volume/encryptors/test_cryptsetup.py

    Change-Id: I01a3104025a9c479e40933ca24024355df9dfb8b
    Closes-Bug: #1439869
    (cherry picked from commit 89a61ab8f4602e018763afb173e1a862f151a222)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/os-brick 1.5.0 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 13.1.1 release.