encrypted iSCSI volume attach fails to attach when multipath-tools installed

[Anthony Lee]

An error was occurring in a devstack setup with nova version:

commit ab25f5f34b6ee37e495aa338aeb90b914f622b9d
Merge "instance termination with update_dns_entries set fails"

A volume-type encrypted with CryptsetupEncryptor was being used. A volume was created using this volume-type and an attempt to attach it to an instance was made. This error also occurred when using the LuksEncryptor for the volume-type.

The following error occurred in n-cpu during attachment:

Stack Trace:

2015-04-02 13:39:54.397 ERROR nova.virt.block_device [req-a8220e7d-8d1e-459d-be1f-4ddd65b7ff66 admin admin] [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Driver failed to attach volume 81c5f69a-9b01-4f
c0-a105-be9d3c966aaf at /dev/vdb
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Traceback (most recent call last):
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/opt/stack/nova/nova/virt/block_device.py", line 251, in attach
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] device_type=self['device_type'], encryption=encryption)
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1064, in attach_volume
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] self._disconnect_volume(connection_info, disk_dev)
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] six.reraise(self.type_, self.value, self.tb)
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1051, in attach_volume
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] encryptor.attach_volume(context, **encryption)
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/opt/stack/nova/nova/volume/encryptors/cryptsetup.py", line 93, in attach_volume
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] self._open_volume(passphrase, **kwargs)
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/opt/stack/nova/nova/volume/encryptors/cryptsetup.py", line 78, in _open_volume
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] check_exit_code=True, run_as_root=True)
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/opt/stack/nova/nova/utils.py", line 206, in execute
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] return processutils.execute(*cmd, **kwargs)
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 233, in execute
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] cmd=sanitized_cmd)
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] ProcessExecutionError: Unexpected error while running command.
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup create --key-file=- ip-10.50.3.20:3260-
iscsi-iqn.2003-10.com.lefthandnetworks:vsa-12-721:853:volume-81c5f69a-9b01-4fc0-a105-be9d3c966aaf-lun-0 /dev/sdb
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Exit code: 5
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Stdout: u''
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5] Stderr: u'Cannot use device /dev/sdb which is in use (already mapped or mounted).\n'
2015-04-02 13:39:54.397 TRACE nova.virt.block_device [instance: 41d0c192-a1ce-45eb-a5ff-bcb96ec0d8e5]

After looking into reasons why the device would be busy it was found that uninstalling multipath-tools seemed to fix the problem. Multipath-tools appears to keep the device busy even if not used.

iscsi_use_multipath was not set to True in nova.conf.

Steps to reproduce:

install multipath-tools and wait for multipath daemon service to start.
attempt to attach an encrypted volume to an instance

The above error should occur.

uninstall multipath-tools and try to attach the volume again and it should work properly.

[Walt Boring (walter-boring)]

Thanks for filing this one Anthony. This is basically user error and invalid.

If you have multipath-tools installed and running on a system, you really need to enable it in nova.conf and cinder.conf (if this is a cinder controller as well).

invalid bug.

[Richard Hedlind (richard-hedlind)]

Walt,
I still see this when multipath-tools are installed and iscsi_use_multipath is set to True in both nova.conf and cinder.conf

[Walt Boring (walter-boring)]

Richard, can you please post the log?

If multipath is running and enabled in nova.conf, then nova should be using the multipath device, not the raw /dev/sdX device.

[Ramy Asselin (ramy-asselin)]

Richard, take a look at this related bug too: https://bugs.launchpad.net/nova/+bug/1439869

[Duncan Thomas (duncan-thomas)]

Marked as 'incomplete' rather than 'invalid' since we're waiting on logs to confirm one way or another

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]