Cinder Volume can't attach to Instance on different host

[Sam Betts]

System fails to attach and returns to available state if volume and instance are on separate physical hosts.

[Mark T. Voelker (mvoelker)]

Can you provide log messages so we can debug?

[Sam Betts (sambetts)]

In this instance nothing seems to be written to any logs, it just does nothing. I have checked the cinder and nova logs on the control and on the compute hosts I'm trying to attach, is there anywhere else that something like this might be logged?

[subconscious (subconscious)]

Sam, can you paste your cinder.conf please?

[Mark T. Voelker (mvoelker)]

Or, better: attach it as a file instead of pasting it? =)

[Shweta P (shweta-ap05)]

There is a blueprint being worked on to address this issue
https://blueprints.launchpad.net/cinder/+spec/local-storage-volume-scheduling

[Shweta P (shweta-ap05)]

Actually ignore the above blueprint, it is not addressing the same issue, they are trying to provide an option to attach a volume to an instance while creating it.

I am seeing the issue stated in the bug as well, volumes can be attached to instances on the same compute host but not instances on a different compute host. I will provide the nova-compute.log shortly.

[Shweta P (shweta-ap05)]

2014-02-25 21:44:42.640 1659 WARNING nova.virt.libvirt.volume [req-b52a45a5-53d5-408b-b5d9-b30c8e01daaf 374357d0b5a847f9ab79cac6220f868a 5ff25f461aa0437db2db67af236d1ddc] ISCSI volume not yet found at: vdc. Will rescan & retry. Try number: 0
2014-02-25 21:44:42.695 1659 ERROR nova.compute.manager [req-b52a45a5-53d5-408b-b5d9-b30c8e01daaf 374357d0b5a847f9ab79cac6220f868a 5ff25f461aa0437db2db67af236d1ddc] [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] Failed to attach volume 28a2cdb2-8ec9-4b23-ab4b-51572b69ed7c at /dev/vdc
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] Traceback (most recent call last):
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3687, in _attach_volume
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] encryption=encryption)
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1083, in attach_volume
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] disk_info)
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1042, in volume_driver_method
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] return method(connection_info, *args, **kwargs)
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 246, in inner
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] return f(*args, **kwargs)
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 285, in connect_volume
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] self._run_iscsiadm(iscsi_properties, ("--rescan",))
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 218, in _run_iscsiadm
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] check_exit_code=check_exit_code)
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 177, in execute
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] return processutils.execute(*cmd, **kwargs)
2014-02-25 21:44:42.695 1659 TRACE nova.compute.manager [instance: 9af3cff4-c8d9-472c-b99a-4c2356e59503] File "/usr/lib/...

[Shweta P (shweta-ap05)]

I did see a warning earlier that said no systool found. I installed the sysfutils package as the docs suggested and restarted the cinder and the nova services. That didnt help.

What should iscsi_ip_address be set too? Its set to 0.0.0.0 in the setup by default. Could this be a problem?

[subconscious (subconscious)]

https://review.openstack.org/#/c/101589/

tested on 2role and AIO

[Mark T. Voelker (mvoelker)]

stable/havana:
https://review.openstack.org/#/c/102879/