Detaching and then attaching a volume to a paused instance has no effect and reports no error

[Artem Plakunov]

Steps:
1. Create VM
2. Create volume
3. Attach volume to VM
4. Pause VM
5. Detach volume
6. Attach volume back (without unpausing the VM)

What happens is the volume enters the state "attaching" for a while but then becomes "available" again without being "in-use". Nova-compute and cinder-volume logs report nothing about it even with debug enabled

Nova version: 2014.2.1
I am aware of the bug https://bugs.launchpad.net/nova/+bug/1242366 but this is a bit different case and the release we use already have the fix installed

[jichenjc (jichenjc)]

are you using libvirt driver? is there any log in nova api layer?

please note that the attach is checked something like: at compute/api.py which is API layer code
looks like PAUSED instance should also be able to attach volume, but could you please help to check again?

3036 @check_instance_state(vm_state=[vm_states.ACTIVE, vm_states.PAUSED,
3037 vm_states.STOPPED, vm_states.RESIZED,
3038 vm_states.SOFT_DELETED])
3039 def attach_volume(self, context, instance, volume_id, device=None,
3040 disk_bus=None, device_type=None):

[Artem Plakunov (artacc)]

yes we are using libvirt

here is what I could find in nova-api log after I did: nova volume-attach 4ee06735-c3c0-4158-85da-8928cd851614 5b81412e-719f-4827-91ad-3323a5ee0a2c

2015-03-10 13:28:14.782 5395 DEBUG nova.api.openstack.wsgi [req-f846e1f9-f4f8-4255-80a7-5e074be33352 None] Action: 'create', calling method: <bound method VolumeAttachmentController.create of <nova.api.openstack.compute.contrib.volumes.VolumeAttachmentController object at 0x7f6d5eab6790>>, body: {"volumeAttachment": {"device": null, "volumeId": "5b81412e-719f-4827-91ad-3323a5ee0a2c"}} _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:934
2015-03-10 13:28:14.783 5395 AUDIT nova.api.openstack.compute.contrib.volumes [req-f846e1f9-f4f8-4255-80a7-5e074be33352 None] Attach volume 5b81412e-719f-4827-91ad-3323a5ee0a2c to instance 4ee06735-c3c0-4158-85da-8928cd851614 at None
2015-03-10 13:28:14.907 5395 DEBUG nova.volume.cinder [req-f846e1f9-f4f8-4255-80a7-5e074be33352 None] Cinderclient connection created using URL: http://controller01:8776/v1/ed9ef5f7ff4f49fe8cb6741b1383cd79 get_cinder_client_version /usr/lib/python2.7/dist-packages/nova/volume/cinder.py:255
2015-03-10 13:28:15.029 5395 DEBUG nova.volume.cinder [req-f846e1f9-f4f8-4255-80a7-5e074be33352 None] Cinderclient connection created using URL: http://controller01:8776/v1/ed9ef5f7ff4f49fe8cb6741b1383cd79 get_cinder_client_version /usr/lib/python2.7/dist-packages/nova/volume/cinder.py:255
2015-03-10 13:28:15.190 5395 INFO nova.osapi_compute.wsgi.server [req-f846e1f9-f4f8-4255-80a7-5e074be33352 None] 192.168.101.2 "POST /v2/ed9ef5f7ff4f49fe8cb6741b1383cd79/servers/4ee06735-c3c0-4158-85da-8928cd851614/os-volume_attachments HTTP/1.1" status: 200 len: 391 time: 0.4172301

I noticed that there is "attach to None" and tried again with path specified: nova volume-attach 9f47d3c7-656e-486c-b2c7-8bba6fa14545 9632ce57-5739-4285-98c6-6af3875439ea /dev/vdb

2015-03-10 13:42:13.711 5395 DEBUG nova.api.openstack.wsgi [req-000d1d78-bd99-4e97-8427-dd16c8e3f435 None] Action: 'create', calling method: <bound method VolumeAttachmentController.create of <nova.api.openstack.compute.contrib.volumes.VolumeAttachmentController object at 0x7f6d5eab6790>>, body: {"volumeAttachment": {"device": "/dev/vdb", "volumeId": "9632ce57-5739-4285-98c6-6af3875439ea"}} _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:934
2015-03-10 13:42:13.712 5395 AUDIT nova.api.openstack.compute.contrib.volumes [req-000d1d78-bd99-4e97-8427-dd16c8e3f435 None] Attach volume 9632ce57-5739-4285-98c6-6af3875439ea to instance 9f47d3c7-656e-486c-b2c7-8bba6fa14545 at /dev/vdb
2015-03-10 13:42:13.815 5395 DEBUG nova.volume.cinder [req-000d1d78-bd99-4e97-8427-dd16c8e3f435 None] Cinderclient connection created using URL: http://controller01:8776/v1/ed9ef5f7ff4f49fe8cb6741b1383cd79 get_cinder_client_version /usr/lib/python2.7/dist-packages/nova/volume/cinder.py:255
2015-03-10 13:42:13.950 5395 DEBUG nova.volume.cinder [req-000d1d78-bd99-4e97-8427-dd16c8e3f435 None] Cinderclient connection created using URL: http://controller01:8776/v1/ed9ef5f7ff4f49fe8cb6741b1383cd79 get_cinder_client_version /usr/lib/python2.7/dist-packages/nova/vol...

[Artem Plakunov (artacc)]

Also I checked compute/api.py
this exact code block is at 2924-2928 for me

[Jason Wong (wantwatering)]

What in my testing is that after detaching a volume from the paused VM, there is no available instance in Attach to instance drop down list.

Version of my env is code from stable/juno.

$ nova --version
2.20.0
$ cinder --version
1.1.1

[Artem Plakunov (artacc)]

Same in our horizon.
So looks like horizon does not allow this operation but nova client does.

[jichenjc (jichenjc)]

Not sure horizon's mechanism, but nova do allow this and some bugs used to be solved for it

can you attach logs instead of a chapter ? guess more info can be get from it .. thanks

[Liyingjun (liyingjun)]

Confirmed, following the steps described in the bug and i got the error in nova-compute.log: `Stderr: u'blockdev: cannot open xxx` details below. But `/dev/disk/by-path/ip-192.168.70.11:3260-iscsi-iqn.2010-10.org.openstack:volume-08bc45cb-e570-44a7-8619-6f575575369c-lun-1` could be found by `ls /dev/disk/by-path`.. even disk config exists by 'virsh dumpxml <domain>' .. After resuming the instance, `virsh dumpxml <domain>` couldn't found the disk config anymore, but `/dev/disk/by-path/xxxx` still exists.

2015-04-01 13:03:37.597 ERROR nova.openstack.common.periodic_task [-] Error during ComputeManager.update_available_resource: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf blockdev --getsize64 /dev/disk/by-path/ip-192.168.70.11:3260-iscsi-iqn.2010-10.org.openstack:volume-08bc45cb-e570-44a7-8619-6f575575369c-lun-1
Exit code: 1
Stdout: u''
Stderr: u'blockdev: cannot open /dev/disk/by-path/ip-192.168.70.11:3260-iscsi-iqn.2010-10.org.openstack:volume-08bc45cb-e570-44a7-8619-6f575575369c-lun-1: No such device or address\n'
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task File "/data/stack/nova/nova/openstack/common/periodic_task.py", line 224, in run_periodic_tasks
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task task(self, context)
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task File "/data/stack/nova/nova/compute/manager.py", line 6067, in update_available_resource
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task rt.update_available_resource(context)
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task File "/data/stack/nova/nova/compute/resource_tracker.py", line 315, in update_available_resource
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task resources = self.driver.get_available_resource(self.nodename)
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task File "/data/stack/nova/nova/virt/libvirt/driver.py", line 4875, in get_available_resource
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task disk_over_committed = self._get_disk_over_committed_size_total()
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task File "/data/stack/nova/nova/virt/libvirt/driver.py", line 6056, in _get_disk_over_committed_size_total
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task self._get_instance_disk_info(dom.name(), xml))
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task File "/data/stack/nova/nova/virt/libvirt/driver.py", line 6009, in _get_instance_disk_info
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task dk_size = lvm.get_volume_size(path)
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task File "/data/stack/nova/nova/virt/libvirt/lvm.py", line 172, in get_volume_size
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task run_as_root=True)
2015-04-01 13:03:37.597 TRACE nova.openstack.common.periodic_task File "/data/stack/nova/nova/virt/libvirt/utils.py", lin...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/200032

[OpenStack Infra (hudson-openstack)]

Change abandoned by lyanchih (<email address hidden>) on branch: master
Review: https://review.openstack.org/200032

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/252170

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/252170
Reason: This patch is quite old, so I am abandoning it to keep the review queue manageable. Feel free to restore the change if you're still interested in working on it.

[Maciej Szankin (mszankin)]

This bug report has an assignee for a while now but there is no patch
for that. It looks like that the chance of getting a patch is low.
I'm going to remove the assignee to signal to others that they can take
over if they like.
If you want to work on this, please:
* add yourself as assignee AND
* set the status to "In Progress" AND
* provide a (WIP) patch within the next 2 weeks after that.
If you need assistance, reach out on the IRC channel #openstack-nova or
use the mailing list.

[Maciej Szankin (mszankin)]

Also, this bug needs to be reverified. It is old as hell!

[Sean Dague (sdague)]

Automatically discovered version juno in description. If this is incorrect, please update the description to include 'nova version: ...'

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]