nova-compute can't detach volume from the instance

[Olga Klochkova]

Environment has ceph as a storage backend
Steps to reproduce:
1. create image:

glance --os-image-api-version 1 image-update <image-id> --location http://releases.ubuntu.com/14.04/ubuntu-14.04.3-server-amd64.iso --disk-format qcow2 --container-format bare
This is important step, as the issue is not reproducible with TestVM image

2. spawn a VM from that image
 nova boot TestVm --flavor m1.tiny --image <image-id> --nic net-id=<net-id>
3. create volume
cinder --os-volume-api-version 2 create --name TestVOL 1
4. attach volume to the VM
nova volume-attach <VM-id> <Volume-id>
5. detach volume from the VM
nova volume-detach <VM-id> <Volume-id>

Volume stays in 'detaching’ state for some time (~30 sec) and then gets back to 'in-use' state.

relevant nova-compute logs:
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall [-] Cannot retry nova.virt.libvirt.guest._do_wait_and_retry_detach upon suggested exception since retry count (7) reached max retry count (7).
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall [-] Dynamic interval looping call 'oslo_service.loopingcall._func' failed
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall Traceback (most recent call last):
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 113, in _run_loop
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw)
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 269, in _func
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall return self._sleep_time
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb)
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 247, in _func
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall result = f(*args, **kwargs)
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 326, in _do_wait_and_retry_detach
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall reason=reason)
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall DeviceDetachFailed: Device detach failed for vdb: Unable to detach from guest transient domain.)
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall
2016-02-26 07:37:11.002 21387 ERROR nova.compute.manager [req-7944b96d-8d80-442e-a8c6-bf2148a34309 5c52fdfdec6f46f2960fdc2108b78c63 29a3255438bc49e7895968ff67c14682 - - -] [instance: 117d58ad-642c-475e-a4e0-8a91a9c0e864] Failed to detach volume 5b07c574-656b-40a4-9c7b-6dd4701235ca from /dev/vdb

relevant libvirt logs:

2016-02-26 07:00:50.876+0000: 3927: error : qemuDomainDetachDeviceConfig:7045 : invalid argument: no target device vdb

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "569"
  build_id: "569"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "33634ec27be77ecfb0b56b7e07497ad86d1fdcd3"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

[Eugene Nikanorov (enikanorov)]

This is important issue that is nearly critical. Very basic functionality doesn't work with commonly used cloud image.

[Roman Podoliaka (rpodolyaka)]

I gave it a try on the 9.0 environment with Ceph used for both ephemerals and volumes and *did not* manage to reproduce the issue:

http://paste.openstack.org/show/494666/

^ the volume is attached / detached properly

Packages versions: http://paste.openstack.org/show/494667/

Image: https://cloud-images.ubuntu.com/trusty/current/

[Roman Podoliaka (rpodolyaka)]

Moreover, the same goes (to my surprise) for a volume with a formatted and mounted FS:

http://paste.openstack.org/show/494669/

[Roman Podoliaka (rpodolyaka)]

Olga, could you please confirm you are still seeing this issue?

[Timur Nurlygayanov (tnurlygayanov)]

I'm going to check the issue.

[Timur Nurlygayanov (tnurlygayanov)]

I can't reproduce the issue:

root@node-1:~# nova volume-attach cf1fce9b-933f-455f-94af-42a369d1ba6c 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d |
| serverId | cf1fce9b-933f-455f-94af-42a369d1ba6c |
| volumeId | 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d |
+----------+--------------------------------------+
root@node-1:~# nova volume-list
WARNING: Command volume-list is deprecated and will be removed after Nova 13.0.0 is released. Use python-cinderclient or openstackclient instead.
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d | attaching | | 1 | - | |
+--------------------------------------+-----------+--------------+------+-------------+-------------+
root@node-1:~#
root@node-1:~#
root@node-1:~# nova volume-list
WARNING: Command volume-list is deprecated and will be removed after Nova 13.0.0 is released. Use python-cinderclient or openstackclient instead.
+--------------------------------------+--------+--------------+------+-------------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+--------+--------------+------+-------------+--------------------------------------+
| 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d | in-use | | 1 | - | cf1fce9b-933f-455f-94af-42a369d1ba6c |
+--------------------------------------+--------+--------------+------+-------------+--------------------------------------+
root@node-1:~#
root@node-1:~#
root@node-1:~# nova volume-detach cf1fce9b-933f-455f-94af-42a369d1ba6c 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d
root@node-1:~# nova volume-list
WARNING: Command volume-list is deprecated and will be removed after Nova 13.0.0 is released. Use python-cinderclient or openstackclient instead.
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d | available | | 1 | - | |
+--------------------------------------+-----------+--------------+------+-------------+-------------+

[Mikhail Chernik (mchernik)]

Reproduced on scale lab for MOS 9.2 snapshot 688.

Rally log (scenario inside) and report: http://mos-scale.vm.mirantis.net/test_results/build_9.0.0-495/jenkins-10_env_run_rally_custom-297-ubuntu-MSK-2017-01-16-14:54:35-2017-01-16-15:34:35/

Diagnostic snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-env-10-custom-296-pymysql-2017-01-16_16-43-05/

[chandra shekar (sekharvajjula)]

We have seen this problem as well in Queens release. Our VMs write huge data to cinder Volumes (/dev/vdb) all the time.
At this point of time if we run "stack delete", it results exactly the same failure.

This problem subsides if we first shutdown the VMs before stack delete. Should this be somehow incorporated in the heat code itself?

(Or)
Should heat simply delete the libvirt VM domain without trying to detach the volume?