Swapping volume will not result in correct status

[Chung Chih, Hung]

Swap volume API can swap a in-use volume with available volume.
Therefore in-use volume should be available and available will be in-use.
In face, volumes will stay in detaching and attaching respectively after send swap volume request to nova.
But those volumes had been swapped.
Following are commands flow.

> cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Multiattach | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+--------------------------------------+
| 8a3440f7-afbf-4581-95fb-14730fbd98c1 | in-use | - | 1 | lvmdriver-1 | false | False | 6d316dab-426e-48f9-956f-e2f0df1aabcd |
| cafc833a-8645-47db-b464-999142afa7be | available | - | 1 | lvmdriver-1 | false | False | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+--------------------------------------+

> sudo iscsiadm -m node
10.0.2.15:3260,-1 iqn.2010-10.org.openstack:volume-8a3440f7-afbf-4581-95fb-14730fbd98c1

* Ready to send swap volume request *
> curl -H "Content-Type: application/json" -H "X-Auth-Token: $TOKEN" 127.0.0.1:8774/v3/servers/6d316dab-426e-48f9-956f-e2f0df1aabcd/os-volume_attachments/8a3440f7-afbf-4581-95fb-14730fbd98c1 -X 'PUT' -d '{"volumeAttachment": {"volumeId":"cafc833a-8645-47db-b464-999142afa7be"}}'

> cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Multiattach | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+--------------------------------------+
| 8a3440f7-afbf-4581-95fb-14730fbd98c1 | detaching | - | 1 | lvmdriver-1 | false | False | 6d316dab-426e-48f9-956f-e2f0df1aabcd |
| cafc833a-8645-47db-b464-999142afa7be | attaching | - | 1 | lvmdriver-1 | false | False | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+--------------------------------------+

> sudo iscsiadm -m node
10.0.2.15:3260,-1 iqn.2010-10.org.openstack:volume-cafc833a-8645-47db-b464-999142afa7be

My devstack environments:
nova (12.0.0.0b3.dev508, /opt/stack/nova)
python-novaclient (2.26.0)
cinder (7.0.0.0b3.dev296, /opt/stack/cinder)
python-cinderclient (1.3.1)

[Chung Chih, Hung (lyanchih)]

I had found attach and detach function were removed since
https://review.openstack.org/#/c/101933/3
Therefore those volumes were stay in attaching or detaching.

[Ryan McNair (rdmcnair)]

Should be fixed by change in Cinder - https://review.openstack.org/#/c/247767/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/249059

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/249093

[OpenStack Infra (hudson-openstack)]

Change abandoned by Takashi NATSUME (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/249059

[Ankit Agrawal (ankitagrawal)]

The patch [1] does not fix this issue correctly and introduced another issue in cinder.

If multiple volumes are attached to an instance and we try to swap a volume "in-use" with another volume "available", swap-volume fails with error message "NotImplementedError: Swap only supports host devices" but the volume status has been swapped by cinder api.

It looks worse as the attached volume status is set to "available" and the volume which is set to "in-use" is not attached to the instance.
Where as volume status remains unchanged in case of swap-volume failure before this patch is merged which looks reasonable.

Steps to reproduce problem on latest master:-

1. Create an instance.
2. Create 3 volumes vol1, vol2 and vol3
3. Attach vol1 and vol2 to instance
4. swap vol1 with vol3
$nova volume-update <instance id> <vol1> <vol3>
5. Swap volume fails with error message "NotImplementedError: Swap only supports host devices".
6. But the volumes status has been updated in cinder.
i.e. vol1 is still attached to the instance but its status has been updated to "available" and vol3 is updated to "in-use" by cinder api while it is not attached to the instance.

This issue is introduced by patch [1], as after reverting these changes volumes status is not updated on failure.

[1] https://review.openstack.org/#/c/247767
Error logs: http://paste.openstack.org/show/479964/

[Ankit Agrawal (ankitagrawal)]

The above issue is only reproducible while configured openstack with ceph backend.

I have used following configuration to install openstack with ceph backend using devstack:
https://github.com/ceph/ceph-devstack/blob/master/localrc

[Ankit Agrawal (ankitagrawal)]

I have testes swap-volume with LVM and ceph backends for following scenarios.

1. LVM driver

1.1 Attached one volume to the instance.
Result: Second swap volume failed with "VolumeBDMNotFound" error but volumes status is updated incorrectly in cinder (source_vol="available", dest_vol="in_use").

1.2 Attached two volumes to the instance.
Result: Second swap volume failed with "VolumeBDMNotFound" error but volumes status is updated incorrectly in cinder (source_vol="available", dest_vol="in_use") .

This issue is already reported at: https://bugs.launchpad.net/nova/+bug/1490236

2. ceph driver

2.1 Attached one volume to the instance.
Result: Swap volume failed with "NotImplementedError" error but volumes status is updated incorrectly in cinder (source_vol="available", dest_vol="in_use") .

2.2 Attached two volumes to the instance.
Result: Swap volume failed with "NotImplementedError" error but volumes status is updated incorrectly in cinder (source_vol="available", dest_vol="in_use") .

Please find detailed execution results at: http://paste.openstack.org/show/480482/

[Ryan McNair (rdmcnair)]

Ankit - thanks for the info. I'll look into the Ceph issue first and then if the https://bugs.launchpad.net/nova/+bug/1490236 is not fixed yet will look into that next.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/252089

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/cinder 8.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/252089
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=fc6e7250457479f221d9a11767a16511ffaf30eb
Submitter: Jenkins
Branch: master

commit fc6e7250457479f221d9a11767a16511ffaf30eb
Author: Ryan McNair <email address hidden>
Date: Wed Dec 9 15:29:27 2015 +0000

    Fix non-migration swap with error

    The fix I242c2c7c4a7197bbce04d0b3d75688f989ea1fd5 for non-migration
    swap volume caused drivers which don't support volume swapping,
    e.g. Ceph, to put the volumes in an incorrect state (additional
    details can be found in bug #1489744 report). This patch adds an
    additional check to ensure no errors occurred during the swap before
    it completes the volume status updates, as well a test-case for
    the non-migration swap scenario.

    Change-Id: Ic2fddbcb12b4c9d251f9c3aab08a73e12d4d73e2
    Closes-Bug: #1489744

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/249093
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=1d36722c2cb335ff078a823ddc8d97a150b3f67c
Submitter: Jenkins
Branch: stable/liberty

commit 1d36722c2cb335ff078a823ddc8d97a150b3f67c
Author: Ryan McNair <email address hidden>
Date: Thu Nov 19 19:48:37 2015 +0000

    Fix swap_volume for case without migration

    Fix swap_volume so that even when a volume-migration does not
    occur, the volumes don't get stuck in "attaching" and "detaching".
    Previously the swap_volume was changed in
    https://review.openstack.org/#/c/101932 and
    https://review.openstack.org/#/c/101933 to move the final
    attach and detach logic to Cinder, however those changes only fix
    the migration path. This change calls Cinder's "attach" and "detach"
    for the non-migration path to update the volumes' states.

    Change-Id: I242c2c7c4a7197bbce04d0b3d75688f989ea1fd5
    Closes-Bug: #1489744
    (cherry picked from commit dc777f1dc736b01b8f8eec58972230d30b3074f6)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/257091

[Thierry Carrez (ttx)]

This issue was fixed in the cinder 7.0.1 release (liberty).

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/cinder 8.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/257091
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=077c8df370546005bdb57c5b8eaa7f32b1017501
Submitter: Jenkins
Branch: stable/liberty

commit 077c8df370546005bdb57c5b8eaa7f32b1017501
Author: Ryan McNair <email address hidden>
Date: Wed Dec 9 15:29:27 2015 +0000

    Fix non-migration swap with error

    The fix I242c2c7c4a7197bbce04d0b3d75688f989ea1fd5 for non-migration
    swap volume caused drivers which don't support volume swapping,
    e.g. Ceph, to put the volumes in an incorrect state (additional
    details can be found in bug #1489744 report). This patch adds an
    additional check to ensure no errors occurred during the swap before
    it completes the volume status updates, as well a test-case for
    the non-migration swap scenario.

    Conflicts:
     cinder/tests/unit/test_volume.py
     cinder/volume/api.py

    Change-Id: Ic2fddbcb12b4c9d251f9c3aab08a73e12d4d73e2
    Closes-Bug: #1489744
    (cherry picked from commit fc6e7250457479f221d9a11767a16511ffaf30eb)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/cinder 7.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 7.0.2 release.