Swap disk on stopped instance fails silently

[Steve Noyes]

Using either the cli or API, if you attempt to swap disks (nova volume-update, compute.api.swap_volume) on a stopped instance, the command appears to succeed, but the swap is not performed. The way you can tell what didn't happen is that the volume status's of the 2 volumes remain as they were before the operation.

It would be better to throw an exception when a swap is attempted on a stopped instance. (this can get tricky since an instance can stop at any time during the swap operation. btw, it seems odd that attach and detach are supported on a stopped instance but swap is not. would it be better to support swap on a stopped instance?)

In the compute log is this exception:

2017-03-15 09:20:13.729 ERROR nova.compute.manager [^[[01;36mreq-e978c2af-d537-4855-ba5f-58ec1976ad2f ^[[00;36mtempest-SwapVolumeTestJSON-1034316961 tempest-SwapVolumeTestJSON-1034316961] ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] Failed to swap volume 11cda3f1-15c3-4195-b404-a7e912f4c517 for 8b6a1904-3e54-43f4-b498-0fa153d7cf17^[[00m
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00mTraceback (most recent call last):
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m File "/opt/stack/nova/nova/compute/manager.py", line 4988, in _swap_volume
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m resize_to)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1300, in swap_volume
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m self._swap_volume(guest, disk_dev, conf.source_path, resize_to)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1257, in _swap_volume
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m dev.rebase(new_path, copy=True, reuse_ext=True)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 748, in rebase
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m self._disk, base, self.REBASE_DEFAULT_BANDWIDTH, flags=flags)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m rv = execute(f, *args, **kwargs)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m six.reraise(c, e, tb)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m rv = meth(*args, **kwargs)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 930, in blockRebase
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m if ret == -1: raise libvirtError ('virDomainBlockRebase() failed', dom=self)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00mlibvirtError: Domain not found: no domain with matching uuid '26fe837e-d72c-4969-8934-09dc92b61897' (instance-00000022)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897]

My setup is devstack, libvirt+kvm, the volume is a cinder iscsi volume, instance is cirros,

nova$ git log -1
commit a3655c311454c09312f12f413535d83302e1fba4
Merge: 1ef9cbd 3c7b73b
Author: Jenkins <email address hidden>
Date: Tue Feb 28 19:21:35 2017 +0000

A somewhat related bug:

https://bugs.launchpad.net/nova/+bug/1635657

[Steve Noyes (steve-noyes)]

Swap works ok when instance is:
- running
- paused
- verify resize

Swap fails silently when instance is:
- stopped
- suspended

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/446708

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Noyes (<email address hidden>) on branch: master
Review: https://review.openstack.org/446708

[Maciej Szankin (mszankin)]

Steve is it safe to mark this one as a duplicate? You mentioned in your comment under https://review.openstack.org/#/c/389798 that there is already a patch that most likely will fix this issue. Can you verify and mark this bug as a duplicate if it turns out that it is?

[Steve Noyes (steve-noyes)]

Hi, bug 1635657 is no longer happening in the latest version in git, so I think it would be better to close that bug, and have this be the bug that review https://review.openstack.org/#/c/389798 is fixing.

[Sean Dague (sdague)]

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/389798
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b40d949b3137473227949b597b2a61da41752ee5
Submitter: Zuul
Branch: master

commit b40d949b3137473227949b597b2a61da41752ee5
Author: Mark Giles <email address hidden>
Date: Mon Jul 24 12:46:21 2017 -0400

    Do not attempt volume swap when guest is stopped/suspended

    A swap on a stopped or suspended instance will fail silently. Remove
    these allowed instance states on swap_volume:

    suspended, stopped, soft_deleted

    Change-Id: Iff17f7cee7a56037b35d1a361a0b3279d0a885d6
    Closes-Bug: #1673090

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/533733

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/533733
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7a51ffc551482ac28eaa493c408084d699a6dc8c
Submitter: Zuul
Branch: stable/pike

commit 7a51ffc551482ac28eaa493c408084d699a6dc8c
Author: Mark Giles <email address hidden>
Date: Mon Jul 24 12:46:21 2017 -0400

    Do not attempt volume swap when guest is stopped/suspended

    A swap on a stopped or suspended instance will fail silently. Remove
    these allowed instance states on swap_volume:

    suspended, stopped, soft_deleted

    Change-Id: Iff17f7cee7a56037b35d1a361a0b3279d0a885d6
    Closes-Bug: #1673090
    (cherry picked from commit b40d949b3137473227949b597b2a61da41752ee5)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.1 release.