cinder retype volume raise exception with shutoff vm

[Sang Tran]

Environment:
- openstack victoria version
- iSCSI storage using Netapp driver

Step to reproduce:
1. Shutoff vm and wait for its complete
2. Run the command cinder retype --migration-policy on-demand <vol-id> <new-type>
3. Task will fail with Error log below

2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager [req-aea12cc1-b7f5-4c38-85d7-1c9094a1006a d3802c30ff814feaa3197d29e8a3a731 2c97f9ded4c843279d2de2c4a5c837
01 - - -] Failed to copy volume f81a905d-05ea-4534-9b3b-a6762adf87f3 to 099ce4e2-8539-43dc-a2fa-8feca2845a3d: novaclient.exceptions.Conflict: Cannot 'swap_volume' instance a14b7344-3054-4d9a-9357-c0adc1b00650 while it is in vm_state stopped (HTTP 409) (Request-ID: req-7913c7fb-51d0-4a2a-8463-d8a9883dea0b)
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager Traceback (most recent call last):
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2292, in _migrate_volume_generic
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager nova_api.update_server_volume(ctxt, instance_uuid,
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/compute/nova.py", line 186, in update_server_volume
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager nova.volumes.update_server_volume(server_id,
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/api_versions.py", line 393, in substitution
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager return methods[-1].func(obj, *args, **kwargs)
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/v2/volumes.py", line 124, in update_server_volume
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager return self._update("/servers/%s/os-volume_attachments/%s" %
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/base.py", line 380, in _update
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager resp, body = self.api.client.put(url, body=body)
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 404, in put
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager return self.request(url, 'PUT', **kwargs)
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/client.py", line 78, in request
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager raise exceptions.from_response(resp, body, url, method)
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager novaclient.exceptions.Conflict: Cannot 'swap_volume' instance a14b7344-3054-4d9a-9357-c0adc1b00650 whileit is in vm_state stopped (HTTP 409) (Request-ID: req-7913c7fb-51d0-4a2a-8463-d8a9883dea0b)
2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server [req-aea12cc1-b7f5-4c38-85d7-1c9094a1006a d3802c30ff814feaa3197d29e8a3a731 2c97f9ded4c843279d2de2c4a5c83701 - - -] Exception during message handling: novaclient.exceptions.Conflict: Cannot 'swap_volume' instance a14b7344-3054-4d9a-9357-c0adc1b00650 while it is in vm_state stopped (HTTP 409) (Request-ID: req-7913c7fb-51d0-4a2a-8463-d8a9883dea0b)
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309,in dispatch
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229,in _do_dispatch
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2957, in retype
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server _retype_error(context, volume, old_reservations,
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server raise value
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2953, in retype
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server self.migrate_volume(context, volume, host,
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2546, in migrate_volume
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server volume.save()
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server raise value
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2539, in migrate_volume
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server self._migrate_volume_generic(ctxt, volume, host, new_type_id)
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2300, in _migrate_volume_generic
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server self._clean_temporary_volume(ctxt, volume,
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server raise value
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2292, in _migrate_volume_generic
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server nova_api.update_server_volume(ctxt, instance_uuid,
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/compute/nova.py", line 186, in update_server_volume
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server nova.volumes.update_server_volume(server_id,
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/api_versions.py", line 393, in substitution
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server return methods[-1].func(obj, *args, **kwargs)
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/v2/volumes.py", line 124, in update_server_volume
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server return self._update("/servers/%s/os-volume_attachments/%s" %
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/base.py", line 380, in _update
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server resp, body = self.api.client.put(url, body=body)
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 404, in put
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs)
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/client.py", line 78, in request
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body, url, method)
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server novaclient.exceptions.Conflict: Cannot 'swap_volume' instance a14b7344-3054-4d9a-9357-c0adc1b00650 while it is in vm_state stopped (HTTP 409) (Request-ID: req-7913c7fb-51d0-4a2a-8463-d8a9883dea0b)
2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server

[Sofia Enriquez (lsofia-enriquez)]

Hi Sang Tran, hope this message finds you well.

From the logs, looks like a nova problem.
`
ERROR oslo_messaging.rpc.server novaclient.exceptions.Conflict: Cannot 'swap_volume' instance a14b7344-3054-4d9a-9357-c0adc1b00650 while it is in vm_state stopped (HTTP 409) (Request-ID: req-7913c7fb-51d0-4a2a-8463-d8a9883dea0b)
`

Question: are you trying to migrate in-use volumes (attached volumes)?
Just for the sake of completeness I need to mentioned that migration of ‘in-use’ volumes will not succeed for any backend when they are attached to an instance in any of these states: SHUTOFF, SUSPENDED, or SOFT-DELETED[1].

Adding Nova team for more feedback.

Thanks,
Sofia

[1] https://docs.openstack.org/cinder/latest/contributor/migration.html

[Sang Tran (sangtq8)]

Hi Sofia,

Exactly, I'm trying to migrate in-use volume, and I indicate the behavior as:

- With volume in-use and VM powered on: Migrate to new cinder volume type and swap volume successfully.
- With volume in-use and VM powered off: Migrate to new cinder volume successfully but swap volume failed because vm_state stopped.

Thanks,
Sang Tran

[Uggla (rene-ribaud)]

Hi Sang Tran,

As Sofia mentionned the VM needs to be in ACTIVE, PAUSED, or RESIZED in order to swap a volume as far as I can see in the Nova code.

    @check_instance_state(vm_state=[vm_states.ACTIVE, vm_states.PAUSED,
                                    vm_states.RESIZED])
    def swap_volume(self, context, instance, old_volume, new_volume):

So I think the error you are seeing is "normal".

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]