When I try to retype (cinder retype --migration-policy on-demand e9b0c6a7-5b5e-4f03-a36c-6fc54ee3dea2 R6-SAS) a volume that is attached to an instance, the migration fails with the following error in cinder.log. No other error located in other logs on the system.
I have added a little bit more logging at the place of the error in order to try to understand it:
2021-08-25 10:02:29.321 201582 INFO cinder.volume.flows.manager.create_volume [req-c926d351-4464-488b-9920-0afeb471dcf7 1971ee1f1edc42469dca0f937fa53b65 7d4c1d640f2e45c9826a1cef32fc08bf - - -] Volume 2bd1a946-74b8-40f6-b84d-709909700476: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2bd1a946-74b8-40f6-b84d-709909700476', 'volume_size': 12}
2021-08-25 10:02:29.353 201582 INFO cinder.volume.flows.manager.create_volume [req-c926d351-4464-488b-9920-0afeb471dcf7 1971ee1f1edc42469dca0f937fa53b65 7d4c1d640f2e45c9826a1cef32fc08bf - - -] Volume volume-2bd1a946-74b8-40f6-b84d-709909700476 (2bd1a946-74b8-40f6-b84d-709909700476): created successfully
2021-08-25 10:02:29.358 201582 INFO cinder.volume.manager [req-c926d351-4464-488b-9920-0afeb471dcf7 1971ee1f1edc42469dca0f937fa53b65 7d4c1d640f2e45c9826a1cef32fc08bf - - -] Created volume successfully.
2021-08-25 10:02:30.320 201585 INFO cinder.volume.manager [req-c926d351-4464-488b-9920-0afeb471dcf7 1971ee1f1edc42469dca0f937fa53b65 7d4c1d640f2e45c9826a1cef32fc08bf - - -] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2021-08-25 10:02:30.321 201585 INFO cinder.volume.manager [req-c926d351-4464-488b-9920-0afeb471dcf7 1971ee1f1edc42469dca0f937fa53b65 7d4c1d640f2e45c9826a1cef32fc08bf - - -] {'_user_id': '1971ee1f1edc42469dca0f937fa53b65', '_project_id': '7d4c1d640f2e45c9826a1cef32fc08bf', '_domain_id': None, '_user_domain_id': None, '_project_domain_id': None, 'auth_token': 'gAAAAABhJhU0lMbH2MKbnC893BP0J0oXw-_SGpCrwDTEisjPupQP1pPTa_oOsfBDcubVYlZr_XdRa01iB1hBz0RjpTfBnMTO0K0QrkXd7j7_2d2rNTlZETQqGn2hGdN04GjqfbvvtB7lA5vWSIJs8I8c0RuydWcRJCbC7Dl2QSNNgLRcf2Qk8jk', 'user_name': None, 'project_name': 'LAB', 'domain_name': None, 'system_scope': None, 'user_domain_name': None, 'project_domain_name': None, 'is_admin': False, 'is_admin_project': True, 'read_only': False, 'show_deleted': False, 'resource_uuid': None, 'roles': ['member', 'reader'], 'service_token': None, 'service_user_id': None, 'service_user_name': None, 'service_user_domain_id': None, 'service_user_domain_name': None, 'service_project_id': None, 'service_project_name': None, 'service_project_domain_id': None, 'service_project_domain_name': None, 'service_roles': [], 'request_id': 'req-c926d351-4464-488b-9920-0afeb471dcf7', 'global_request_id': 'req-33c9c4d5-4a53-4248-9caa-4897465dd6c0', '_read_deleted': 'no', 'remote_address': '10.101.1.1', 'timestamp': datetime.datetime(2021, 8, 25, 10, 2, 29, 100823, tzinfo=datetime.timezone(datetime.timedelta(0), '+00:00')), 'quota_class': None, 'service_catalog': [{'type': 'compute', 'name': 'nova', 'endpoints': [{'region': 'vdc2', 'publicURL': 'http://controller:8774/v2.1', 'internalURL': 'http://controller:8774/v2.1', 'adminURL': 'http://controller:8774/v2.1'}]}, {'type': 'identity', 'name': 'keystone', 'endpoints': [{'region': 'vdc2', 'adminURL': 'http://controller:5000/v3/', 'publicURL': 'http://controller:5000/v3/', 'internalURL': 'http://controller:5000/v3/'}]}, {'type': 'image', 'name': 'glance', 'endpoints': [{'region': 'vdc2', 'publicURL': 'http://controller:9292', 'internalURL': 'http://controller:9292', 'adminURL': 'http://controller:9292'}]}], 'user_auth_plugin': None}
2021-08-25 10:02:30.321 201585 INFO cinder.volume.manager [req-c926d351-4464-488b-9920-0afeb471dcf7 1971ee1f1edc42469dca0f937fa53b65 7d4c1d640f2e45c9826a1cef32fc08bf - - -] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager [req-c926d351-4464-488b-9920-0afeb471dcf7 1971ee1f1edc42469dca0f937fa53b65 7d4c1d640f2e45c9826a1cef32fc08bf - - -] Failed to copy volume e9b0c6a7-5b5e-4f03-a36c-6fc54ee3dea2 to 2bd1a946-74b8-40f6-b84d-709909700476: keystoneauth1.exceptions.http.BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-0a167d96-f80d-4743-a145-e878ef0311c4)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager Traceback (most recent call last):
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 2360, in _migrate_volume_generic
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager nova_api.update_server_volume(ctxt, instance_uuid,
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/compute/nova.py", line 186, in update_server_volume
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager nova.volumes.update_server_volume(server_id,
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/novaclient/api_versions.py", line 393, in substitution
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager return methods[-1].func(obj, *args, **kwargs)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/novaclient/v2/volumes.py", line 124, in update_server_volume
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager return self._update("/servers/%s/os-volume_attachments/%s" %
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/novaclient/base.py", line 380, in _update
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager resp, body = self.api.client.put(url, body=body)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 404, in put
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager return self.request(url, 'PUT', **kwargs)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/novaclient/client.py", line 69, in request
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager resp, body = super(SessionClient, self).request(url,
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 554, in request
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager return self.session.request(url, method, **kwargs)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 790, in request
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager auth_headers = self.get_auth_headers(auth)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1201, in get_auth_headers
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager return auth.get_headers(self, **kwargs)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/keystoneauth1/plugin.py", line 106, in get_headers
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager token = self.get_token(session)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager return self.get_access(session).auth_token
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager self.auth_ref = self.get_auth_ref(session)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager return self._plugin.get_auth_ref(session, **kwargs)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 187, in get_auth_ref
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager resp = session.post(token_url, json=body, headers=headers,
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1149, in post
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager return self.request(url, 'POST', **kwargs)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 986, in request
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager raise exceptions.from_response(resp, method, url)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager keystoneauth1.exceptions.http.BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-0a167d96-f80d-4743-a145-e878ef0311c4)
2021-08-25 10:02:30.420 201585 ERROR cinder.volume.manager
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server [req-c926d351-4464-488b-9920-0afeb471dcf7 1971ee1f1edc42469dca0f937fa53b65 7d4c1d640f2e45c9826a1cef32fc08bf - - -] Exception during message handling: keystoneauth1.exceptions.http.BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-0a167d96-f80d-4743-a145-e878ef0311c4)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 3043, in retype
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server _retype_error(context, volume, old_reservations,
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server raise self.value
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 3039, in retype
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server self.migrate_volume(context, volume, host,
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 2622, in migrate_volume
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server volume.save()
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server raise self.value
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 2615, in migrate_volume
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server self._migrate_volume_generic(ctxt, volume, host, new_type_id)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 2368, in _migrate_volume_generic
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server self._clean_temporary_volume(ctxt, volume,
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server raise self.value
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 2360, in _migrate_volume_generic
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server nova_api.update_server_volume(ctxt, instance_uuid,
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/compute/nova.py", line 186, in update_server_volume
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server nova.volumes.update_server_volume(server_id,
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/novaclient/api_versions.py", line 393, in substitution
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server return methods[-1].func(obj, *args, **kwargs)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/novaclient/v2/volumes.py", line 124, in update_server_volume
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server return self._update("/servers/%s/os-volume_attachments/%s" %
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/novaclient/base.py", line 380, in _update
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server resp, body = self.api.client.put(url, body=body)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 404, in put
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/novaclient/client.py", line 69, in request
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server resp, body = super(SessionClient, self).request(url,
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 554, in request
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server return self.session.request(url, method, **kwargs)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 790, in request
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server auth_headers = self.get_auth_headers(auth)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1201, in get_auth_headers
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server return auth.get_headers(self, **kwargs)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/plugin.py", line 106, in get_headers
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server token = self.get_token(session)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server return self._plugin.get_auth_ref(session, **kwargs)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 187, in get_auth_ref
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server resp = session.post(token_url, json=body, headers=headers,
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1149, in post
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server return self.request(url, 'POST', **kwargs)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 986, in request
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, method, url)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.http.BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-0a167d96-f80d-4743-a145-e878ef0311c4)
2021-08-25 10:02:30.449 201585 ERROR oslo_messaging.rpc.server
2021-08-25 10:02:30.497 201582 INFO cinder.volume.manager [req-c926d351-4464-488b-9920-0afeb471dcf7 1971ee1f1edc42469dca0f937fa53b65 7d4c1d640f2e45c9826a1cef32fc08bf - - -] Deleted volume successfully.
root@block1: /usr/lib/ python3/ dist-packages/ cinder/ compute# dpkg -l | grep cinder 0-0ubuntu3~ cloud0 all Cinder storage service - common files 0-0ubuntu3~ cloud0 all Cinder storage service - Volume server 0-0ubuntu3~ cloud0 all Cinder Python 3 libraries
ii cinder-common 2:18.0.
ii cinder-volume 2:18.0.
ii python3-cinder 2:18.0.