Has this bug been fixed in some other way? I think I ran into it on Queens when I tried to do a "nova live-migration-force-complete <id>".
The result was a big mess, with the VM left in some half-migrated state, because this error stopped processing of the migration in its tracks. The VM was running on the target host, but nova thought it was still migrating. The volume attachments were duplicated: once for the target, once for the source. We needed to remove the old attachments and change the nova database directly to fix the state.
2021-09-06 11:20:55.932 48419 ERROR nova.volume.cinder [req-6f8d526d-4516-4162-9111-ea261eee460a 5073df8f6c9147aa8802953e07d8d825 6fa231b814d04f1e87b88548872333c9 - default default] Delete attachment failed for attachment 4b21f601-1bac-408f-8985-eaba051cc81a. Error: The request you have made requires authentication. (HTTP 401) Code: 401: Unauthorized: The request you have made requires authentication. (HTTP 401)
2021-09-06 11:20:55.934 48419 ERROR nova.compute.manager [req-6f8d526d-4516-4162-9111-ea261eee460a 5073df8f6c9147aa8802953e07d8d825 6fa231b814d04f1e87b88548872333c9 - default default] [instance: a2567a27-aa3f-4b16-8fd9-fc96d2427da3] Volume attachment 4b21f601-1bac-408f-8985-eaba051cc81a not deleted on source host dbl031623.dbl.sys11cloud.net during post_live_migration: The request you have made requires authentication. (HTTP 401): Unauthorized: The request you have made requires authentication. (HTTP 401)
2021-09-06 11:21:01.255 48419 ERROR nova.compute.manager [req-6f8d526d-4516-4162-9111-ea261eee460a 5073df8f6c9147aa8802953e07d8d825 6fa231b814d04f1e87b88548872333c9 - default default] [instance: a2567a27-aa3f-4b16-8fd9-fc96d2427da3] Post live migration at destination dbl021501.dbl.sys11cloud.net failed: NeutronAdminCredentialConfigurationInvalid_Remote: Networking client is experiencing an unauthorized exception.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
res = self.dispatcher.dispatch(message)
File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)
File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
result = func(ctxt, **new_args)
File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 76, in wrapped
function_name, call_dict, binary)
File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
self.force_reraise()
File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 67, in wrapped
return f(self, context, *args, **kw)
File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 1000, in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 216, in decorated_function
kwargs['instance'], e, sys.exc_info())
File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
self.force_reraise()
File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 204, in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6667, in post_live_migration_at_destination
self.host)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 344, in setup_networks_on_host
data = self.list_ports(context, **search_opts)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1401, in list_ports
return get_client(context).list_ports(**search_opts)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
ret = obj(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 786, in list_ports
**_params)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
ret = obj(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 354, in get
headers=headers, params=params)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
ret = obj(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 331, in retry_request
headers=headers, params=params)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
ret = obj(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 294, in do_request
self._handle_fault_response(status_code, replybody, resp)
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 130, in wrapper
raise exception.NeutronAdminCredentialConfigurationInvalid()
NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
Has this bug been fixed in some other way? I think I ran into it on Queens when I tried to do a "nova live-migration- force-complete <id>".
The result was a big mess, with the VM left in some half-migrated state, because this error stopped processing of the migration in its tracks. The VM was running on the target host, but nova thought it was still migrating. The volume attachments were duplicated: once for the target, once for the source. We needed to remove the old attachments and change the nova database directly to fix the state.
2021-09-06 11:20:55.932 48419 ERROR nova.volume.cinder [req-6f8d526d- 4516-4162- 9111-ea261eee46 0a 5073df8f6c9147a a8802953e07d8d8 25 6fa231b814d04f1 e87b88548872333 c9 - default default] Delete attachment failed for attachment 4b21f601- 1bac-408f- 8985-eaba051cc8 1a. Error: The request you have made requires authentication. (HTTP 401) Code: 401: Unauthorized: The request you have made requires authentication. (HTTP 401) manager [req-6f8d526d- 4516-4162- 9111-ea261eee46 0a 5073df8f6c9147a a8802953e07d8d8 25 6fa231b814d04f1 e87b88548872333 c9 - default default] [instance: a2567a27- aa3f-4b16- 8fd9-fc96d2427d a3] Volume attachment 4b21f601- 1bac-408f- 8985-eaba051cc8 1a not deleted on source host dbl031623. dbl.sys11cloud. net during post_live_ migration: The request you have made requires authentication. (HTTP 401): Unauthorized: The request you have made requires authentication. (HTTP 401) manager [req-6f8d526d- 4516-4162- 9111-ea261eee46 0a 5073df8f6c9147a a8802953e07d8d8 25 6fa231b814d04f1 e87b88548872333 c9 - default default] [instance: a2567a27- aa3f-4b16- 8fd9-fc96d2427d a3] Post live migration at destination dbl021501. dbl.sys11cloud. net failed: NeutronAdminCre dentialConfigur ationInvalid_ Remote: Networking client is experiencing an unauthorized exception.
2021-09-06 11:20:55.934 48419 ERROR nova.compute.
2021-09-06 11:21:01.255 48419 ERROR nova.compute.
Traceback (most recent call last):
File "/usr/lib/ python2. 7/dist- packages/ oslo_messaging/ rpc/server. py", line 166, in _process_incoming .dispatch( message)
res = self.dispatcher
File "/usr/lib/ python2. 7/dist- packages/ oslo_messaging/ rpc/dispatcher. py", line 220, in dispatch dispatch( endpoint, method, ctxt, args)
return self._do_
File "/usr/lib/ python2. 7/dist- packages/ oslo_messaging/ rpc/dispatcher. py", line 190, in _do_dispatch
result = func(ctxt, **new_args)
File "/usr/lib/ python2. 7/dist- packages/ nova/exception_ wrapper. py", line 76, in wrapped
function_name, call_dict, binary)
File "/usr/lib/ python2. 7/dist- packages/ oslo_utils/ excutils. py", line 220, in __exit__ force_reraise( )
self.
File "/usr/lib/ python2. 7/dist- packages/ oslo_utils/ excutils. py", line 196, in force_reraise reraise( self.type_ , self.value, self.tb)
six.
File "/usr/lib/ python2. 7/dist- packages/ nova/exception_ wrapper. py", line 67, in wrapped
return f(self, context, *args, **kw)
File "/usr/lib/ python2. 7/dist- packages/ nova/compute/ utils.py" , line 1000, in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/ python2. 7/dist- packages/ nova/compute/ manager. py", line 216, in decorated_function 'instance' ], e, sys.exc_info())
kwargs[
File "/usr/lib/ python2. 7/dist- packages/ oslo_utils/ excutils. py", line 220, in __exit__ force_reraise( )
self.
File "/usr/lib/ python2. 7/dist- packages/ oslo_utils/ excutils. py", line 196, in force_reraise reraise( self.type_ , self.value, self.tb)
six.
File "/usr/lib/ python2. 7/dist- packages/ nova/compute/ manager. py", line 204, in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/ python2. 7/dist- packages/ nova/compute/ manager. py", line 6667, in post_live_ migration_ at_destination
self.host)
File "/usr/lib/ python2. 7/dist- packages/ nova/network/ neutronv2/ api.py" , line 344, in setup_networks_ on_host ports(context, **search_opts)
data = self.list_
File "/usr/lib/ python2. 7/dist- packages/ nova/network/ neutronv2/ api.py" , line 1401, in list_ports context) .list_ports( **search_ opts)
return get_client(
File "/usr/lib/ python2. 7/dist- packages/ nova/network/ neutronv2/ api.py" , line 114, in wrapper
ret = obj(*args, **kwargs)
File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 786, in list_ports
**_params)
File "/usr/lib/ python2. 7/dist- packages/ nova/network/ neutronv2/ api.py" , line 114, in wrapper python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 354, in get headers, params=params)
ret = obj(*args, **kwargs)
File "/usr/lib/
headers=
File "/usr/lib/ python2. 7/dist- packages/ nova/network/ neutronv2/ api.py" , line 114, in wrapper
ret = obj(*args, **kwargs)
File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 331, in retry_request headers, params=params)
headers=
File "/usr/lib/ python2. 7/dist- packages/ nova/network/ neutronv2/ api.py" , line 114, in wrapper
ret = obj(*args, **kwargs)
File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 294, in do_request _handle_ fault_response( status_ code, replybody, resp)
self.
File "/usr/lib/ python2. 7/dist- packages/ nova/network/ neutronv2/ api.py" , line 130, in wrapper NeutronAdminCre dentialConfigur ationInvalid( )
raise exception.
NeutronAdminCre dentialConfigur ationInvalid: Networking client is experiencing an unauthorized exception.