Comment 6 for bug 1657774

Revision history for this message
Olaf Seibert (oseibert-sys11) wrote (last edit ):

Has this bug been fixed in some other way? I think I ran into it on Queens when I tried to do a "nova live-migration-force-complete <id>".

The result was a big mess, with the VM left in some half-migrated state, because this error stopped processing of the migration in its tracks. The VM was running on the target host, but nova thought it was still migrating. The volume attachments were duplicated: once for the target, once for the source. We needed to remove the old attachments and change the nova database directly to fix the state.

2021-09-06 11:20:55.932 48419 ERROR nova.volume.cinder [req-6f8d526d-4516-4162-9111-ea261eee460a 5073df8f6c9147aa8802953e07d8d825 6fa231b814d04f1e87b88548872333c9 - default default] Delete attachment failed for attachment 4b21f601-1bac-408f-8985-eaba051cc81a. Error: The request you have made requires authentication. (HTTP 401) Code: 401: Unauthorized: The request you have made requires authentication. (HTTP 401)
2021-09-06 11:20:55.934 48419 ERROR nova.compute.manager [req-6f8d526d-4516-4162-9111-ea261eee460a 5073df8f6c9147aa8802953e07d8d825 6fa231b814d04f1e87b88548872333c9 - default default] [instance: a2567a27-aa3f-4b16-8fd9-fc96d2427da3] Volume attachment 4b21f601-1bac-408f-8985-eaba051cc81a not deleted on source host dbl031623.dbl.sys11cloud.net during post_live_migration: The request you have made requires authentication. (HTTP 401): Unauthorized: The request you have made requires authentication. (HTTP 401)
2021-09-06 11:21:01.255 48419 ERROR nova.compute.manager [req-6f8d526d-4516-4162-9111-ea261eee460a 5073df8f6c9147aa8802953e07d8d825 6fa231b814d04f1e87b88548872333c9 - default default] [instance: a2567a27-aa3f-4b16-8fd9-fc96d2427da3] Post live migration at destination dbl021501.dbl.sys11cloud.net failed: NeutronAdminCredentialConfigurationInvalid_Remote: Networking client is experiencing an unauthorized exception.
Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
    res = self.dispatcher.dispatch(message)

  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
    result = func(ctxt, **new_args)

  File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 76, in wrapped
    function_name, call_dict, binary)

  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()

  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)

  File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 67, in wrapped
    return f(self, context, *args, **kw)

  File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 1000, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 216, in decorated_function
    kwargs['instance'], e, sys.exc_info())

  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()

  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 204, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6667, in post_live_migration_at_destination
    self.host)

  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 344, in setup_networks_on_host
    data = self.list_ports(context, **search_opts)

  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1401, in list_ports
    return get_client(context).list_ports(**search_opts)

  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
    ret = obj(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 786, in list_ports
    **_params)

  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
    ret = obj(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 354, in get
    headers=headers, params=params)

  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
    ret = obj(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 331, in retry_request
    headers=headers, params=params)

  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
    ret = obj(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 294, in do_request
    self._handle_fault_response(status_code, replybody, resp)

  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 130, in wrapper
    raise exception.NeutronAdminCredentialConfigurationInvalid()

NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.