DX: Audit-Instance callback, not completed

[Sivonaldo Diogo Santos Silva]

Brief Description
-----------------
nova API audit-Instance callback, not completed after lock/unlock

Severity
--------
Critical

Steps to Reproduce
------------------
create a VM then a backup of that VM after that lock/unlock controller-1 then swact to controller-1 and lock/unlock controller-0

Expected Behavior
------------------
Audit-Instance callback, has to be completed

Actual Behavior
----------------
Audit-Instance callback, has not completed causing an exception

Reproducibility
---------------
intermittent

System Configuration
--------------------
Multi-node system DX

Branch/Pull Time/Commit
-----------------------
StarlingX_Upstream_build/2021-04-19_06-00-00

Last Pass
---------
unknow

Timestamp/Logs
--------------
OpenStackRestAPIException: [OpenStack Rest-API Exception: method=GET, url=http://cinder.openstack.svc.cluster.local/v2/77e088b60e2c4220980354ada4e272c4/volumes?limit=32&all_tenants=1, headers={}, body=None, status_code=500, reason=HTTP Error 500: Internal Server Error, response_headers=[('Content-Length', '128'), ('X-Compute-Request-Id', 'req-63a068e1-ad39-4ef6-8281-26171ded9846'), ('Date', 'Tue, 20 Apr 2021 05:38:54 GMT'), ('Content-Type', 'application/json'), ('X-Openstack-Request-Id', 'req-63a068e1-ad39-4ef6-8281-26171ded9846')], response_body={"computeFault": {"code": 500, "message": "The server has either erred or is incapable of performing the requested operation."}}]
2021-04-20T05:38:54.884 controller-1 VIM_Thread[472480] ERROR _vim_nfvi_audits.py.635 Audit-Volumes callback, not completed, responses={'completed': False, 'reason': '', 'page-request-id': 'ba6c7033-dc7a-454f-91dc-fbfee7f783b7'}.
2021-04-20T05:38:56.986 controller-1 VIM_Thread[472480] ERROR Caught exception while trying to get volume list, error=[OpenStack Rest-API Exception: method=GET, url=http://cinder.openstack.svc.cluster.local/v2/77e088b60e2c4220980354ada4e272c4/snapshots?all_tenants=1, headers={}, body=None, status_code=500, reason=HTTP Error 500: Internal Server Error, response_headers=[('Content-Length', '128'), ('X-Compute-Request-Id', 'req-fcb673c8-c8f8-40da-a88d-a20243ac4745'), ('Date', 'Tue, 20 Apr 2021 05:38:56 GMT'), ('Content-Type', 'application/json'), ('X-Openstack-Request-Id', 'req-fcb673c8-c8f8-40da-a88d-a20243ac4745')], response_body={"computeFault": {"code": 500, "message": "The server has either erred or is incapable of performing the requested operation."}}].
Traceback (most recent call last):

OpenStackRestAPIException: [OpenStack Rest-API Exception: method=GET, url=http://nova-api-internal.openstack.svc.cluster.local:80/v2.1/77e088b60e2c4220980354ada4e272c4/servers/24bc3ad2-cecf-434b-bda9-4ecd5f1023a6, headers={'X-OpenStack-Nova-API-Version': '2.53'}, body=None, status_code=500, reason=HTTP Error 500: Internal Server Error, response_headers=[('Content-Length', '199'), ('X-Compute-Request-Id', 'req-dc7d76b3-311b-49ce-8fc8-3f1a4c0c215a'), ('Vary', 'OpenStack-API-Version, X-OpenStack-Nova-API-Version'), ('Openstack-Api-Version', 'compute 2.53'), ('X-Openstack-Nova-Api-Version', '2.53'), ('Date', 'Tue, 20 Apr 2021 05:38:57 GMT'), ('Content-Type', 'application/json; charset=UTF-8'), ('X-Openstack-Request-Id', 'req-dc7d76b3-311b-49ce-8fc8-3f1a4c0c215a')], response_body={"computeFault": {"code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'oslo_db.exception.DBError'>"}}]

2021-04-20T05:38:57.652 controller-1 VIM_Thread[472480] ERROR _vim_nfvi_audits.py.970 Audit-Instance callback, not completed, response={'completed': False, 'reason': "unexpected api error. please report this at http://bugs.launchpad.net/nova/ and attach the nova api log if possible.\n<class 'oslo_db.exception.dberror'>"}

Test Activity
-------------
Feature Testing

[Austin Sun (sunausti)]

Silva: is openstack application applied successfully ? would you like upload collect logs for further analysis?

[Sivonaldo Diogo Santos Silva (sdiogosa)]

it failed to apply the openstack and in the log for appy have this

n deploying charts: ['openstack-mariadb']
2021-04-20 12:03:53.860 133 ERROR armada.cli Traceback (most recent call last):
2021-04-20 12:03:53.860 133 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2021-04-20 12:03:53.860 133 ERROR armada.cli self.invoke()
2021-04-20 12:03:53.860 133 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2021-04-20 12:03:53.860 133 ERROR armada.cli resp = self.handle(documents, tiller)
2021-04-20 12:03:53.860 133 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2021-04-20 12:03:53.860 133 ERROR armada.cli return future.result()
2021-04-20 12:03:53.860 133 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2021-04-20 12:03:53.860 133 ERROR armada.cli return self.__get_result()
2021-04-20 12:03:53.860 133 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-04-20 12:03:53.860 133 ERROR armada.cli raise self._exception
2021-04-20 12:03:53.860 133 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2021-04-20 12:03:53.860 133 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2021-04-20 12:03:53.860 133 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle
2021-04-20 12:03:53.860 133 ERROR armada.cli return armada.sync()
2021-04-20 12:03:53.860 133 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 252, in sync
2021-04-20 12:03:53.860 133 ERROR armada.cli raise armada_exceptions.ChartDeployException(failures)
2021-04-20 12:03:53.860 133 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openstack-mariadb']

[Austin Sun (sunausti)]

same as 1922299, once fixed, please verify it.