DX: ERROR Caught exception while trying to get a volume

[Sivonaldo Diogo Santos Silva]

Brief Description
-----------------
Controller is failing after lock/unlock and swact and getting error related with volume

Severity
--------
Provide the severity of the defect.
Critical

Steps to Reproduce
------------------
create a vm and backup that vm then do a lock/unlock controller-1 swact to that controller and do same procedure lock/unlock controller-0 the controller get in a fail state.

Expected Behavior
------------------
controller get in an available state after lock/unlock and swact

Actual Behavior
----------------
controller get in fail state after lock/unlock and swact

Reproducibility
---------------
intermittent

System Configuration
--------------------
Two node system DX

Branch/Pull Time/Commit
-----------------------
/StarlingX_Upstream_build/2021-04-19_06-00-00

Last Pass
---------
unknow

Timestamp/Logs
--------------

2021-04-20T05:49:44.256 controller-1 VIM_Thread[472480] ERROR Caught exception while trying to get a volume, error=[OpenStack Exception: method=GET, url=http://cinder.openstack.svc.cluster.local/v2/77e088b60e2c4220980354ada4e272c4/volumes/a2966ad4-8dce-42c8-9410-4151812ca524, headers={}, body=None, reason=timed out].
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nfv_plugins/nfvi_plugins/nfvi_block_storage_api.py", line 436, in get_volume
    future.result = (yield)

Test Activity
-------------
Feature Testing

[Austin Sun (sunausti)]

would you like to provide "collect log " , is it openstack re-applied successfully?

[zhipeng liu (zhipengs)]

From your log, it looks like a timeout.
You can test it on other setup since it is an intermittent issue.

Thanks!
Zhipeng

[Ghada Khalil (gkhalil)]

screening: Given this is an intermittent issue, it will not hold up stx.5.0, so will mark it for stx.6.0 until further investigation by the distro.openstack team.

[Sivonaldo Diogo Santos Silva (sdiogosa)]

The error did happen again in a new installation

021-04-27T07:33:50.859 controller-0 VIM_Thread[1007310] ERROR Caught exception while trying to get volume list, error=[OpenStack Exception: method=GET, url=http://cinder.openstack.svc.cluster.local/v2/dbaa031b8c9b4fbdac1bf1aafdc7a8e7/volumes?limit=32&all_tenants=1, headers={}, body=None, reason=timed out].
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nfv_plugins/nfvi_plugins/nfvi_block_storage_api.py", line 140, in get_volumes
    future.result = (yield)
OpenStackException: [OpenStack Exception: method=GET, url=http://cinder.openstack.svc.cluster.local/v2/dbaa031b8c9b4fbdac1bf1aafdc7a8e7/volumes?limit=32&all_tenants=1, headers={}, body=None, reason=timed out]
2021-04-27T07:33:50.859 controller-0 VIM_Thread[1007310] ERROR _vim_nfvi_audits.py.635 Audit-Volumes callback, not completed, responses={'completed': False, 'reason': '', 'page-request-id': 'd2f2bf4c-048a-478c-9ced-2d130f97a061'}.
2021-04-27T07:33:50.857 controller-0 VIM_BlockStorage-Worker-0_Thread[1007686] INFO _timer_scheduler.py.57 Not scheduling on time, elapsed=20500 ms.
2021-04-27T07:33:50.860 controller-0 VIM_BlockStorage-Worker-0_Thread[1007686] INFO _thread.py.218 Thread BlockStorage-Worker-0: not scheduling on time
2021-04-27T07:33:51.361 controller-0 VIM_BlockStorage-Worker-0_Thread[1007686] INFO _thread.py.218 Thread BlockStorage-Worker-0: not scheduling on time
2021-04-27T07:33:51.432 controller-0 VIM_BlockStorage-Worker-0_Thread[1007686] INFO _thread.py.218 Thread BlockStorage-Worker-0: not scheduling on time
2021-04-27T07:33:51.934 controller-0 VIM_BlockStorage-Worker-0_Thread[1007686] INFO _thread.py.218 Thread BlockStorage-Worker-0: not scheduling on time
2021-04-27T07:33:52.434 controller-0 VIM_BlockStorage-Worker-0_Thread[1007686] INFO _thread.py.218 Thread BlockStorage-Worker-0: not scheduling on time
2021-04-27T07:33:52.935 controller-0 VIM_BlockStorage-Worker-0_Thread[1007686] INFO _thread.py.218 Thread BlockStorage-Worker-0: not scheduling on time
2021-04-27T07:33:53.436 controller-0 VIM_BlockStorage-Worker-0_Thread[1007686] INFO _thread.py.218 Thread BlockStorage-Worker-0: not scheduling on time
2021-04-27T07:33:53.937 controller-0 VIM_BlockStorage-Worker-0_Thread[1007686] INFO _thread.py.218 Thread BlockStorage-Worker-0: not scheduling on time
2021-04-27T07:33:54.438 controller-0 VIM_BlockStorage-Worker-0_Thread[1007686] INFO _thread.py.218 Thread BlockStorage-Worker-0: not scheduling on time
2021-04-27T07:33:54.939 controller-0 VIM_BlockStorage-Worker-0_Thread[1007686] INFO _thread.py.218 Thread BlockStorage-Worker-0: not scheduling on time
2021-04-27T...

[Sivonaldo Diogo Santos Silva (sdiogosa)]

this issue no longer can be reproduced