In neutron ci jobs we noticed couple of times last week failures related to shelve/unshelve instance tests.
Failed test: tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_shelve_unshelve_server
Example of failure: https://8efce70e72afb45d6ec7-62d3a5548ea094caef4a9963ba6c55d1.ssl.cf5.rackcdn.com/880461/6/check/neutron-ovs-tempest-multinode-full/1ee5b2f/testr_results.html
Stacktrace:
traceback-1: {{{
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 80, in tearDown
self.server_check_teardown()
File "/opt/stack/tempest/tempest/api/compute/base.py", line 228, in server_check_teardown
waiters.wait_for_server_status(cls.servers_client,
File "/opt/stack/tempest/tempest/common/waiters.py", line 101, in wait_for_server_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (ServerActionsTestJSON:tearDown) Server b8b508ae-61ab-4501-b116-9574a1de0045 failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: SHELVED_OFFLOADED. Current task state: None.
}}}
traceback-2: {{{
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 705, in _unshelve_server
server_info = self.client.show_server(self.server_id)['server']
File "/opt/stack/tempest/tempest/lib/services/compute/servers_client.py", line 157, in show_server
resp, body = self.get("servers/%s" % server_id)
File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 322, in get
return self.request('GET', url, extra_headers, headers,
File "/opt/stack/tempest/tempest/lib/services/compute/base_compute_client.py", line 47, in request
resp, resp_body = super(BaseComputeClient, self).request(
File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request
self._error_checker(resp, resp_body)
File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 852, in _error_checker
raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {'code': 404, 'message': 'Instance None could not be found.'}
}}}
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 70, in wrapper
return f(*func_args, **func_kwargs)
File "/opt/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 722, in test_shelve_unshelve_server
waiters.wait_for_server_status(self.client, self.server_id, 'ACTIVE')
File "/opt/stack/tempest/tempest/common/waiters.py", line 101, in wait_for_server_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (ServerActionsTestJSON:test_shelve_unshelve_server) Server b8b508ae-61ab-4501-b116-9574a1de0045 failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: SHELVED_OFFLOADED. Current task state: spawning.
Artom suggested that the first step could be to add a trace that would allow for easy tracking of Tempest's log, thereby facilitating an understanding of what went wrong with the request.
Additionally, in my opinion, the _unshelve_server method seems unnecessary. I plan to address both of these points by submitting patches to rectify them.