Rally tired waiting for Server to become SHELVED_OFFLOADED current status SHUTOFF

[Andrew Kalach]

Description
===========
During 9.2 Certification Testing Sutie run, the following task failed with error:

  NovaServers.boot_and_bounce_server

failed with error:

TimeoutException: Rally tired waiting for Server s_rally_76401ebb_Ozu0vAx9:8404a201-00f7-40d1-a240-7da12f3c5857 to become ('SHELVED_OFFLOADED') current statu

Steps to reproduce
==================
Install Rally on masternode and run specified rally tasks (see attachment):

1. cd /opt/stack
2. source .venv/bin/activate
3. rally task start <task YAML file>

Expected result
===============
Task run without errors

Actual result
=============
The following error logs:

2016-12-02 08:35:21.614 6117 ERROR rally.task.runner [-] Rally tired waiting for Server s_rally_76401ebb_Ozu0vAx9:8404a201-00f7-40d1-a240-7da12f3c5857 to become ('SHELVED_OFFLOADED') current status SHUTOFF
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner Traceback (most recent call last):
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/runner.py", line 73, in _run_scenario_once
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner getattr(scenario_inst, method_name)(**scenario_kwargs)
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/nova/servers.py", line 218, in run
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner action()
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 383, in _f
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner func(*args, **kwargs)
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner f = func(self, *args, **kwargs)
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/nova/utils.py", line 1153, in _shelve_and_unshelve_server
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner self._shelve_server(server)
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner f = func(self, *args, **kwargs)
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/nova/utils.py", line 389, in _shelve_server
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner check_interval=CONF.benchmark.nova_server_shelve_poll_interval
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/common/logging.py", line 236, in wrapper
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner return f(*args, **kwargs)
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 148, in wait_for
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner id_attr=id_attr)
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 251, in wait_for_status
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner resource_status=get_status(resource))
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner TimeoutException: Rally tired waiting for Server s_rally_76401ebb_Ozu0vAx9:8404a201-00f7-40d1-a240-7da12f3c5857 to become ('SHELVED_OFFLOADED') current status SHUTOFF
2016-12-02 08:35:21.614 6117 ERROR rally.task.runner
2016-12-02 08:35:21.616 6117 INFO rally.task.runner [-] Task 76401ebb-eea5-4fee-8354-2c1021582430 | ITER: 1158 END: Error TimeoutException: Rally tired waiting for Server s_rally_76401ebb_Ozu0vAx9:8404a201-00f7-40d1-a240-7da12f3c5857 to become ('SHELVED_OFFLOADED') current status SHUTOFF

Environment
===========
Scale 200-node lab (ENV-10):
  3 controllers
  200 compute nodes
  20 ceph nodes
Image build:
  Build-9.0.0-495_snap_557
Diagnostic snapshot:
  http://mos-scale-share.mirantis.com/fuel-dsc-161208.tar.gz

[Roman Podoliaka (rpodolyaka)]

Andrew,

I checked the logs you attached and apparently nova-compute ones are missing :(

eea5-4fee-8354-2c1021582430 | ITER: 1158 END: Error TimeoutException: Rally tired waiting for Server s_rally_76401ebb_Ozu0vAx9:8404a201-00f7-40d1-a240-7da12f3c5857 to become ('SHELVED_OFFLOADED') current status SHUTOFF

as it is only tells us that Rally timed out waiting for a VM to reach the target status, but does not imply there were any errors - the compute node might have been overloaded and, thus, Nova did not manage to complete the task in time.

Is there any chance you still have this environment available? Or maybe you can provide the logs from the compute node instance s_rally_76401ebb_Ozu0vAx9:8404a201-00f7-40d1-a240-7da12f3c5857 was running on?