tempest.scenario.test_large_ops.test_large_ops_scenario fails

[Guangya Liu (Jay Lau)]

2013-11-07 12:04:53.646 | x-openstack-request-id: req-5ff80347-150a-4739-a468-d1fdbffe5384
2013-11-07 12:04:53.646 | }}}
2013-11-07 12:04:53.646 |
2013-11-07 12:04:53.647 | Traceback (most recent call last):
2013-11-07 12:04:53.647 | File "tempest/scenario/test_large_ops.py", line 105, in test_large_ops_scenario
2013-11-07 12:04:53.647 | self.nova_boot()
2013-11-07 12:04:53.647 | File "tempest/scenario/test_large_ops.py", line 98, in nova_boot
2013-11-07 12:04:53.648 | self._wait_for_server_status('ACTIVE')
2013-11-07 12:04:53.648 | File "tempest/scenario/test_large_ops.py", line 42, in _wait_for_server_status
2013-11-07 12:04:53.648 | self.compute_client.servers, server.id, status)
2013-11-07 12:04:53.649 | File "tempest/scenario/manager.py", line 295, in status_timeout
2013-11-07 12:04:53.649 | not_found_exception=not_found_exception)
2013-11-07 12:04:53.649 | File "tempest/scenario/manager.py", line 352, in _status_timeout
2013-11-07 12:04:53.649 | raise exceptions.TimeoutException(message)
2013-11-07 12:04:53.650 | TimeoutException: Request timed out
2013-11-07 12:04:53.650 | Details: Timed out waiting for thing b13ac6d2-af5d-4035-a87b-1fbfd1bde5bd to become ACTIVE
2013-11-07 12:04:53.650 | ======================================================================
2013-11-07 12:04:53.651 | FAIL: process-returncode
2013-11-07 12:04:53.651 | tags: worker-0
2013-11-07 12:04:53.651 | ----------------------------------------------------------------------
2013-11-07 12:04:53.651 | Binary content:
2013-11-07 12:04:53.652 | traceback (test/plain; charset="utf8")
2013-11-07 12:04:53.652 | Ran 2 tests in 618.004s
2013-11-07 12:04:53.652 | FAILED (id=0, failures=2)
2013-11-07 12:04:53.653 | error: testr failed (1)

[Guangya Liu (Jay Lau) (jay-lau-513)]

Please go to http://logs.openstack.org/10/55110/4/check/gate-tempest-devstack-vm-neutron-large-ops/ba55c1b/console.html for detail.

[Christopher Yeoh (cyeoh-0)]

This is re-occuring in the gate: http://logs.openstack.org/91/58191/2/check/gate-tempest-devstack-vm-neutron-large-ops/ba576e6/

[Christopher Yeoh (cyeoh-0)]

Looks like the neutron network allocation is taking so long for the 150 servers that the test times out. Grep for 'Instance network_info:' in the screen-n-cpu log and compare to when the test times out. Its only slow by a few seconds which is why the test only fails sometimes. Perhaps we should drop TEMPEST_LARGE_OPS_NUMBER a little bit so it fits within the timeout?

[Christopher Yeoh (cyeoh-0)]

Doing a bit of debugging here: https://review.openstack.org/#/c/58422/