tempest.api.compute.admin.test_servers.ServersAdminTestJSON.test_rebuild_server_in_error_state[gate]

[Gavin B]

A gating run for https://review.openstack.org/#/c/65481/ failed. This is the 3rd time the tests have run for this particular minor change to python-novaclient, but the first time that this particular failure was seen.

In http://logs.openstack.org/81/65481/1/check/check-tempest-dsvm-full/9d98635/console.html I see :

2014-01-09 14:46:26.569 | FAIL: tempest.api.compute.admin.test_servers.ServersAdminTestJSON.test_rebuild_server_in_error_state[gate]
2014-01-09 14:46:26.570 | tempest.api.compute.admin.test_servers.ServersAdminTestJSON.test_rebuild_server_in_error_state[gate]

<snip>

2014-01-09 14:46:26.946 | 2014-01-09 14:13:13,125 Request Headers: {'X-Auth-Token': '<Token omitted>'}
2014-01-09 14:46:26.946 | 2014-01-09 14:13:13,521 Response Status: 200
2014-01-09 14:46:26.946 | 2014-01-09 14:13:13,522 Nova request id: req-bc0864aa-524c-43ba-aa49-0bea5d11355d
2014-01-09 14:46:26.946 | 2014-01-09 14:13:13,522 Response Headers: {'content-length': '1532', 'content-location': u'http://127.0.0.1:8774/v2/2dfdcf9222984ca8ad77e082c28ae3b7/servers/6e495137-7a4e-4745-a767-ccafe9145e07', 'date': 'Thu, 09 Jan 2014 14:13:13 GMT', 'content-type': 'application/json', 'connection': 'close'}
2014-01-09 14:46:26.946 | 2014-01-09 14:13:13,523 Response Body: {"server": {"status": "ERROR", "updated": "2014-01-09T14:09:53Z", "hostId": "c461fbd56e33082c15b545ccf7468971d30eee32ea0320f67496c195", "addresses": {"private": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:9b:2c:6c", "version": 4, "addr": "10.1.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://127.0.0.1:8774/v2/2dfdcf9222984ca8ad77e082c28ae3b7/servers/6e495137-7a4e-4745-a767-ccafe9145e07", "rel": "self"}, {"href": "http://127.0.0.1:8774/2dfdcf9222984ca8ad77e082c28ae3b7/servers/6e495137-7a4e-4745-a767-ccafe9145e07", "rel": "bookmark"}], "key_name": null, "image": {"id": "29908536-97d6-4a24-9d1d-85e97239ea43", "links": [{"href": "http://127.0.0.1:8774/2dfdcf9222984ca8ad77e082c28ae3b7/images/29908536-97d6-4a24-9d1d-85e97239ea43", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": "rebuilding", "OS-EXT-STS:vm_state": "error", "OS-SRV-USG:launched_at": "2014-01-09T14:09:34.000000", "flavor": {"id": "42", "links": [{"href": "http://127.0.0.1:8774/2dfdcf9222984ca8ad77e082c28ae3b7/flavors/42", "rel": "bookmark"}]}, "id": "6e495137-7a4e-4745-a767-ccafe9145e07", "security_groups": [{"name": "default"}], "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "b811f0c47f2c4d03bc94da187e7c664a", "name": "server-tempest-1130342031", "created": "2014-01-09T14:09:20Z", "tenant_id": "2dfdcf9222984ca8ad77e082c28ae3b7", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 1, "config_drive": "", "metadata": {}}}
2014-01-09 14:46:26.946 | }}}
2014-01-09 14:46:26.947 |
2014-01-09 14:46:26.947 | traceback-1: {{{
2014-01-09 14:46:26.947 | Traceback (most recent call last):
2014-01-09 14:46:26.947 | File "tempest/services/compute/json/servers_client.py", line 240, in rebuild
2014-01-09 14:46:26.947 | return self.action(server_id, 'rebuild', 'server', **kwargs)
2014-01-09 14:46:26.947 | File "tempest/services/compute/json/servers_client.py", line 198, in action
2014-01-09 14:46:26.947 | post_body, self.headers)
2014-01-09 14:46:26.947 | File "tempest/common/rest_client.py", line 302, in post
2014-01-09 14:46:26.947 | return self.request('POST', url, headers, body)
2014-01-09 14:46:26.947 | File "tempest/common/rest_client.py", line 436, in request
2014-01-09 14:46:26.947 | resp, resp_body)
2014-01-09 14:46:26.947 | File "tempest/common/rest_client.py", line 491, in _error_checker
2014-01-09 14:46:26.948 | raise exceptions.Conflict(resp_body)
2014-01-09 14:46:26.948 | Conflict: An object with that identifier already exists
2014-01-09 14:46:26.948 | Details: {u'conflictingRequest': {u'message': u"Cannot 'rebuild' while instance is in task_state rebuilding", u'code': 409}}
2014-01-09 14:46:26.948 | }}}
2014-01-09 14:46:26.948 |
2014-01-09 14:46:26.948 | traceback-2: {{{
2014-01-09 14:46:26.948 | Traceback (most recent call last):
2014-01-09 14:46:26.948 | File "tempest/services/compute/json/servers_client.py", line 162, in wait_for_server_status
2014-01-09 14:46:26.948 | raise_on_error=raise_on_error)
2014-01-09 14:46:26.948 | File "tempest/common/waiters.py", line 76, in wait_for_server_status
2014-01-09 14:46:26.948 | raise exceptions.BuildErrorException(server_id=server_id)
2014-01-09 14:46:26.948 | BuildErrorException: Server 6e495137-7a4e-4745-a767-ccafe9145e07 failed to build and is in ERROR status
2014-01-09 14:46:26.949 | }}}
2014-01-09 14:46:26.949 |
2014-01-09 14:46:26.949 | Traceback (most recent call last):
2014-01-09 14:46:26.949 | File "tempest/api/compute/admin/test_servers.py", line 138, in test_rebuild_server_in_error_state
2014-01-09 14:46:26.949 | raise_on_error=False)
2014-01-09 14:46:26.949 | File "tempest/services/compute/json/servers_client.py", line 162, in wait_for_server_status
2014-01-09 14:46:26.949 | raise_on_error=raise_on_error)
2014-01-09 14:46:26.949 | File "tempest/common/waiters.py", line 91, in wait_for_server_status
2014-01-09 14:46:26.949 | raise exceptions.TimeoutException(message)
2014-01-09 14:46:26.949 | TimeoutException: Request timed out
2014-01-09 14:46:26.949 | Details: Server 6e495137-7a4e-4745-a767-ccafe9145e07 failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: ERROR. Current task state: rebuilding.
2014-01-09 14:46:26.949 |
2014-01-09 14:46:26.950 |
2014-01-09 14:46:26.950 | ======================================================================
2014-01-09 14:46:26.950 | FAIL: process-returncode
2014-01-09 14:46:26.950 | process-returncode
2014-01-09 14:46:26.950 | ----------------------------------------------------------------------
2014-01-09 14:46:26.950 | _StringException: Binary content:
2014-01-09 14:46:26.950 | traceback (test/plain; charset="utf8")
2014-01-09 14:46:26.950 |
2014-01-09 14:46:26.950 |
2014-01-09 14:46:26.950 | ----------------------------------------------------------------------
2014-01-09 14:46:26.950 | Ran 2251 tests in 2374.814s
2014-01-09 14:46:26.950 |
2014-01-09 14:46:26.951 | FAILED (failures=2, skipped=128)
2014-01-09 14:46:26.951 | ERROR: InvocationError: '/bin/bash tools/pretty_tox.sh (?!.*\\[.*\\bslow\\b.*\\])(^tempest\\.(api|scenario|thirdparty|cli)) '
2014-01-09 14:46:26.951 | ___________________________________ summary ____________________________________
2014-01-09 14:46:26.951 | ERROR: full: commands failed
2014-01-09 14:48:22.369 | Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information

[Gavin B (gavin-brebner-orange)]

possible dup of https://bugs.launchpad.net/nova/+bug/1262153 ?