At [1][2] the periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-ussuri fails during tempest tests with trace like
(tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON
test_delete_server_while_in_building_state[id-9e6e0c87-3352-42f7-9faf-5d6210dbd159])
Body: b'{"computeFault": {"code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\\n<class \'AttributeError\'>"}}'
}}}
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tempest/api/compute/servers/test_delete_server.py", line 43, in test_delete_server_while_in_building_state
self.client.delete_server(server['id'])
File "/usr/lib/python3.6/site-packages/tempest/lib/services/compute/servers_client.py", line 158, in delete_server
resp, body = self.delete("servers/%s" % server_id)
File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 314, in delete
return self.request('DELETE', url, extra_headers, headers, body)
File "/usr/lib/python3.6/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
method, url, extra_headers, headers, body, chunked)
File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 687, in request
self._error_checker(resp, resp_body)
File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 864, in _error_checker
message=message)
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'AttributeError'>
This blocks the Ussuri integration promotion pipeline (current-tripleo) [3][4]. Digging a bit at the nova api log [5] you can see:
2020-08-09 17:52:05.482 19 ERROR oslo.messaging._drivers.impl_rabbit [-] [cd0d7ebd-530f-44b7-a9b4-d46adc86ee2a] AMQP server on undercloud.ctlplane.localdomain:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
2020-08-09 17:52:05.483 19 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
[1] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-ussuri/45d7fc9/logs/undercloud/var/log/tempest/stestr_results.html.gz
[2] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-ussuri/0a45ef9/logs/undercloud/var/log/tempest/stestr_results.html.gz
[3] https://github.com/rdo-infra/ci-config/blob/6c71fee284d225f463b68fbe066856b0ddb522c6/ci-scripts/dlrnapi_promoter/config/CentOS-8/ussuri.ini#L45
[4] http://38.145.34.55/config/CentOS-8/ussuri.ini
[5] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-ussuri/0a45ef9/logs/undercloud/var/log/containers/nova/nova-api.log.txt.gz
From API logs,
https:/ /logserver. rdoproject. org/openstack- periodic- integration- stable1/ opendev. org/openstack/ tripleo- ci/master/ periodic- tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t035-ussuri/ 45d7fc9/ logs/overcloud- controller- 1/var/log/ containers/ nova/nova- api.log. txt.gz
2020-08-09 23:54:13.408 11 ERROR nova.api. openstack. wsgi [req-1305def1- 184b-4e3e- 9397-a033898bde b5 9a28aa13588843d 7a85454243e67b2 f7 0a253617f3f146f 3bceaf87cbced62 7f - default default] Unexpected exception in API method: AttributeError: 'NoneType' object has no attribute 'uuid' openstack. wsgi Traceback (most recent call last): openstack. wsgi File "/usr/lib/ python3. 6/site- packages/ nova/objects/ instance. py", line 655, in destroy openstack. wsgi hard_delete= hard_delete) openstack. wsgi File "/usr/lib/ python3. 6/site- packages/ nova/db/ api.py" , line 549, in instance_destroy openstack. wsgi hard_delete= hard_delete) openstack. wsgi File "/usr/lib/ python3. 6/site- packages/ nova/db/ sqlalchemy/ api.py" , line 179, in wrapper openstack. wsgi return f(*args, **kwargs) openstack. wsgi File "/usr/lib/ python3. 6/site- packages/ oslo_db/ api.py" , line 154, in wrapper openstack. wsgi ectxt.value = e.inner_exc openstack. wsgi File "/usr/lib/ python3. 6/site- packages/ oslo_utils/ excutils. py", line 220, in __exit__ openstack. wsgi self.force_ reraise( ) openstack. wsgi File "/usr/lib/ python3. 6/site- packages/ oslo_utils/ excutils. py", line 196, in force_reraise openstack. wsgi six.reraise( self.type_ , self.value, self.tb) openstack. wsgi File "/usr/lib/ python3. 6/site- packages/ six.py" , line 703, in reraise openstack. wsgi raise value openstack. wsgi File "/usr/lib/ python3. 6/site- packages/ oslo_db/ api.py" , line 142, in wrapper openstack. wsgi return f(*args, **kwargs) openstack. wsgi File "/usr/lib/ python3. 6/site- packages/ nova/db/ sqlalchemy/ api.py" , line 222, in wrapped openstack. wsgi return f(context, *args, **kwargs) openstack. wsgi File "/usr/lib/ python3. 6/site- packages/ nova/db/ sqlalchemy/ api.py" , line 1166, in instance_destroy openstack. wsgi raise exception. ConstraintNotMe t() openstack. wsgi nova.exception. ConstraintNotMe t: Constraint not met. openstack. wsgi openstack. wsgi During ...
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.
2020-08-09 23:54:13.408 11 ERROR nova.api.