cs9 fs35 master is failing on tempest tests with error 'tempest.exceptions.BuildErrorException: Server failed to build and is in ERROR status'

Bug #1981460 reported by Soniya Murlidhar Vyas
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Won't Fix
Critical
Unassigned

Bug Description

periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master is failing on tempest tests with error 'tempest.exceptions.BuildErrorException: Server failed to build and is in ERROR status'

Following is the traceback observed:-

1. tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic[compute,id-c5adff73-e961-41f1-b4a9-343614f18cfa,network]

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tempest/common/utils/__init__.py", line 70, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 534, in test_hotplug_nic
    self._setup_network_and_servers()
  File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 120, in _setup_network_and_servers
    server = self._create_server(self.network, port_id)
  File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 169, in _create_server
    server = self.create_server(
  File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 309, in create_server
    body, _ = compute.create_test_server(
  File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 353, in create_test_server
    LOG.exception('Server %s failed to delete in time',
  File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 319, in create_test_server
    waiters.wait_for_server_status(
  File "/usr/lib/python3.9/site-packages/tempest/common/waiters.py", line 81, in wait_for_server_status
    raise exceptions.BuildErrorException(details, server_id=server_id)
tempest.exceptions.BuildErrorException: Server c3c49352-be8e-4b8b-a76f-434cbfa0616a failed to build and is in ERROR status
Details: Fault: {'code': 500, 'created': '2022-07-12T03:37:16Z', 'message': 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c3c49352-be8e-4b8b-a76f-434cbfa0616a.'}. Server boot request ID: req-2b2488a6-8b19-46e9-bb51-ebb9be92e89e.

2. tempest.api.compute.images.test_images.ImagesTestJSON.test_create_image_from_stopped_server[id-aaacd1d0-55a2-4ce8-818a-b5439df8adc9]

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tempest/api/compute/images/test_images.py", line 83, in test_create_image_from_stopped_server
    waiters.wait_for_server_status(self.servers_client,
  File "/usr/lib/python3.9/site-packages/tempest/common/waiters.py", line 67, in wait_for_server_status
    body = client.show_server(server_id)['server']
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/servers_client.py", line 157, in show_server
    resp, body = self.get("servers/%s" % server_id)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 314, in get
    return self.request('GET', url, extra_headers, headers)
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/base_compute_client.py", line 47, in request
    resp, resp_body = super(BaseComputeClient, self).request(
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 720, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 896, in _error_checker
    raise exceptions.ServerFault(resp_body, resp=resp,
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_db.exception.DBConnectionError'>

For more details please refer following links:-
1. https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/f44e204/logs/undercloud/var/log/tempest/stestr_results.html.gz
2. https://logserver.rdoproject.org/60/43560/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/61be28c/logs/undercloud/var/log/tempest/stestr_results.html.gz
3. https://logserver.rdoproject.org/60/43560/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/5cbf6ab/logs/undercloud/var/log/tempest/stestr_results.html.gz

Revision history for this message
Douglas Viroel (dviroel) wrote :
Download full text (3.3 KiB)

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/f44e204/logs/overcloud-controller-0/var/log/extra/errors.txt.gz

ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi self.dialect.do_execute(
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi cursor.execute(statement, parameters)
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi result = self._query(query)
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi conn.query(q)
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi self._affected_rows = self._read_query_result(unbuffered=unbuffered)
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi result.read()
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi first_packet = self.connection._read_packet()
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi packet.raise_for_error()
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi err.raise_mysql_exception(self._data)
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
ERROR /var/log/containers/nova/nova-api.log: 9 ERROR nova.api.openstack.wsgi raise errorclass(errno, errval)
ERROR /var/log/containers/no...

Read more...

Revision history for this message
Soniya Murlidhar Vyas (svyas) wrote :
Revision history for this message
Douglas Viroel (dviroel) wrote :

DB errors as reported in https://bugs.launchpad.net/tripleo/+bug/1981478. Lets close this one and add more info there.

Changed in tripleo:
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.