tripleo

fs01 tripleo wallaby is failing consistently with the transient issues

Bug #1981337 reported by Soniya Murlidhar Vyas on 2022-07-11

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Won't Fix	Critical	Unassigned	tripleo zed-1

Bug Description

periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby is failing consistently with the transient issues.

1. undercloud installation failed

Traceback:-
====================

2022-07-11 03:23:17.035843 | fa163e9c-f480-373d-fbd3-000000002dfc | IGNORED | Run async deployment UndercloudPostPyDeployment | undercloud | error={"async_result": {"ansible_job_id": "868958368977.122800", "finished": 0, "invocation": {"module_args": {"_async_dir": "/tmp/.ansible_async", "jid": "868958368977.122800", "mode": "status"}}, "results_file": "/tmp/.ansible_async/868958368977.122800", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}, "changed": false, "msg": "async task did not complete within the requested time - 1800s"}
2022-07-11 03:23:17.037231 | fa163e9c-f480-373d-fbd3-000000002dfc | TIMING | Run async deployment UndercloudPostPyDeployment | undercloud | 0:47:59.481752 | 2036.07s
2022-07-11 03:23:17.063111 | fa163e9c-f480-373d-fbd3-000000002dfd | TASK | Output for async deployment UndercloudPostPyDeployment
2022-07-11 03:23:17.102506 | fa163e9c-f480-373d-fbd3-000000002dfd | FATAL | Output for async deployment UndercloudPostPyDeployment | undercloud | error={
    "msg": [
        {
            "stderr": [
                "Timed out"
            ]
        },
        {
            "status_code": "-1"
        }
    ]
}
Deployment Failed!

ERROR: Heat log files: /home/zuul/tripleo-deploy/undercloud/heat_launcher/tripleo_deploy-u38llu_m

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Exception: Ansible execution failed. playbook: deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1447, in take_action
    self._standalone_deploy(parsed_args)
  File "/usr/lib/python3.9/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1348, in _standalone_deploy
    utils.run_ansible_playbook(
  File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 739, in run_ansible_playbook
    raise RuntimeError(err_msg)
RuntimeError: Ansible execution failed. playbook: deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2
None
Ansible execution failed. playbook: deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2

2. Tempest tests failed

Traceback:-
=======================

2022-07-10 22:23:40,780 371125 ERROR [tempest.common.compute] Deleting server 1e905846-55f6-4e04-b7a4-908b90e57fdd failed
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute Traceback (most recent call last):
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 319, in create_test_server
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute waiters.wait_for_server_status(
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/common/waiters.py", line 67, in wait_for_server_status
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute body = client.show_server(server_id)['server']
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/servers_client.py", line 157, in show_server
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute resp, body = self.get("servers/%s" % server_id)
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 314, in get
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute return self.request('GET', url, extra_headers, headers)
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/base_compute_client.py", line 47, in request
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute resp, resp_body = super(BaseComputeClient, self).request(
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 703, in request
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute self._error_checker(resp, resp_body)
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 879, in _error_checker
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute raise exceptions.ServerFault(resp_body, resp=resp,
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute tempest.lib.exceptions.ServerFault: Got server fault
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute <class 'neutronclient.common.exceptions.InternalServerError'>
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute During handling of the above exception, another exception occurred:
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute Traceback (most recent call last):
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 337, in create_test_server
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute clients.servers_client.delete_server(
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/servers_client.py", line 170, in delete_server
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute resp, body = self.delete("servers/%s" % server_id)
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 330, in delete
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute return self.request('DELETE', url, extra_headers, headers, body)
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/base_compute_client.py", line 47, in request
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute resp, resp_body = super(BaseComputeClient, self).request(
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 703, in request
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute self._error_checker(resp, resp_body)
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 879, in _error_checker
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute raise exceptions.ServerFault(resp_body, resp=resp,
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute tempest.lib.exceptions.ServerFault: Got server fault
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute <class 'oslo_db.exception.DBConnectionError'>
2022-07-10 22:23:40.780 371125 ERROR tempest.common.compute
2022-07-10 22:23:40,812 371125 INFO [tempest.lib.common.rest_client] Request (VolumesActionsTest:test_force_detach_volume): 500 GET https://10.0.0.5:13774/v2.1/servers/1e905846-55f6-4e04-b7a4-908b90e57fdd 0.027s
2022-07-10 22:23:40,813 371125 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Sun, 10 Jul 2022 22:23:40 GMT', 'server': 'Apache', 'content-length': '209', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-a25c51d5-5b68-442f-ad3f-0524c9953c35', 'x-compute-request-id': 'req-a25c51d5-5b68-442f-ad3f-0524c9953c35', 'content-type': 'application/json; charset=UTF-8', 'connection': 'close', 'status': '500', 'content-location': 'https://10.0.0.5:13774/v2.1/servers/1e905846-55f6-4e04-b7a4-908b90e57fdd'}
        Body: b'{"computeFault": {"code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\\n<class \'oslo_db.exception.DBConnectionError\'>"}}'
2022-07-10 22:23:40,813 371125 ERROR [tempest.common.compute] Server 1e905846-55f6-4e04-b7a4-908b90e57fdd failed to delete in time
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute Traceback (most recent call last):
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 319, in create_test_server
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute waiters.wait_for_server_status(
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/common/waiters.py", line 67, in wait_for_server_status
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute body = client.show_server(server_id)['server']
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/servers_client.py", line 157, in show_server
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute resp, body = self.get("servers/%s" % server_id)
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 314, in get
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute return self.request('GET', url, extra_headers, headers)
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/base_compute_client.py", line 47, in request
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute resp, resp_body = super(BaseComputeClient, self).request(
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 703, in request
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute self._error_checker(resp, resp_body)
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 879, in _error_checker
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute raise exceptions.ServerFault(resp_body, resp=resp,
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute tempest.lib.exceptions.ServerFault: Got server fault
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute <class 'neutronclient.common.exceptions.InternalServerError'>
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute During handling of the above exception, another exception occurred:
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute Traceback (most recent call last):
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 350, in create_test_server
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute waiters.wait_for_server_termination(
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/common/waiters.py", line 109, in wait_for_server_termination
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute body = client.show_server(server_id)['server']
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/servers_client.py", line 157, in show_server
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute resp, body = self.get("servers/%s" % server_id)
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 314, in get
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute return self.request('GET', url, extra_headers, headers)
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/base_compute_client.py", line 47, in request
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute resp, resp_body = super(BaseComputeClient, self).request(
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 703, in request
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute self._error_checker(resp, resp_body)
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 879, in _error_checker
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute raise exceptions.ServerFault(resp_body, resp=resp,
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute tempest.lib.exceptions.ServerFault: Got server fault
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute <class 'oslo_db.exception.DBConnectionError'>
2022-07-10 22:23:40.813 371125 ERROR tempest.common.compute
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tempest/common/utils/__init__.py", line 70, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python3.9/site-packages/tempest/api/volume/admin/test_volumes_actions.py", line 83, in test_force_detach_volume
    server_id = self.create_server()['id']
  File "/usr/lib/python3.9/site-packages/tempest/api/volume/base.py", line 204, in create_server
    body, _ = compute.create_test_server(
  File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 353, in create_test_server
    LOG.exception('Server %s failed to delete in time',
  File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 319, in create_test_server
    waiters.wait_for_server_status(
  File "/usr/lib/python3.9/site-packages/tempest/common/waiters.py", line 67, in wait_for_server_status
    body = client.show_server(server_id)['server']
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/servers_client.py", line 157, in show_server
    resp, body = self.get("servers/%s" % server_id)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 314, in get
    return self.request('GET', url, extra_headers, headers)
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/base_compute_client.py", line 47, in request
    resp, resp_body = super(BaseComputeClient, self).request(
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 879, in _error_checker
    raise exceptions.ServerFault(resp_body, resp=resp,
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.InternalServerError'>

For more details please refer the following links:-
- https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/f3c6d44/logs/undercloud/var/log/tempest/stestr_results.html.gz
- https://logserver.rdoproject.org/83/40083/11/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/e089e55/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

Tags:

Soniya Murlidhar Vyas (svyas) on 2022-07-11

summary:

- fs01 tripleo wallaby fails with different issues
+ fs01 tripleo wallaby is failing consistently with the transient issues

Revision history for this message

Soniya Murlidhar Vyas (svyas) wrote on 2022-07-12 (last edit on 2022-07-12):

The job passed in the testproject yesterday[1]. But, seems failing again in tripleo component line, hence rekicked the job again

[1] https://review.rdoproject.org/zuul/build/f77676218ee74799a6e849b5c39b256a
[2] Rekicked here:- https://review.rdoproject.org/r/c/testproject/+/40083

Revision history for this message

Douglas Viroel (dviroel) wrote on 2022-07-12:

https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/f3c6d44/logs/overcloud-controller-0/var/log/extra/errors.txt.gz

We can see the following errors:

ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 771, in _commit_impl
ERROR oslo_db.sqlalchemy.exc_filters self.engine.dialect.do_commit(self.connection)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python3.9/site-packages/sqlalchemy/dialects/mysql/base.py", line 2501, in do_commit
ERROR oslo_db.sqlalchemy.exc_filters dbapi_connection.commit()
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 422, in commit
ERROR oslo_db.sqlalchemy.exc_filters self._read_ok_packet()
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 396, in _read_ok_packet
ERROR oslo_db.sqlalchemy.exc_filters pkt = self._read_packet()
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
ERROR oslo_db.sqlalchemy.exc_filters packet.raise_for_error()
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
ERROR oslo_db.sqlalchemy.exc_filters raise errorclass(errno, errval)
ERROR oslo_db.sqlalchemy.exc_filters pymysql.err.OperationalError: (1180, 'Got error 6 "No such device or address" during COMMIT')

Revision history for this message

Soniya Murlidhar Vyas (svyas) wrote on 2022-07-13:

The failure seems inconsistent because it has one green run and red on the same day and it happened twice. So, it is a bit instable[1]

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby&skip=0

Revision history for this message

Douglas Viroel (dviroel) wrote on 2022-07-14 (last edit on 2022-07-14):

Lots of DB connection issues like commented on https://bugs.launchpad.net/tripleo/+bug/1981478. Lets gather all the information into a single LP bug.

Douglas Viroel (dviroel) on 2022-07-14

Changed in tripleo:
status:	Triaged → Won't Fix

Soniya Murlidhar Vyas (svyas) on 2022-08-25

Changed in tripleo:
status:	Won't Fix → Triaged
status:	Triaged → Won't Fix

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.