tripleo

tripleo-standalone centos-8 tempest fail: estSnapshotPattern:_run_cleanups): 500 DELETE http://192.168.24.3:9292/v2/images

Bug #1895601 reported by wes hayutin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Triaged
Critical
Unassigned
tripleo victoria-3 "tripleo victoria"

Bug Description

2020-09-14 22:38:06,056 229045 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Mon, 14 Sep 2020 22:38:05 GMT', 'server': 'Apache', 'content-length': '209', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-eefe2959-7561-4af4-937d-220d2ace80ae', 'x-compute-request-id': 'req-eefe2959-7561-4af4-937d-220d2ace80ae', 'connection': 'close', 'content-type': 'application/json; charset=UTF-8', 'status': '500', 'content-location': 'http://192.168.24.3:8774/v2.1/servers/1afbe35e-568c-4544-b9c4-4d350aac67fb'}
        Body: b'{"computeFault": {"code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\\n<class \'oslo_db.exception.DBConnectionError\'>"}}'
2020-09-14 22:38:06,057 229045 ERROR [tempest.common.compute] Deleting server 1afbe35e-568c-4544-b9c4-4d350aac67fb failed
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute Traceback (most recent call last):
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/common/compute.py", line 242, in create_test_server
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute clients.servers_client, server['id'], wait_until)
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 65, in wait_for_server_status
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute body = client.show_server(server_id)['server']
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/services/compute/servers_client.py", line 145, in show_server
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute resp, body = self.get("servers/%s" % server_id)
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 313, in get
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute return self.request('GET', url, extra_headers, headers)
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute method, url, extra_headers, headers, body, chunked)
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 702, in request
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute self._error_checker(resp, resp_body)
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 879, in _error_checker
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute message=message)
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute tempest.lib.exceptions.ServerFault: Got server fault
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute <class 'oslo_db.exception.DBConnectionError'>
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute During handling of the above exception, another exception occurred:
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute Traceback (most recent call last):
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/common/compute.py", line 255, in create_test_server
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute server['id'])
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/services/compute/servers_client.py", line 158, in delete_server
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute resp, body = self.delete("servers/%s" % server_id)
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 329, in delete
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute return self.request('DELETE', url, extra_headers, headers, body)
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute method, url, extra_headers, headers, body, chunked)
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 702, in request
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute self._error_checker(resp, resp_body)
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 879, in _error_checker
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute message=message)
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute tempest.lib.exceptions.ServerFault: Got server fault
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute <class 'oslo_db.exception.DBConnectionError'>
2020-09-14 22:38:06.057 229045 ERROR tempest.common.compute
2020-09-14 22:38:06,141 229045 INFO [tempest.lib.common.rest_client] Request (TestSnapshotPattern:test_snapshot_pattern): 500 GET http://192.168.24.3:8774/v2.1/servers/1afbe35e-568c-4544-b9c4-4d350aac67fb 0.048s
2020-09-14 22:38:06,142 229045 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Mon, 14 Sep 2020 22:38:06 GMT', 'server': 'Apache', 'content-length': '209', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-32769c6d-7d7c-411c-a008-93f945d4076b', 'x-compute-request-id': 'req-32769c6d-7d7c-411c-a008-93f945d4076b', 'connection': 'close', 'content-type': 'application/json; charset=UTF-8', 'status': '500', 'content-location': 'http://192.168.24.3:8774/v2.1/servers/1afbe35e-568c-4544-b9c4-4d350aac67fb'}
        Body: b'{"computeFault": {"code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\\n<class \'oslo_db.exception.DBConnectionError\'>"}}'
2020-09-14 22:38:06,142 229045 ERROR [tempest.common.compute] Server 1afbe35e-568c-4544-b9c4-4d350aac67fb failed to delete in time
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute Traceback (most recent call last):
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/common/compute.py", line 242, in create_test_server
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute clients.servers_client, server['id'], wait_until)
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 65, in wait_for_server_status
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute body = client.show_server(server_id)['server']
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/services/compute/servers_client.py", line 145, in show_server
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute resp, body = self.get("servers/%s" % server_id)
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 313, in get
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute return self.request('GET', url, extra_headers, headers)
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute method, url, extra_headers, headers, body, chunked)
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 702, in request
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute self._error_checker(resp, resp_body)
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 879, in _error_checker
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute message=message)
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute tempest.lib.exceptions.ServerFault: Got server fault
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute <class 'oslo_db.exception.DBConnectionError'>
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute During handling of the above exception, another exception occurred:
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute Traceback (most recent call last):
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/common/compute.py", line 268, in create_test_server
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute clients.servers_client, server['id'])
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 104, in wait_for_server_termination
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute body = client.show_server(server_id)['server']
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/services/compute/servers_client.py", line 145, in show_server
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute resp, body = self.get("servers/%s" % server_id)
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 313, in get
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute return self.request('GET', url, extra_headers, headers)
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute method, url, extra_headers, headers, body, chunked)
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 702, in request
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute self._error_checker(resp, resp_body)
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 879, in _error_checker
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute message=message)
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute tempest.lib.exceptions.ServerFault: Got server fault
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute <class 'oslo_db.exception.DBConnectionError'>
2020-09-14 22:38:06.142 229045 ERROR tempest.common.compute
2020-09-14 22:38:06,684 229045 INFO [tempest.lib.common.rest_client] Request (TestSnapshotPattern:_run_cleanups): 500 DELETE http://192.168.24.3:9292/v2/images/3243edb3-0ae7-4ee0-b336-fce750c7a80a 0.519s

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54f/751828/2/gate/tripleo-ci-centos-8-scenario001-standalone/54f184e/logs/undercloud/var/log/tempest/stestr_results.html

Revision history for this message
wes hayutin (weshayutin) wrote :

https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario001-standalone

Revision history for this message
wes hayutin (weshayutin) wrote :

2020-09-14 22:38:19.851 ERROR /var/log/containers/cinder/cinder-api.log: 11 ERROR oslo_db.sqlalchemy.engines oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (TimeoutError(110, 'Connection timed out'))")

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54f/751828/2/gate/tripleo-ci-centos-8-scenario001-standalone/54f184e/logs/undercloud/var/log/extra/errors.txt.txt

Revision history for this message
yatin (yatinkarel) wrote :

https://review.opendev.org/#/c/752187/ will clear the issue, reverting the skipped tests with https://review.opendev.org/#/c/752209/, also proposed https://review.rdoproject.org/r/29356 to sync missing tempest tests from periodic counterparts.

Revision history for this message
Giulio Fidente (gfidente) wrote :

nova is logging db connection errors as well [1] at 22:38:05 which is when the test was executed and failed [2]

mysql is logging connection issues [3] like "(Got an error reading communication packets)" from pretty much all services starting at 22:37:31

1. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54f/751828/2/gate/tripleo-ci-centos-8-scenario001-standalone/54f184e/logs/undercloud/var/log/containers/nova/nova-api.log

2. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54f/751828/2/gate/tripleo-ci-centos-8-scenario001-standalone/54f184e/logs/undercloud/var/log/tempest/stestr_results.html

3.

Revision history for this message
Giulio Fidente (gfidente) wrote :

3. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54f/751828/2/gate/tripleo-ci-centos-8-scenario001-standalone/54f184e/logs/undercloud/var/log/containers/mysql/mysqld.log

Revision history for this message
Giulio Fidente (gfidente) wrote :

In jobs which are green the mysql errors are logged only once [1] while in failed runs the errors appear in the mysql logs twice [2]

1. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_308/750307/1/gate/tripleo-ci-centos-8-scenario002-standalone/308e153/logs/undercloud/var/log/containers/mysql/mariadb.log

2. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54f/751828/2/gate/tripleo-ci-centos-8-scenario001-standalone/54f184e/logs/undercloud/var/log/containers/mysql/mysqld.log

Revision history for this message
Giulio Fidente (gfidente) wrote :

It looks like we're hitting OOM killer and losing ceph-osd and rbd at 22:38:06

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_54f/751828/2/gate/tripleo-ci-centos-8-scenario001-standalone/54f184e/logs/undercloud/var/log/extra/journal.txt

Revision history for this message
Giulio Fidente (gfidente) wrote :

Ultimately this was resolved by https://review.opendev.org/#/c/752187/ so I am marking this DUP of bug 1895288

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.