Bug #1911020 “Ugrades ussuri jobs fail in CI” : Bugs : tripleo

Revision history for this message

Sagi (Sergey) Shnaidman (sshnaidm) wrote on 2021-01-11:

#1

maybe related: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/769510

Revision history for this message

wes hayutin (weshayutin) wrote on 2021-01-11:

#2

Download full text (11.8 KiB)

This looks like the mariadb connection is fubard.

https://6f2db3aca8a21b7bf2ca-43939889d4a60991ed49de41534ef9b0.ssl.cf5.rackcdn.com/769595/1/gate/tripleo-ci-centos-8-undercloud-upgrade-ussuri/1323868/logs/undercloud/var/log/extra/big-errors.txt.txt

CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2021-01-09 09:28:29.503 ERROR /var/log/containers/keystone/keystone.log.1: 153 ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')

2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines [req-d76944e1-6b0a-4208-8db3-327fbb60bb2a f297f11b2175441caa3f28589a3d9c36 98d7ebc325ed40ae878d07cb59a9ddaf - default default] Database connection was found disconnected; reconnecting: oslo_db.exception.DBConnectionError: (pymysql.err.InternalError) (1927, 'Connection was killed')
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines cursor, statement, parameters, context
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 165, in execute
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _query
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines conn.query(q)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 860, in query
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1061, in _read_query_result
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines result.read()
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db....

This looks like the mariadb connection is fubard.

https://6f2db3aca8a21b7bf2ca-43939889d4a60991ed49de41534ef9b0.ssl.cf5.rackcdn.com/769595/1/gate/tripleo-ci-centos-8-undercloud-upgrade-ussuri/1323868/logs/undercloud/var/log/extra/big-errors.txt.txt

CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2021-01-09 09:28:29.503 ERROR /var/log/containers/keystone/keystone.log.1: 153 ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')

2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines [req-d76944e1-6b0a-4208-8db3-327fbb60bb2a f297f11b2175441caa3f28589a3d9c36 98d7ebc325ed40ae878d07cb59a9ddaf - default default] Database connection was found disconnected; reconnecting: oslo_db.exception.DBConnectionError: (pymysql.err.InternalError) (1927, 'Connection was killed')
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     cursor, statement, parameters, context
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     cursor.execute(statement, parameters)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 165, in execute
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     result = self._query(query)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _query
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     conn.query(q)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 860, in query
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1061, in _read_query_result
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     result.read()
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1349, in read
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     first_packet = self.connection._read_packet()
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1018, in _read_packet
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     packet.check_error()
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 384, in check_error
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     err.raise_mysql_exception(self._data)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     raise errorclass(errno, errval)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines pymysql.err.InternalError: (1927, 'Connection was killed')
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines 
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines The above exception was the direct cause of the following exception:
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines 
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     connection.scalar(select([1]))
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 920, in scalar
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     return self.execute(object_, *multiparams, **params).scalar()
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     return meth(self, multiparams, params)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     return connection._execute_clauseelement(self, multiparams, params)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     distilled_params,
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     e, statement, parameters, cursor, context
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1464, in _handle_dbapi_exception
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     util.raise_from_cause(newraise, exc_info)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     raise value.with_traceback(tb)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     cursor, statement, parameters, context
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     cursor.execute(statement, parameters)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 165, in execute
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     result = self._query(query)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _query
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     conn.query(q)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 860, in query
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1061, in _read_query_result
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     result.read()
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1349, in read
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     first_packet = self.connection._read_packet()
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1018, in _read_packet
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     packet.check_error()
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 384, in check_error
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     err.raise_mysql_exception(self._data)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines     raise errorclass(errno, errval)
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines oslo_db.exception.DBConnectionError: (pymysql.err.InternalError) (1927, 'Connection was killed')
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines [SQL: SELECT 1]
2021-01-09 09:09:18.196 ERROR /var/log/containers/ironic/app.log: 14 ERROR oslo_db.sqlalchemy.engines (Background on this error at: http://sqlalche.me/e/2j85)

Revision history for this message

wes hayutin (weshayutin) wrote on 2021-01-11:

#3

IMHO.. this should have failed in the upgrade, and not completed and moved to tempest.

Changed in tripleo:
milestone:	none → wallaby-2

Revision history for this message

wes hayutin (weshayutin) wrote on 2021-01-11:

#4

in the overcloud upgrade.. it looks like the mariadb container is caught w/ a health check

https://4c7502961c4643b6854e-e3236fcfb41e99e283b71fa476352983.ssl.cf5.rackcdn.com/769595/1/gate/tripleo-ci-centos-8-scenario000-multinode-oooq-container-upgrades-ussuri/203dc2b/logs/subnode-1/var/log/extra/failed_services.txt

Jan 09 09:55:17 centos-8-ovh-bhs1-0022460204 podman[180634]: 2021-01-09 09:55:17.643981633 +0000 UTC m=+0.527802145 container exec cc49c932d9f609a97e0992f586158db5270cf333e2c0488d35cc981f972448fe (image=192.168.24.1:8787/tripleou/centos-binary-mariadb:57e21c8afa1aacd4f03195f409ff827b, name=clustercheck)
Jan 09 09:55:17 centos-8-ovh-bhs1-0022460204 podman[180634]: unhealthy

Revision history for this message

Alex Schultz (alex-schultz) wrote on 2021-01-12:

#5

https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770188 tempest was switched to use an rpm and it doesn't work

Revision history for this message

Marios Andreou (marios-b) wrote on 2021-01-14:

#6

the revert has been blocked by workflow -1 at https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770188

apparently arx is trying to fix this in another way (I was pointed at https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770357 but there may be others?)

@Arx please update this with the fixes you have proposed. It is still blocking ussuri. IMO we should have let the revert merge and then fix it as this blocks gates

Revision history for this message

Marios Andreou (marios-b) wrote on 2021-01-14:

#7

09:39 < chandankumar> marios: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770357 was one of the fix

09:52 < chandankumar> marios: https://review.opendev.org/c/openstack/tripleo-quickstart/+/770359

Revision history for this message

Arx Cruz (arxcruz) wrote on 2021-01-14:

#8

Hello,

The problem was mixed variables between validate-tempest and os_tempest.
Since we were running part of the jobs with os_tempest and another part (old releases) with validate-tempest, we had run_tempest and use_os_tempest variables.

When we move everything to run with os_tempest, by mistake, I assume all the jobs were running tempest, and set the default use_os_tempest to True. This make update/upgrade jobs run tempest and fail (which is something we should investigate later).

The patch https://review.opendev.org/c/openstack/tripleo-quickstart/+/770359 fix this problem, replace run_tempest with use_os_tempest, and set it to false on all the featuresets that had it set to false.

A followup patch will remove the remain variables and explicitly set use_os_tempest to true on remain featuresets, for the sake of documentation.

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2021-01-18:

#9

the patches are merged, was it fixed yet?

Changed in tripleo:
status:	Triaged → Fix Committed

Revision history for this message

wes hayutin (weshayutin) wrote on 2021-01-18:

#10

http://dashboard-ci.tripleo.org/d/Z4vLSmOGk/cockpit?orgId=1&var-launchpad_tags=alert&var-releases=master&var-promotion_names=current-tripleo&var-promotion_names=current-tripleo-rdo&var-influxdb_filter=job_name%7C%3D~%7C%2Ftripleo-ci.*-up.*%2F&from=now-2d&to=now

marking this fixed released

Changed in tripleo:
status:	Fix Committed → Fix Released

tripleo

Ugrades ussuri jobs fail in CI

Bug Description

Duplicates of this bug

Other bug subscribers

Remote bug watches