In testrun https://solutions.qa.canonical.com/testruns/testRun/5a882276-bbb0-4be1-91a7-ea61bf14d61f, which is yoga focal, all the selected telemetry tests fail with:
```
telemetry_tempest_plugin.aodh.api.test_alarming_api.TelemetryAlarmingAPITest.test_set_get_alarm_state
Traceback (most recent call last):
File "/home/ubuntu/snap/fcbtest/34/.rally/verification/verifier-fd0c7b1f-04af-49d5-b6e4-0ca11c1e7ce0/repo/tempest/test.py", line 168, in setUpClass
raise value.with_traceback(trace)
File "/home/ubuntu/snap/fcbtest/34/.rally/verification/verifier-fd0c7b1f-04af-49d5-b6e4-0ca11c1e7ce0/repo/tempest/test.py", line 161, in setUpClass
cls.resource_setup()
File "/snap/fcbtest/34/lib/python3.10/site-packages/telemetry_tempest_plugin/aodh/api/test_alarming_api.py", line 38, in resource_setup
cls.create_alarm(event_rule=cls.rule)
File "/snap/fcbtest/34/lib/python3.10/site-packages/telemetry_tempest_plugin/aodh/api/base.py", line 47, in create_alarm
body = cls.alarming_client.create_alarm(
File "/snap/fcbtest/34/lib/python3.10/site-packages/telemetry_tempest_plugin/aodh/service/client.py", line 78, in create_alarm
resp, body = self.post(uri, body)
File "/home/ubuntu/snap/fcbtest/34/.rally/verification/verifier-fd0c7b1f-04af-49d5-b6e4-0ca11c1e7ce0/repo/tempest/lib/common/rest_client.py", line 299, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File "/home/ubuntu/snap/fcbtest/34/.rally/verification/verifier-fd0c7b1f-04af-49d5-b6e4-0ca11c1e7ce0/repo/tempest/lib/common/rest_client.py", line 720, in request
self._error_checker(resp, resp_body)
File "/home/ubuntu/snap/fcbtest/34/.rally/verification/verifier-fd0c7b1f-04af-49d5-b6e4-0ca11c1e7ce0/repo/tempest/lib/common/rest_client.py", line 810, in _error_checker
raise exceptions.UnexpectedContentType(str(resp.status),
tempest.lib.exceptions.UnexpectedContentType: Unexpected content type provided
Details: 400
```
In the aodh unit logs I see a lot of connection refused messages from rabbit:
```
aodh/aodh-listener.log:2022-09-13 11:44:45.240 15676 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 111] Connection refused (retrying in 31.0 seconds): ConnectionRefusedError: [Errno 111] Connection refused
aodh/aodh-listener.log:2022-09-13 11:45:16.266 15676 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 111] Connection refused (retrying in 31.0 seconds): ConnectionRefusedError: [Errno 111] Connection refused
aodh/aodh-listener.log:2022-09-13 11:45:47.294 15676 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 111] Connection refused (retrying in 31.0 seconds): ConnectionRefusedError: [Errno 111] Connection refused
aodh/aodh-listener.log:2022-09-13 11:46:18.324 15676 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 111] Connection refused (retrying in 31.0 seconds): ConnectionRefusedError: [Errno 111] Connection refused
aodh/aodh-listener.log:2022-09-13 11:46:49.368 15676 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 111] Connection refused (retrying in 31.0 seconds): ConnectionRefusedError: [Errno 111] Connection refused
```
As well as MySQL related errors:
```
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines self.dialect.do_execute(
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 148, in execute
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 310, in _query
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines conn.query(q)
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 548, in query
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 775, in _read_query_result
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines result.read()
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1156, in read
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 692, in _read_packet
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 748, in _read_bytes
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines raise err.OperationalError(
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines The above exception was the direct cause of the following exception:
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines connection.scalar(select(1))
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1171, in scalar
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines return self.execute(object_, *multiparams, **params).scalar()
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1289, in execute
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines ret = self._execute_context(
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines self._handle_dbapi_exception(
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2024, in _handle_dbapi_exception
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines util.raise_(newraise, with_traceback=exc_info[2], from_=e)
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines raise exception
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines self.dialect.do_execute(
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 148, in execute
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 310, in _query
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines conn.query(q)
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 548, in query
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 775, in _read_query_result
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines result.read()
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1156, in read
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 692, in _read_packet
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 748, in _read_bytes
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines raise err.OperationalError(
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines [SQL: SELECT 1]
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines (Background on this error at: https://sqlalche.me/e/14/e3q8)
aodh/aodh-evaluator.log:2022-09-13 12:00:02.413 100613 ERROR oslo_db.sqlalchemy.engines
```
and
```
aodh/aodh-evaluator.log:2022-09-13 11:45:34.790 15893 ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
aodh/aodh-evaluator.log:2022-09-13 11:45:34.790 15893 ERROR aodh.evaluator self.dialect.do_execute(
aodh/aodh-evaluator.log:2022-09-13 11:45:34.790 15893 ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute
aodh/aodh-evaluator.log:2022-09-13 11:45:34.790 15893 ERROR aodh.evaluator cursor.execute(statement, parameters)
aodh/aodh-evaluator.log:2022-09-13 11:45:34.790 15893 ERROR aodh.evaluator sqlite3.OperationalError: no such table: alarm
```
Crashdumps and testrun configs can be found here:
https://oil-jenkins.canonical.com/artifacts/5a882276-bbb0-4be1-91a7-ea61bf14d61f/index.html
Marking as invalid due to inactivity