Comment 4 for bug 1909385

Were seeing something similar with SQA on wallaby in this testrun:https://solutions.qa.canonical.com/testruns/testRun/2125bbad-971b-4bd4-9521-5f28dd36d853

Only 1 out of 3 aodh units is stuck waiting.

In the aodh-evaluator logs we see:
------------------------------------------------------------------
 Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
     self.dialect.do_execute(
   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 609, in do_execute
     cursor.execute(statement, parameters)
 sqlite3.OperationalError: no such table: alarm

 The above exception was the direct cause of the following exception:

 Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/aodh/evaluator/__init__.py", line 248, in _evaluate_assigned_alarms
     alarms = self._assigned_alarms()
   File "/usr/lib/python3/dist-packages/aodh/evaluator/__init__.py", line 292, in _assigned_alarms
     selected = self.storage_conn.get_alarms(
   File "/usr/lib/python3/dist-packages/aodh/storage/impl_sqlalchemy.py", line 250, in get_alarms
     alarms = self._retrieve_alarms(query)
   File "/usr/lib/python3/dist-packages/aodh/storage/impl_sqlalchemy.py", line 205, in _retrieve_alarms
     return [self._row_to_alarm_model(x) for x in query.all()]
   File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3373, in all
     return list(self)
   File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
     return self._execute_and_instances(context)
   File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
     result = conn.execute(querycontext.statement, self._params)
   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1011, in execute
     return meth(self, multiparams, params)
   File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
     return connection._execute_clauseelement(self, multiparams, params)
   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
     ret = self._execute_context(
   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
     self._handle_dbapi_exception(
   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception
     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
   File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_
     raise exception
   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
     self.dialect.do_execute(
   File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 609, in do_execute
     cursor.execute(statement, parameters)
 oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) no such table: alarm
 [SQL: SELECT alarm.alarm_id AS alarm_alarm_id, alarm.enabled AS alarm_enabled, alarm.name AS alarm_name, alarm.type AS alarm_type, alarm.severity AS alarm_severity, alarm.description AS alarm_description, alarm.timestamp AS alarm_timestamp, alarm.user_id AS alarm_user_id, alarm.project_id AS alarm_project_id, alarm.state AS alarm_state, alarm.state_reason AS alarm_state_reason, alarm.state_timestamp AS alarm_state_timestamp, alarm.ok_actions AS alarm_ok_actions, alarm.alarm_actions AS alarm_alarm_actions, alarm.insufficient_data_actions AS alarm_insufficient_data_actions, alarm.repeat_actions AS alarm_repeat_actions, alarm.rule AS alarm_rule, alarm.time_constraints AS alarm_time_constraints, alarm.evaluate_timestamp AS alarm_evaluate_timestamp
 FROM alarm
 WHERE alarm.type != ? AND alarm.evaluate_timestamp < ? AND alarm.enabled = 1 ORDER BY alarm.timestamp DESC]
 [parameters: ('event', '2022-03-29 20:56:30.124280')]
 (Background on this error at: http://sqlalche.me/e/13/e3q8)
------------------------------------------------------------------

Link to crashdump:
https://oil-jenkins.canonical.com/artifacts/2125bbad-971b-4bd4-9521-5f28dd36d853/index.html