OpenStack AODH Charm

  1. Bug #1904599
  2. Comment #10

Comment 10 for bug 1904599

Revision history for this message
Moises Emilio Benzan Mora (moisesbenzan) wrote :

Triggered this bug once more on a Bionic deployment.

On keystone[1], we see `No such table: credential` error. The other two units (keystone[0] and keystone[2]) boot up correctly and start serving requests for a while:

```
2021-10-02 05:22:54.767 23885 ERROR keystone Traceback (most recent call last):
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
2021-10-02 05:22:54.767 23885 ERROR keystone cursor, statement, parameters, context
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 581, in do_execute
2021-10-02 05:22:54.767 23885 ERROR keystone cursor.execute(statement, parameters)
2021-10-02 05:22:54.767 23885 ERROR keystone sqlite3.OperationalError: no such table: credential
2021-10-02 05:22:54.767 23885 ERROR keystone
2021-10-02 05:22:54.767 23885 ERROR keystone The above exception was the direct cause of the following exception:
2021-10-02 05:22:54.767 23885 ERROR keystone
2021-10-02 05:22:54.767 23885 ERROR keystone Traceback (most recent call last):
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/bin/keystone-manage", line 10, in <module>
2021-10-02 05:22:54.767 23885 ERROR keystone sys.exit(main())
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/keystone/cmd/manage.py", line 41, in main
2021-10-02 05:22:54.767 23885 ERROR keystone cli.main(argv=sys.argv, developer_config_file=developer_config)
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/keystone/cmd/cli.py", line 1379, in main
2021-10-02 05:22:54.767 23885 ERROR keystone CONF.command.cmd_class.main()
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/keystone/cmd/cli.py", line 780, in main
2021-10-02 05:22:54.767 23885 ERROR keystone klass.migrate_credentials()
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/keystone/cmd/cli.py", line 753, in migrate_credentials
2021-10-02 05:22:54.767 23885 ERROR keystone driver_hints.Hints()
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/keystone/common/driver_hints.py", line 42, in wrapper
2021-10-02 05:22:54.767 23885 ERROR keystone return f(self, hints, *args, **kwargs)
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/keystone/credential/backends/sql.py", line 69, in list_credentials
2021-10-02 05:22:54.767 23885 ERROR keystone return [s.to_dict() for s in credentials]
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3367, in __iter__
2021-10-02 05:22:54.767 23885 ERROR keystone return self._execute_and_instances(context)
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3392, in _execute_and_instances
2021-10-02 05:22:54.767 23885 ERROR keystone result = conn.execute(querycontext.statement, self._params)
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
2021-10-02 05:22:54.767 23885 ERROR keystone return meth(self, multiparams, params)
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
2021-10-02 05:22:54.767 23885 ERROR keystone return connection._execute_clauseelement(self, multiparams, params)
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
2021-10-02 05:22:54.767 23885 ERROR keystone distilled_params,
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context
2021-10-02 05:22:54.767 23885 ERROR keystone e, statement, parameters, cursor, context
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1474, in _handle_dbapi_exception
2021-10-02 05:22:54.767 23885 ERROR keystone util.raise_from_cause(newraise, exc_info)
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
2021-10-02 05:22:54.767 23885 ERROR keystone reraise(type(exception), exception, tb=exc_tb, cause=cause)
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
2021-10-02 05:22:54.767 23885 ERROR keystone raise value.with_traceback(tb)
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
2021-10-02 05:22:54.767 23885 ERROR keystone cursor, statement, parameters, context
2021-10-02 05:22:54.767 23885 ERROR keystone File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 581, in do_execute
2021-10-02 05:22:54.767 23885 ERROR keystone cursor.execute(statement, parameters)
2021-10-02 05:22:54.767 23885 ERROR keystone oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) no such table: credential
2021-10-02 05:22:54.767 23885 ERROR keystone [SQL: SELECT credential.encrypted_blob AS credential_encrypted_blob, credential.id AS credential_id, credential.user_id AS credential_user_id, credential.project_id AS credential_project_id, credential.type AS credential_type, credential.key_hash AS credential_key_hash, credential.extra AS credential_extra
2021-10-02 05:22:54.767 23885 ERROR keystone FROM credential]
2021-10-02 05:22:54.767 23885 ERROR keystone (Background on this error at: http://sqlalche.me/e/e3q8)
2021-10-02 05:22:54.767 23885 ERROR keystone
```

Link to run: https://solutions.qa.canonical.com/testruns/testRun/2fcebd5d-3b92-4a4c-8e6b-dedc606430be

Link to jenkins job:https://oil-jenkins.canonical.com/job/fce_build/12572//consoleFull

Link to artifacts: https://oil-jenkins.canonical.com/artifacts/2fcebd5d-3b92-4a4c-8e6b-dedc606430be/index.html

All occurrences can be found: https://solutions.qa.canonical.com/bugs/bugs/bug/1904599