Multiple threads start using persistence sqlalchemy fails

[Ann Taraday]

When we call backend function https://github.com/openstack/taskflow/blob/master/taskflow/persistence/backends/__init__.py#L81 in multiple threads simultaneously - it resulted in errors like:

octavia-worker[10454]: ERROR cotyledon._utils [-] Unhandled exception: taskflow.exceptions.StorageFailure: Failed upgrading database version
octavia-worker[10454]: ERROR cotyledon._utils Traceback (most recent call last):
octavia-worker[10454]: ERROR cotyledon._utils File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
octavia-worker[10454]: ERROR cotyledon._utils cursor, statement, parameters, context
octavia-worker[10454]: ERROR cotyledon._utils File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 581, in do_execute
octavia-worker[10454]: ERROR cotyledon._utils cursor.execute(statement, parameters)
octavia-worker[10454]: ERROR cotyledon._utils File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 170, in execute
octavia-worker[10454]: ERROR cotyledon._utils result = self._query(query)
octavia-worker[10454]: ERROR cotyledon._utils File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 328, in _query
octavia-worker[10454]: ERROR cotyledon._utils conn.query(q)
octavia-worker[10454]: ERROR cotyledon._utils File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 517, in query
octavia-worker[10454]: ERROR cotyledon._utils self._affected_rows = self._read_query_result(unbuffered=unbuffered)
octavia-worker[10454]: ERROR cotyledon._utils File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 732, in _read_query_result
octavia-worker[10454]: ERROR cotyledon._utils result.read()
octavia-worker[10454]: ERROR cotyledon._utils File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 1075, in read
octavia-worker[10454]: ERROR cotyledon._utils first_packet = self.connection._read_packet()
octavia-worker[10454]: ERROR cotyledon._utils File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 684, in _read_packet
octavia-worker[10454]: ERROR cotyledon._utils packet.check_error()
octavia-worker[10454]: ERROR cotyledon._utils File "/usr/local/lib/python3.6/dist-packages/pymysql/protocol.py", line 220, in check_error
octavia-worker[10454]: ERROR cotyledon._utils err.raise_mysql_exception(self._data)
octavia-worker[10454]: ERROR cotyledon._utils File "/usr/local/lib/python3.6/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
octavia-worker[10454]: ERROR cotyledon._utils raise errorclass(errno, errval)
octavia-worker[10454]: ERROR cotyledon._utils pymysql.err.InternalError: (1050, "Table 'logbooks' already exists")
This is not blocking work of service as sooner or later all migrations applied and service start working, but for multi thread usage - phase of initializing of db and applying migrations should be done in one thread and before service started.

[Sowmya Divvi (sowmya)]

Hello Ann Taraday,

Could you please provide more information on the Bug to reproduce the issue.

Best Regards,
Sowmya Divvi

[Ann Taraday (akamyshnikova)]

This issue appeared during implementation of taskflow jobboard in Octavia [1] - it was workarounded by adding separate initilized method - run migrations before server start (patch set 82) [2][3].
This can be reproduced by running several conductors which stated initialization.

[1] - https://review.opendev.org/#/c/647406
[2] - https://review.opendev.org/#/c/647406/106/octavia/controller/worker/v2/taskflow_jobboard_driver.py@46
[3] - https://review.opendev.org/#/c/647406/106/octavia/db/migration/cli.py@93
[4] - https://review.opendev.org/#/c/647406/106/octavia/controller/queue/v2/consumer.py@51