As Goutham just pointed out to me, we just had an instance of what is almost certainly the same issue at root. The report here is for an insert but here's a failure just like it except that it's a deadlock on deletion of quota usages: From the manila-share log: Mar 21 18:08:46.263907 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: DEBUG manila.quota [None req-91556767-fbbd-43ba-81f2-5a49c7eb7f86 None None] Created reservations ['e2ffd686-3ec9-4129-bf46-909eca546638'] {{(pid=16959) reserve /opt/stack/new/manila/manila/quota.py:916}} Mar 21 18:08:46.278873 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: DEBUG eventlet.wsgi.server [-] (16959) accepted ('::ffff:158.69.74.30', 57108, 0, 0) {{(pid=16959) server /usr/local/lib/python3.6/dist-packages/eventlet/wsgi.py:956}} Mar 21 18:08:46.279440 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: DEBUG eventlet.wsgi.server [-] (16959) accepted ('::ffff:158.69.74.30', 57110, 0, 0) {{(pid=16959) server /usr/local/lib/python3.6/dist-packages/eventlet/wsgi.py:956}} Mar 21 18:08:46.295613 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: INFO manila.api.openstack.wsgi [None req-fa8107c9-1bd2-423c-ba8d-e2a6e93a5620 None None] DELETE https://158.69.74.30:8786/v1/081d92bd03fb49288548b31f48c36340/share-networks/7c850b58-8ce8-433e-8f53-fba572dc4644 Mar 21 18:08:46.296256 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: DEBUG manila.api.openstack.wsgi [None req-fa8107c9-1bd2-423c-ba8d-e2a6e93a5620 None None] Empty body provided in request {{(pid=16959) get_body /opt/stack/new/manila/manila/api/openstack/wsgi.py:653}} Mar 21 18:08:46.296789 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: DEBUG manila.api.openstack.wsgi [None req-fa8107c9-1bd2-423c-ba8d-e2a6e93a5620 None None] Calling method 'ShareNetworkController.delete' {{(pid=16959) _process_stack /opt/stack/new/manila/manila/api/openstack/wsgi.py:799}} Mar 21 18:08:46.347740 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault [None req-3d6475a9-72fb-42c5-9f66-8deaa54d4e49 None None] Caught error: (pymysql.err.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: 'UPDATE quota_usages SET updated_at=quota_usages.updated_at, deleted_at=%(deleted_at)s, deleted=quota_usages.id WHERE quota_usages.deleted = %(deleted_1)s AND quota_usages.share_type_id = %(share_type_id_1)s'] [parameters: {'deleted_at': datetime.datetime(2019, 3, 21, 18, 8, 46, 267266), 'deleted_1': 0, 'share_type_id_1': 'd01f19cc-c877-4404-b34f-52fd743e3c5e'}] (Background on this error at: http://sqlalche.me/e/e3q8): oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: 'UPDATE quota_usages SET updated_at=quota_usages.updated_at, deleted_at=%(deleted_at)s, deleted=quota_usages.id WHERE quota_usages.deleted = %(deleted_1)s AND quota_usages.share_type_id = %(share_type_id_1)s'] [parameters: {'deleted_at': datetime.datetime(2019, 3, 21, 18, 8, 46, 267266), 'deleted_1': 0, 'share_type_id_1': 'd01f19cc-c877-4404-b34f-52fd743e3c5e'}] (Background on this error at: http://sqlalche.me/e/e3q8) Mar 21 18:08:46.347740 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault Traceback (most recent call last): Mar 21 18:08:46.347740 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context Mar 21 18:08:46.347740 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault cursor, statement, parameters, context Mar 21 18:08:46.347740 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 536, in do_execute Mar 21 18:08:46.347740 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault cursor.execute(statement, parameters) Mar 21 18:08:46.347740 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 170, in execute Mar 21 18:08:46.347740 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault result = self._query(query) Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 328, in _query Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault conn.query(q) Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 517, in query Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault self._affected_rows = self._read_query_result(unbuffered=unbuffered) Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 732, in _read_query_result Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault result.read() Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 1075, in read Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault first_packet = self.connection._read_packet() Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 684, in _read_packet Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault packet.check_error() Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/pymysql/protocol.py", line 220, in check_error Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault err.raise_mysql_exception(self._data) Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault raise errorclass(errno, errval) Mar 21 18:08:46.348356 ubuntu-bionic-ovh-bhs1-0004095398 manila-api[16852]: ERROR manila.api.middleware.fault pymysql.err.OperationalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction')