DBDeadlock for INSERT INTO resourcedeltas

Bug #1822382 reported by Michael Johnson on 2019-03-29
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Undecided
Unassigned

Bug Description

Recently we started seeing instances fail to become reachable in the Octavia tempest jobs. This is intermittent, but recurring. This may be related to other DBDeadlock bugs recently reported for quotas, but since the SQL is different here I am reporting it.

This is on Master/Train.

Summary of the error in q-svc:

Mar 29 20:04:12.816598 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters oslo_db.exception.DBDeadlock: (pymysql.err.InternalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: 'INSERT INTO resourcedeltas (resource, reservation_id, amount) VALUES (%(resource)s, %(reservation_id)s, %(amount)s)'] [parameters: {'reservation_id': '4f198b7d-ac31-42bb-98bd-686c830322ab', 'resource': 'port', 'amount': 1}] (Background on this error at: http://sqlalche.me/e/2j85)

Full traceback from q-svc for once occurrence:

Mar 29 20:04:12.790909 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters [req-56d6484e-b182-4dbd-8bb9-8db4ceb3c38a req-ddb65494-cdaf-4dec-ab19-84efbede0da7 admin admin] DBAPIError exception wrapped from (pymysql.err.InternalError) (1305, 'SAVEPOINT sa_savepoint_9 does not exist') [SQL: 'ROLLBACK TO SAVEPOINT sa_savepoint_9'] (Background on this error at: http://sqlalche.me/e/2j85): pymysql.err.InternalError: (1305, 'SAVEPOINT sa_savepoint_9 does not exist')
Mar 29 20:04:12.791441 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
Mar 29 20:04:12.791889 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
Mar 29 20:04:12.792380 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters context)
Mar 29 20:04:12.792860 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line 507, in do_execute
Mar 29 20:04:12.793296 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
Mar 29 20:04:12.793872 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 165, in execute
Mar 29 20:04:12.794320 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query)
Mar 29 20:04:12.794743 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 321, in _query
Mar 29 20:04:12.795219 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters conn.query(q)
Mar 29 20:04:12.795668 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 860, in query
Mar 29 20:04:12.796102 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered)
Mar 29 20:04:12.796505 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1061, in _read_query_result
Mar 29 20:04:12.796904 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters result.read()
Mar 29 20:04:12.797336 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1349, in read
Mar 29 20:04:12.797730 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet()
Mar 29 20:04:12.798022 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1018, in _read_packet
Mar 29 20:04:12.798305 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters packet.check_error()
Mar 29 20:04:12.798600 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 384, in check_error
Mar 29 20:04:12.798894 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data)
Mar 29 20:04:12.799224 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
Mar 29 20:04:12.799516 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters raise errorclass(errno, errval)
Mar 29 20:04:12.799802 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters pymysql.err.InternalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction')
Mar 29 20:04:12.800094 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
Mar 29 20:04:12.800509 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters The above exception was the direct cause of the following exception:
Mar 29 20:04:12.800944 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
Mar 29 20:04:12.801332 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
Mar 29 20:04:12.801636 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 2333, in _flush
Mar 29 20:04:12.801922 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters flush_context.execute()
Mar 29 20:04:12.802210 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
Mar 29 20:04:12.802555 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters rec.execute(self)
Mar 29 20:04:12.802851 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
Mar 29 20:04:12.803240 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters uow
Mar 29 20:04:12.803792 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
Mar 29 20:04:12.804108 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters mapper, table, insert)
Mar 29 20:04:12.804435 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py", line 830, in _emit_insert_statements
Mar 29 20:04:12.804743 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters execute(statement, multiparams)
Mar 29 20:04:12.805056 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 948, in execute
Mar 29 20:04:12.805337 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters return meth(self, multiparams, params)
Mar 29 20:04:12.805697 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
Mar 29 20:04:12.806036 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters return connection._execute_clauseelement(self, multiparams, params)
Mar 29 20:04:12.806449 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
Mar 29 20:04:12.806739 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters compiled_sql, distilled_params
Mar 29 20:04:12.807148 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
Mar 29 20:04:12.807480 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters context)
Mar 29 20:04:12.807773 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
Mar 29 20:04:12.808062 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters util.raise_from_cause(newraise, exc_info)
Mar 29 20:04:12.808388 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
Mar 29 20:04:12.808732 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters reraise(type(exception), exception, tb=exc_tb, cause=cause)
Mar 29 20:04:12.809104 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 186, in reraise
Mar 29 20:04:12.809390 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters raise value.with_traceback(tb)
Mar 29 20:04:12.809730 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
Mar 29 20:04:12.810057 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters context)
Mar 29 20:04:12.810334 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line 507, in do_execute
Mar 29 20:04:12.810632 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
Mar 29 20:04:12.811087 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 165, in execute
Mar 29 20:04:12.811477 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query)
Mar 29 20:04:12.811776 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 321, in _query
Mar 29 20:04:12.812094 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters conn.query(q)
Mar 29 20:04:12.812378 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 860, in query
Mar 29 20:04:12.812661 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered)
Mar 29 20:04:12.812937 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1061, in _read_query_result
Mar 29 20:04:12.813324 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters result.read()
Mar 29 20:04:12.813722 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1349, in read
Mar 29 20:04:12.814133 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet()
Mar 29 20:04:12.814666 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1018, in _read_packet
Mar 29 20:04:12.815045 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters packet.check_error()
Mar 29 20:04:12.815417 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 384, in check_error
Mar 29 20:04:12.815711 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data)
Mar 29 20:04:12.815990 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.5/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
Mar 29 20:04:12.816272 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters raise errorclass(errno, errval)
Mar 29 20:04:12.816598 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters oslo_db.exception.DBDeadlock: (pymysql.err.InternalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: 'INSERT INTO resourcedeltas (resource, reservation_id, amount) VALUES (%(resource)s, %(reservation_id)s, %(amount)s)'] [parameters: {'reservation_id': '4f198b7d-ac31-42bb-98bd-686c830322ab', 'resource': 'port', 'amount': 1}] (Background on this error at: http://sqlalche.me/e/2j85)

Michael Johnson (johnsom) wrote :

Looking at this deeper, it appears neutron did properly retry this DB action and the instance connection issue may be unrelated. Marking this invalid.

Changed in neutron:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers