nova conductor update database get deadlock

Bug #1791204 reported by Henry on 2018-09-07
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Henry

Bug Description

In order to test HA for controller node: shudown one controller node and then vip switch to another node. Creating instance at this situation got the Traceback in conductor:

2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server [req-503920a4-f2d8-47a8-a8d2-f9e4297be0ed a5be69ec54db4db2bf5aedc997da6371 0cdf4389dd09480098f4874e31585882 - - -] Exception during message handling
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 930, in schedule_and_build_instances
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server instance.flavor, instance.uuid, block_device_mapping)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 799, in _create_block_device_mapping
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server bdm.update_or_create()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return fn(self, *args, **kwargs)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 166, in update_or_create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self._create(self._context, update_or_create=True)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 145, in _create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server context, updates, legacy=False)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1295, in block_device_mapping_update_or_create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return IMPL.block_device_mapping_update_or_create(context, values, legacy)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 251, in wrapped
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return f(context, *args, **kwargs)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.gen.next()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 945, in _transaction_scope
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server yield resource
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.gen.next()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 562, in _session
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self._end_session_transaction(self.session)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 583, in _end_session_transaction
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server session.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 813, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.transaction.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 396, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server t[1].commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1574, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self._do_commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1605, in _do_commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.connection._commit_impl()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 690, in _commit_impl
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self._handle_dbapi_exception(e, None, None, None, None)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server util.raise_from_cause(newraise, exc_info)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server reraise(type(exception), exception, tb=exc_tb)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 688, in _commit_impl
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.engine.dialect.do_commit(self.connection)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2514, in do_commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server dbapi_connection.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self._read_ok_packet()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 752, in _read_ok_packet
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server pkt = self._read_packet()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server packet.check_error()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server err.raise_mysql_exception(self._data)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server raise errorclass(errno, errval)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction')

Always got that in _create_block_device_mapping.

openstack version:ocata
nova --version
10.2.0

Henry (guanzy) on 2018-09-07
Changed in nova:
assignee: nobody → Henry (guanzy)
tags: added: conductor
Matt Riedemann (mriedem) wrote :

What does the server create request look like? Is there a block_device_mapping_v2 parameter? Does the flavor have swap specified?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers