OpenStack Compute (nova)

nova conductor update database get deadlock

Bug #1791204 reported by Henry
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Incomplete
Undecided
Henry

Bug Description

In order to test HA for controller node: shudown one controller node and then vip switch to another node. Creating instance at this situation got the Traceback in conductor:

2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server [req-503920a4-f2d8-47a8-a8d2-f9e4297be0ed a5be69ec54db4db2bf5aedc997da6371 0cdf4389dd09480098f4874e31585882 - - -] Exception during message handling
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 930, in schedule_and_build_instances
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server instance.flavor, instance.uuid, block_device_mapping)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 799, in _create_block_device_mapping
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server bdm.update_or_create()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return fn(self, *args, **kwargs)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 166, in update_or_create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self._create(self._context, update_or_create=True)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 145, in _create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server context, updates, legacy=False)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1295, in block_device_mapping_update_or_create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return IMPL.block_device_mapping_update_or_create(context, values, legacy)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 251, in wrapped
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return f(context, *args, **kwargs)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.gen.next()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 945, in _transaction_scope
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server yield resource
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.gen.next()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 562, in _session
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self._end_session_transaction(self.session)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 583, in _end_session_transaction
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server session.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 813, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.transaction.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 396, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server t[1].commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1574, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self._do_commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1605, in _do_commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.connection._commit_impl()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 690, in _commit_impl
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self._handle_dbapi_exception(e, None, None, None, None)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server util.raise_from_cause(newraise, exc_info)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server reraise(type(exception), exception, tb=exc_tb)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 688, in _commit_impl
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.engine.dialect.do_commit(self.connection)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2514, in do_commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server dbapi_connection.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self._read_ok_packet()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 752, in _read_ok_packet
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server pkt = self._read_packet()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server packet.check_error()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server err.raise_mysql_exception(self._data)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server raise errorclass(errno, errval)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction')

Always got that in _create_block_device_mapping.

openstack version：ocata
nova --version
10.2.0

Tags: conductor
Henry (guanzy)
Changed in nova:
assignee: nobody → Henry (guanzy)
Takashi Natsume (natsume-takashi)
tags: added: conductor
Revision history for this message
Matt Riedemann (mriedem) wrote :

What does the server create request look like? Is there a block_device_mapping_v2 parameter? Does the flavor have swap specified?

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

Marking the bug Incomplete. Please but the bug back to New state after you answered Matt's question above.

Changed in nova:
status: New → Incomplete
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.