DBDeadlock inserting into instance_extra

[Matt Riedemann]

http://logs.openstack.org/78/193278/15/check/gate-tempest-dsvm-full/7e16644/logs/screen-n-api.txt.gz?level=TRACE#_2015-07-31_01_52_20_792

2015-07-31 01:52:20.792 ERROR nova.api.openstack [req-ea281178-5948-47a1-815d-e97f12b2412b tempest-FloatingIPsTestJSON-1949129213 tempest-FloatingIPsTestJSON-96720078] Caught error: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'INSERT INTO instance_extra (created_at, updated_at, deleted_at, deleted, instance_uuid, numa_topology, pci_requests, flavor, vcpu_model) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (datetime.datetime(2015, 7, 31, 1, 52, 20, 772556), None, None, 0, '645dcef8-0852-40b5-ac1f-e422b7909e90', None, '[]', '{"new": null, "old": null, "cur": {"nova_object.version": "1.1", "nova_object.name": "Flavor", "nova_object.data": {"disabled": false, "root_gb": 0, "name": "m1.nano", "flavorid": "42", "deleted": false, "created_at": "2015-07-31T01:50:53Z", "ephemeral_gb": 0, "updated_at": null, "memory_mb": 64, "vcpus": 1, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "is_public": true, "deleted_at": null, "vcpu_weight": 0, "id": 6}, "nova_object.namespace": "nova"}}', None)]
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack Traceback (most recent call last):
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/__init__.py", line 128, in __call__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack return req.get_response(self.application)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack application, catch_exc_info=False)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack return resp(environ, start_response)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 434, in __call__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack response = req.get_response(self._app)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack application, catch_exc_info=False)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack return resp(environ, start_response)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack return resp(environ, start_response)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack response = self.app(environ, start_response)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack return resp(environ, start_response)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 756, in __call__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack content_type, body, accept)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 821, in _process_stack
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 911, in dispatch
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack return method(req=request, **action_args)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 643, in create
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack self._handle_create_exception(*sys.exc_info())
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 472, in _handle_create_exception
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack six.reraise(*exc_info)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 628, in create
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack check_server_group_quota=check_server_group_quota)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/hooks.py", line 149, in inner
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack rv = f(*args, **kwargs)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/compute/api.py", line 1488, in create
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack check_server_group_quota=check_server_group_quota)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/compute/api.py", line 1121, in _create_instance
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack instance_group, check_server_group_quota)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/compute/api.py", line 976, in _provision_instances
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack quotas.rollback()
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/compute/api.py", line 939, in _provision_instances
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack num_instances, i, shutdown_terminate)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/compute/api.py", line 1388, in create_db_entry_for_new_instance
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack instance.create()
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/objects/base.py", line 116, in wrapper
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack return fn(self, *args, **kwargs)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/objects/instance.py", line 609, in create
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack db_inst = db.instance_create(self._context, updates)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/db/api.py", line 633, in instance_create
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack return IMPL.instance_create(context, values)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 229, in wrapper
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack return f(*args, **kwargs)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1652, in instance_create
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack session.add(instance_ref)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 490, in __exit__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack self.rollback()
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack compat.reraise(exc_type, exc_value, exc_tb)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 487, in __exit__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack self.commit()
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 392, in commit
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack self._prepare_impl()
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack self.session.flush()
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2004, in flush
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack self._flush(objects)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2122, in _flush
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack transaction.rollback(_capture_exception=True)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack compat.reraise(exc_type, exc_value, exc_tb)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2086, in _flush
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack flush_context.execute()
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack rec.execute(self)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack uow
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack mapper, table, insert)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 781, in _emit_insert_statements
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack execute(statement, params)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack return meth(self, multiparams, params)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack return connection._execute_clauseelement(self, multiparams, params)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack compiled_sql, distilled_params
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack context)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack util.raise_from_cause(newraise, exc_info)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack reraise(type(exception), exception, tb=exc_tb)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack context)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack cursor.execute(statement, parameters)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 134, in execute
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack result = self._query(query)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 282, in _query
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack conn.query(q)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 768, in query
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 929, in _read_query_result
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack result.read()
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1125, in read
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack first_packet = self.connection._read_packet()
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 893, in _read_packet
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack packet.check_error()
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 369, in check_error
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack err.raise_mysql_exception(self._data)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack _check_mysql_exception(errinfo)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 115, in _check_mysql_exception
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack raise InternalError(errno, errorvalue)
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'INSERT INTO instance_extra (created_at, updated_at, deleted_at, deleted, instance_uuid, numa_topology, pci_requests, flavor, vcpu_model) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (datetime.datetime(2015, 7, 31, 1, 52, 20, 772556), None, None, 0, '645dcef8-0852-40b5-ac1f-e422b7909e90', None, '[]', '{"new": null, "old": null, "cur": {"nova_object.version": "1.1", "nova_object.name": "Flavor", "nova_object.data": {"disabled": false, "root_gb": 0, "name": "m1.nano", "flavorid": "42", "deleted": false, "created_at": "2015-07-31T01:50:53Z", "ephemeral_gb": 0, "updated_at": null, "memory_mb": 64, "vcpus": 1, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "is_public": true, "deleted_at": null, "vcpu_weight": 0, "id": 6}, "nova_object.namespace": "nova"}}', None)]
2015-07-31 01:52:20.792 18304 ERROR nova.api.openstack

5 hits in 7 days, check and gate, all failures:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiY3JlYXRlX2RiX2VudHJ5X2Zvcl9uZXdfaW5zdGFuY2VcIiBBTkQgbWVzc2FnZTpcIkRlYWRsb2NrIGZvdW5kIHdoZW4gdHJ5aW5nIHRvIGdldCBsb2NrOyB0cnkgcmVzdGFydGluZyB0cmFuc2FjdGlvbicpIFtTUUw6IHUnSU5TRVJUIElOVE8gaW5zdGFuY2VfZXh0cmFcIiBBTkQgdGFnczpcInNjcmVlbi1uLWFwaS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQzODM0OTYzNTY4OX0=

[Matt Riedemann (mriedem)]

This is a better query:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22create_db_entry_for_new_instance%5C%22%20AND%20message:%5C%22DBDeadlock%5C%22%20AND%20message:%5C%22u'INSERT%20INTO%20instance_extra%5C%22%20AND%20tags:%5C%22screen-n-api.txt%5C%22%20AND%20voting:%5C%221%5C%22

[Ihar Hrachyshka (ihar-hrachyshka)]

Affects Liberty: http://logs.openstack.org/98/268698/1/check/gate-tempest-dsvm-neutron-dvr/0a7f4ac/logs/screen-n-api.txt.gz?level=TRACE

[Armando Migliaccio (armando-migliaccio)]

I have seen this show up here:

http://logs.openstack.org/20/275420/25/gate/gate-tempest-dsvm-neutron-linuxbridge/cb556d1/logs/screen-n-api.txt.gz?#_2016-03-15_02_04_36_061

message:"DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'INSERT INTO instance_extra"

5 occurrences in 7 days

[Armando Migliaccio (armando-migliaccio)]

Looks like this is in elastic recheck already.

[Armando Migliaccio (armando-migliaccio)]

http://status.openstack.org//elastic-recheck/#1480305

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/292800

[Sylvain Bauza (sylvain-bauza)]

Agreed on IRC to drop the RC potentiality of that bugfix.

[OpenStack Infra (hudson-openstack)]

Change abandoned by John Garbutt (<email address hidden>) on branch: master
Review: https://review.openstack.org/292800
Reason: I don't have the bandwidth to chase this down the more complicated fix.

Since we don't seem happy with the quick fix, abandoning this so someone else which more knowledge and time can take a closer look into the problem here.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/292800
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=df15e467b61fee781e78b07bf910d6b411bafd44
Submitter: Jenkins
Branch: master

commit df15e467b61fee781e78b07bf910d6b411bafd44
Author: John Garbutt <email address hidden>
Date: Tue Mar 15 10:21:47 2016 +0000

    db: retry on deadlocks while adding an instance

    We are hitting deadlocks in the gate when we are inserting the new
    instance_extra row into the DB.

    We should follow up this fix and look at way to avoid the deadlock
    happening rather than retrying it. It currently doesn't happen too
    often, so this should be enough to stop the problem while we work on a
    better fix.

    Closes-Bug: #1480305

    Change-Id: Iba218bf28c7d1e6040c551fe836d6fa5e5e45f4d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/367508
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fdfab99ba8e66ac15c25fcf5f2b3503aeab73742
Submitter: Jenkins
Branch: stable/mitaka

commit fdfab99ba8e66ac15c25fcf5f2b3503aeab73742
Author: John Garbutt <email address hidden>
Date: Tue Mar 15 10:21:47 2016 +0000

    db: retry on deadlocks while adding an instance

    We are hitting deadlocks in the gate when we are inserting the new
    instance_extra row into the DB.

    We should follow up this fix and look at way to avoid the deadlock
    happening rather than retrying it. It currently doesn't happen too
    often, so this should be enough to stop the problem while we work on a
    better fix.

    Closes-Bug: #1480305

    Change-Id: Iba218bf28c7d1e6040c551fe836d6fa5e5e45f4d
    (cherry picked from commit df15e467b61fee781e78b07bf910d6b411bafd44)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 13.1.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 13.1.2 release.