DBDeadlock during floating ip deletion

[Viktoria Efimova]

Dublicated bug filed in neutron space https://bugs.launchpad.net/neutron/+bug/1422504

api: '1.0'
astute_sha: 3f1ece0318e5e93eaf48802fefabf512ca1dce40
auth_required: true
build_id: 2015-03-26_21-32-43
build_number: '233'
feature_groups:
- mirantis
fuellib_sha: 9c7716bc2ce6075065d7d9dcf96f4c94662c0b56
fuelmain_sha: 320b5f46fc1b2798f9e86ed7df51d3bda1686c10
nailgun_sha: b163f6fc77d6639aaffd9dd992e1ad96951c3bbf
ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867
production: docker
python-fuelclient_sha: e5e8389d8d481561a4d7107a99daae07c6ec5177
release: '6.1'

Deployed the following environment:
Baremetal, Ubuntu,IBP, Neutron-vlan,Сeph-all,Nova-debug,nova-quotas,6.1_233
Controllers:1 Computes:5

Run shacker and after successful test passing shacker failed to clear the environment: failed to delete heat stack.

Neutron reports about the following error in neutron-server.log:
2015-04-21 13:50:16.035 11013 TRACE neutron.api.v2.resource DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'DELETE FROM floatingips WHERE floatingips.id = %s' ('c9c5a3eb-5a7b-453a-ab32-e229e0bfcc63',)

[Eugene Nikanorov (enikanorov)]

Victoria, please attach neutron.conf and plugin.ini from the environment

[Eugene Nikanorov (enikanorov)]

Also, configuration of the env would be helpful: which nodes are controllers and how many computes are there.

[Viktoria Efimova (vefimova)]

Eugene,
I attached neutron.conf and plugin.ini, also there is log snapshop at http://mos-scale-share.mirantis.com/fuel-snapshot-2015-04-21_15-24-20.tar.xz if you'll need something else...
Regarding configuration... in description above it was said that there are 1 controller (node-1) and 5 Computes.

[Oleg Bondarev (obondarev)]

A generic solution for such kind of error would be to add transaction retry on DBDeadlock errors to port_delete operation

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/6355

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/6355
Submitter: Eugene Nikanorov <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 906c4449178435e4df709689ff8ed7e4bb8fe087
Author: Oleg Bondarev <email address hidden>
Date: Thu Apr 30 09:40:53 2015

Wrap ML2 delete_port with db retry decorator

ML2 delete_port operation currently involves locking ports
and bindings tables which may lead to DBDeadlock errors in certain
cases when several ports are deleted concurrently.
That may happen due to specifics of Galera working in active-active
mode when it may throw fictitious deadlock errors
(in MOS that may happen during Galera master node failover)
The fix adds retries to delete port operation to overcome such
deadlocks

wrap_db_retry decorator was backported from master oslo_db

Closes-Bug: #1446783
Change-Id: Ibfa0b2d8c64b5fe05ef15ed4fa592126ab2ba258

[Leontii Istomin (listomin)]

Have not faced with the DBDeadlock error on 6.1-425 build yet.

[Leontii Istomin (listomin)]

can't reproduce original issue when can't delete floating ips, but have found DBDeadlock when update heartbeat_timestamp for neutron agents:
http://paste.openstack.org/show/237483/

[Sergii Turivnyi (sturivnyi)]

{u'build_id': u'2015-05-26_20-59-56', u'build_number': u'471', u'auth_required': True, u'fuel-ostf_sha': u'87819878bc0ca572900e1f6933d9b99e666d6f62', u'fuel-library_sha': u'8cfeca1a86179ebed1e4e03b2133b49c27350f6f', u'nailgun_sha': u'f737675091bd1903aace0e36812e855ce47dfec7', u'openstack_version': u'2014.2.2-6.1', u'production': u'docker', u'api': u'1.0', u'python-fuelclient_sha': u'e19f1b65792f84c4a18b5a9473f85ef3ba172fce', u'astute_sha': u'0bd72c72369e743376864e8e8dabfe873d40450a', u'fuelmain_sha': u'13b3e9cf074ba1cf1ae06509c55fbab613c73f4e', u'feature_groups': [u'mirantis'], u'release': u'6.1', u'release_versions': {u'2014.2.2-6.1': {u'VERSION': {u'build_id': u'2015-05-26_20-59-56', u'build_number': u'471', u'fuel-library_sha': u'8cfeca1a86179ebed1e4e03b2133b49c27350f6f', u'nailgun_sha': u'f737675091bd1903aace0e36812e855ce47dfec7', u'fuel-ostf_sha': u'87819878bc0ca572900e1f6933d9b99e666d6f62', u'production': u'docker', u'api': u'1.0', u'python-fuelclient_sha': u'e19f1b65792f84c4a18b5a9473f85ef3ba172fce', u'astute_sha': u'0bd72c72369e743376864e8e8dabfe873d40450a', u'fuelmain_sha': u'13b3e9cf074ba1cf1ae06509c55fbab613c73f4e', u'feature_groups': [u'mirantis'], u'release': u'6.1', u'openstack_version': u'2014.2.2-6.1'}}}}

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/8035

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Oleg Bondarev <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8035
Reason: Abondoning this one as backport to kilo was merged https://review.openstack.org/#/c/194174