Database deadlocks in neutron-server logs

[Eugene Nikanorov]

Several thousands of 'deadlock' errors like:

 Exception during message handling: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction')

Could be found in neutron-server logs during rally tests run.

There is a suspicion that it is a result of the fix that has changed transaction isolation level for mysql.

There are also similar bugs that result in deadlock/lock wait timeouts but happen quite rarely.

Info: This errors appears when we try run transaction in agent tables

[Alexander Ignatov (aignatov)]

Please don't mark it as dublicate of https://bugs.launchpad.net/mos/+bug/1392386 It is different issues.

[OSCI Robot (oscirobot)]

RPM package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0.mira14.git.5c97239.a299407

Changeset: https://review.fuel-infra.org/1383
project: openstack/neutron
branch: openstack-ci/fuel-6.0/2014.2
author: Eugene Nikanorov
committer: Eugene Nikanorov
subject: Change transaction isolation level back after create_network
status: patchset-created

Files placed on repository:
openstack-neutron-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-bigswitch-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-brocade-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-cisco-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-hyperv-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-ibm-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-linuxbridge-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-mellanox-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-metaplugin-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-metering-agent-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-midonet-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-ml2-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-nec-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-nuage-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-ofagent-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-oneconvergence-nvsd-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-opencontrail-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-openvswitch-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-plumgrid-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-ryu-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-vmware-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
openstack-neutron-vpn-agent-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm
python-neutron-2014.2-fuel6.0.mira14.git.5c97239.a299407.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-1383/centos

[OSCI Robot (oscirobot)]

DEB package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0~mira13+git.5c97239.a299407

Changeset: https://review.fuel-infra.org/1383
project: openstack/neutron
branch: openstack-ci/fuel-6.0/2014.2
author: Eugene Nikanorov
committer: Eugene Nikanorov
subject: Change transaction isolation level back after create_network
status: patchset-created

Files placed on repository:
neutron-common_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-dhcp-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-l3-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-lbaas-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-metadata-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-metering-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-bigswitch-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-bigswitch_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-brocade_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-cisco_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-hyperv_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-ibm-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-ibm_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-linuxbridge-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-linuxbridge_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-metaplugin_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-metering-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-midonet_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-ml2_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-mlnx-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-mlnx_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-nec-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-nec_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-nicira_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-oneconvergence-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-oneconvergence_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-openflow-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-openvswitch-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-openvswitch_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-plumgrid_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-ryu-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-ryu_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-vmware_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-plugin-vpn-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-server_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
neutron-vpn-agent_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb
python-neutron_2014.2-fuel6.0~mira13+git.5c97239.a299407_all.deb

NOTE: Changeset is not merged,...

[Mike Scherbakov (mihgen)]

Eugene,
how does it affect user experience? How often this occurs?

[Bogdan Dobrelya (bogdando)]

Please elaborate "the fix that has changed transaction isolation level for mysql." ?

[Alexander Ignatov (aignatov)]

Bogdan, Eugene meant fix for this bug https://bugs.launchpad.net/mos/+bug/1381338 which changed isolation level.

Mike, this bug causes situtation that agents can't report their state to db, it leads for dyssynchrony between agents.

[OSCI Robot (oscirobot)]

RPM package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0.mira14

Changeset: https://review.fuel-infra.org/1383
project: openstack/neutron
branch: openstack-ci/fuel-6.0/2014.2
author: Eugene Nikanorov
committer: Eugene Nikanorov
subject: Change transaction isolation level back after create_network
status: change-merged

Files placed on repository:
openstack-neutron-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-bigswitch-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-brocade-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-cisco-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-hyperv-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-ibm-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-linuxbridge-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-mellanox-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-metaplugin-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-metering-agent-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-midonet-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-ml2-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-nec-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-nuage-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-ofagent-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-oneconvergence-nvsd-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-opencontrail-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-openvswitch-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-plumgrid-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-ryu-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-vmware-2014.2-fuel6.0.mira14.noarch.rpm
openstack-neutron-vpn-agent-2014.2-fuel6.0.mira14.noarch.rpm
python-neutron-2014.2-fuel6.0.mira14.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable/centos

[OSCI Robot (oscirobot)]

DEB package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0~mira13

Changeset: https://review.fuel-infra.org/1383
project: openstack/neutron
branch: openstack-ci/fuel-6.0/2014.2
author: Eugene Nikanorov
committer: Eugene Nikanorov
subject: Change transaction isolation level back after create_network
status: change-merged

Files placed on repository:
neutron-common_2014.2-fuel6.0~mira13_all.deb
neutron-dhcp-agent_2014.2-fuel6.0~mira13_all.deb
neutron-l3-agent_2014.2-fuel6.0~mira13_all.deb
neutron-lbaas-agent_2014.2-fuel6.0~mira13_all.deb
neutron-metadata-agent_2014.2-fuel6.0~mira13_all.deb
neutron-metering-agent_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-bigswitch-agent_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-bigswitch_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-brocade_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-cisco_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-hyperv_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-ibm-agent_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-ibm_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-linuxbridge-agent_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-linuxbridge_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-metaplugin_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-metering-agent_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-midonet_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-ml2_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-mlnx-agent_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-mlnx_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-nec-agent_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-nec_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-nicira_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-oneconvergence-agent_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-oneconvergence_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-openflow-agent_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-openvswitch-agent_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-openvswitch_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-plumgrid_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-ryu-agent_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-ryu_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-vmware_2014.2-fuel6.0~mira13_all.deb
neutron-plugin-vpn-agent_2014.2-fuel6.0~mira13_all.deb
neutron-server_2014.2-fuel6.0~mira13_all.deb
neutron-vpn-agent_2014.2-fuel6.0~mira13_all.deb
python-neutron_2014.2-fuel6.0~mira13_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable/ubuntu

[Eugene Nikanorov (enikanorov)]

Unfortunately at this point there is no strict evidence that this bug has been caused by the fix for https://bugs.launchpad.net/mos/+bug/1381338

Database deadlocks are quite common issues with neutron DB access which don't have simple solution.

The issue observed here has not visible user experience impact, however apparently it lead to inability of agents to update their status on neutron server, which then may lead to various failures.

The fix proposed for this bug is more a safety measure against possible negative side effect of the fix for https://bugs.launchpad.net/mos/+bug/1381338

[Tatyanka (tatyana-leontovich)]

Still reproduced
{"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}}, "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}

[Tatyanka (tatyana-leontovich)]

here is output from node as about packages:
[root@node-1 neutron]# rpm -qa | grep neutron
openstack-neutron-ml2-2014.2-fuel6.0.mira14.noarch
openstack-neutron-openvswitch-2014.2-fuel6.0.mira14.noarch
openstack-neutron-2014.2-fuel6.0.mira14.noarch
python-neutronclient-2.3.9-fuel6.0.mira21.noarch
python-neutron-2014.2-fuel6.0.mira14.noarch
[root@node-1 neutron]#

Scenario was - deploy ha with Centos (neutron vlan)
When cluster ready - run ostf
ssh on primary controller delete public vip
wait until it recovers
run ostf - delete network failed with error:
http://paste.openstack.org/show/155060/

[Tatyanka (tatyana-leontovich)]

Move to High status according to issue is not reproduced all the time

[Tatyanka (tatyana-leontovich)]

As sson as thre is not a lot of such error sin the log on a small env - move to the medium for 6.0.1. If we see tons of the same erros - we should increase importance

[Eugene Nikanorov (enikanorov)]

 'Deadlock found when trying to get lock; try restarting transaction' could be caused either by neutron issues with db or by particular issues with galera.

Right now we don't have enough information to analyze with this bug and this issue was only seen once.

[Alexander Ignatov (aignatov)]

Moved to Medium since it's very rare issue and doesn't seem it has functional impact

[Dmitry Mescheryakov (dmitrymex)]

This is a medium issue, so are not going to fix it in maintenance release

[Dmitry Mescheryakov (dmitrymex)]

The bug is reproduced on 6.0.1 staging job:
http://jenkins-product.srt.mirantis.net:8080/job/6.0.1.staging.centos.bvt_1/49/

Also, this time there was functional impact of DBDeadlock - it caused InternalServerError on network deletion request and that in turn caused OSTF tests failure.

[Alexander Ignatov (aignatov)]

This bug seems to be fixed by upstream patch https://review.openstack.org/#/c/115360/
related to https://bugs.launchpad.net/neutron/+bug/1332917

[OSCI Robot (oscirobot)]

package neutron has been built for project openstack/neutron
Package version == 2014.2.2, package release == fuel6.1.mira3.git.a1b651a.ea24413

Changeset: https://review.fuel-infra.org/3570
project: openstack/neutron
branch: openstack-ci/fuel-6.1/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Remove locking from network and subnet delete op
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-3570/

[OSCI Robot (oscirobot)]

package neutron has been built for project openstack/neutron
Package version == 2014.2.2, package release == fuel6.1~mira2+git.a1b651a.ea24413

Changeset: https://review.fuel-infra.org/3570
project: openstack/neutron
branch: openstack-ci/fuel-6.1/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Remove locking from network and subnet delete op
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /ubuntu-fuel-6.1-stable-3570/

[OSCI Robot (oscirobot)]

RPM package neutron has been built for project openstack/neutron
Package version == 2014.2.2, package release == fuel6.0.1.mira0.git.9c0c4d3.77ff2e8

Changeset: https://review.fuel-infra.org/3648
project: openstack/neutron
branch: openstack-ci/fuel-6.0.1/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Remove locking from network and subnet delete op
status: patchset-created

Files placed on repository:
openstack-neutron-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-bigswitch-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-brocade-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-cisco-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-hyperv-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-ibm-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-linuxbridge-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-mellanox-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-metaplugin-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-metering-agent-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-midonet-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-ml2-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-nec-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-nuage-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-ofagent-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-oneconvergence-nvsd-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-opencontrail-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-openvswitch-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-plumgrid-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-ryu-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-vmware-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
openstack-neutron-vpn-agent-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm
python-neutron-2014.2.2-fuel6.0.1.mira0.git.9c0c4d3.77ff2e8.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-3648/centos

[OSCI Robot (oscirobot)]

DEB package neutron has been built for project openstack/neutron
Package version == 2014.2.2, package release == fuel6.0.1~mira0+git.9c0c4d3.77ff2e8

Changeset: https://review.fuel-infra.org/3648
project: openstack/neutron
branch: openstack-ci/fuel-6.0.1/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Remove locking from network and subnet delete op
status: patchset-created

Files placed on repository:
neutron-common_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-dhcp-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-l3-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-lbaas-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-metadata-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-metering-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-bigswitch-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-bigswitch_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-brocade_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-cisco_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-hyperv_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-ibm-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-ibm_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-linuxbridge-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-linuxbridge_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-metaplugin_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-metering-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-midonet_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-ml2_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-mlnx-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-mlnx_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-nec-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-nec_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-nicira_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-oneconvergence-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-oneconvergence_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-openflow-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-openvswitch-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-openvswitch_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-plumgrid_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-ryu-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-ryu_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-vmware_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-plugin-vpn-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-server_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_all.deb
neutron-vpn-agent_2014.2.2-fuel6.0.1~mira0+git.9c0c4d3.77ff2e8_a...

[OSCI Robot (oscirobot)]

RPM package neutron has been built for project openstack/neutron
Package version == 2014.2.2, package release == fuel6.0.1.mira0.git.ded499a.77ff2e8

Changeset: https://review.fuel-infra.org/3648
project: openstack/neutron
branch: openstack-ci/fuel-6.0.1/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Remove locking from network and subnet delete op
status: patchset-created

Files placed on repository:
openstack-neutron-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-bigswitch-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-brocade-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-cisco-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-hyperv-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-ibm-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-linuxbridge-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-mellanox-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-metaplugin-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-metering-agent-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-midonet-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-ml2-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-nec-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-nuage-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-ofagent-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-oneconvergence-nvsd-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-opencontrail-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-openvswitch-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-plumgrid-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-ryu-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-vmware-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
openstack-neutron-vpn-agent-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm
python-neutron-2014.2.2-fuel6.0.1.mira0.git.ded499a.77ff2e8.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-3648/centos

[OSCI Robot (oscirobot)]

DEB package neutron has been built for project openstack/neutron
Package version == 2014.2.2, package release == fuel6.0.1~mira0+git.ded499a.77ff2e8

Changeset: https://review.fuel-infra.org/3648
project: openstack/neutron
branch: openstack-ci/fuel-6.0.1/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Remove locking from network and subnet delete op
status: patchset-created

Files placed on repository:
neutron-common_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-dhcp-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-l3-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-lbaas-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-metadata-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-metering-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-bigswitch-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-bigswitch_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-brocade_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-cisco_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-hyperv_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-ibm-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-ibm_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-linuxbridge-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-linuxbridge_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-metaplugin_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-metering-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-midonet_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-ml2_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-mlnx-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-mlnx_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-nec-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-nec_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-nicira_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-oneconvergence-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-oneconvergence_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-openflow-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-openvswitch-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-openvswitch_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-plumgrid_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-ryu-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-ryu_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-vmware_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-plugin-vpn-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-server_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_all.deb
neutron-vpn-agent_2014.2.2-fuel6.0.1~mira0+git.ded499a.77ff2e8_a...

[OSCI Robot (oscirobot)]

package neutron has been built for project openstack/neutron
Package version == 2014.2.2, package release == fuel6.1.mira3

Changeset: https://review.fuel-infra.org/3570
project: openstack/neutron
branch: openstack-ci/fuel-6.1/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Remove locking from network and subnet delete op
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /centos-fuel-6.1-stable/

[OSCI Robot (oscirobot)]

package neutron has been built for project openstack/neutron
Package version == 2014.2.2, package release == fuel6.1~mira2

Changeset: https://review.fuel-infra.org/3570
project: openstack/neutron
branch: openstack-ci/fuel-6.1/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Remove locking from network and subnet delete op
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /ubuntu-fuel-6.1-stable/

[OSCI Robot (oscirobot)]

RPM package neutron has been built for project openstack/neutron
Package version == 2014.2.2, package release == fuel6.0.1.mira1

Changeset: https://review.fuel-infra.org/3648
project: openstack/neutron
branch: openstack-ci/fuel-6.0.1/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Remove locking from network and subnet delete op
status: change-merged

Files placed on repository:
openstack-neutron-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-bigswitch-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-brocade-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-cisco-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-hyperv-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-ibm-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-linuxbridge-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-mellanox-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-metaplugin-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-metering-agent-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-midonet-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-ml2-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-nec-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-nuage-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-ofagent-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-oneconvergence-nvsd-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-opencontrail-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-openvswitch-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-plumgrid-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-ryu-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-vmware-2014.2.2-fuel6.0.1.mira1.noarch.rpm
openstack-neutron-vpn-agent-2014.2.2-fuel6.0.1.mira1.noarch.rpm
python-neutron-2014.2.2-fuel6.0.1.mira1.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable/centos

[OSCI Robot (oscirobot)]

DEB package neutron has been built for project openstack/neutron
Package version == 2014.2.2, package release == fuel6.0.1~mira1

Changeset: https://review.fuel-infra.org/3648
project: openstack/neutron
branch: openstack-ci/fuel-6.0.1/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Remove locking from network and subnet delete op
status: change-merged

Files placed on repository:
neutron-common_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-dhcp-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-l3-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-lbaas-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-metadata-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-metering-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-bigswitch-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-bigswitch_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-brocade_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-cisco_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-hyperv_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-ibm-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-ibm_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-linuxbridge-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-linuxbridge_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-metaplugin_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-metering-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-midonet_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-ml2_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-mlnx-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-mlnx_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-nec-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-nec_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-nicira_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-oneconvergence-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-oneconvergence_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-openflow-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-openvswitch-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-openvswitch_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-plumgrid_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-ryu-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-ryu_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-vmware_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-plugin-vpn-agent_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-server_2014.2.2-fuel6.0.1~mira1_all.deb
neutron-vpn-agent_2014.2.2-fuel6.0.1~mira1_all.deb
python-neutron_2014.2.2-fuel6.0.1~mira1_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

[Eugene Nikanorov (enikanorov)]

I'm not sure we did anything to fix this issue, however i have an idea that this bug might have been caused by incorrect configuration of haproxy endpoint that is balancing mysql galera.
If for some reason galera went into multi-writer mode (which is actually defined by haproxy configuration) then multiple deadlock errors are possible.

[Alexander Ignatov (aignatov)]

Will wait until issue reproduced again

[Alexander Ignatov (aignatov)]

Moving this bug to Invalid since it was not reproduced still.

[Eugene Nikanorov (enikanorov)]

This was observed on customer environment and indeed has negative effect on user experience.
Agents begin to flap, resources begin to be rescheduled, pacemaker restarts agents interrupting connectivity.

The root cause seems to be a moment when galera is falling apart.
In the process of sync balancing changes and it becomes multi-master, unlike regular mode where we have 1 master and 2 backups.

The minimal fix of the problem should include wrapping operations on agent's table in a retrying decorator to be able to update agents and avoid aforementioned problems.

[Eugene Nikanorov (enikanorov)]

To artificially reproduce the bug i suggest to change galera's VIP config to point on all 3 servers (or on 2) instead of 1 active+2 backups.