Deadlock with quota when deleting port

Bug #1771293 reported by Thomas Herve
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
Critical
Oleg Bondarev

Bug Description

Found here: http://logs.openstack.org/38/567238/3/check/heat-functional-convg-mysql-lbaasv2-py35/295509a/logs/screen-q-svc.txt.gz#_May_14_09_25_12_996826

The following query fails:

oslo_db.exception.DBDeadlock: (pymysql.err.InternalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: 'UPDATE quotausages SET dirty=%(dirty)s WHERE quotausages.project_id = %(quotausages_project_id)s AND quotausages.resource = %(quotausages_resource)s'] [parameters: {'dirty': 1, 'quotausages_project_id': '9774958ed24f4e28b5d2f5d72863861d', 'quotausages_resource': 'network'}

The incoming DELETE API call fails with a 500.

Gary Kotton (garyk)
tags: added: db
Changed in neutron:
importance: Undecided → Critical
Revision history for this message
LIU Yulong (dragon889) wrote :

I just submitted a new bug with similar exception, but it's during port creating:
https://bugs.launchpad.net/neutron/+bug/1777965

Revision history for this message
pandatt (pandatt) wrote :
pandatt (pandatt)
Changed in neutron:
status: New → In Progress
assignee: nobody → pandatt (pandatt)
status: In Progress → Confirmed
pandatt (pandatt)
Changed in neutron:
assignee: pandatt (pandatt) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/683128

Changed in neutron:
assignee: nobody → Oleg Bondarev (obondarev)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/683128
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ab286bcdaccb788ab9df3186e0605e93a9b10bbc
Submitter: Zuul
Branch: master

commit ab286bcdaccb788ab9df3186e0605e93a9b10bbc
Author: Oleg Bondarev <email address hidden>
Date: Thu Sep 19 16:11:06 2019 +0400

    Set DB retry for quota_enforcement pecan_wsgi hook

    The hook starts a DB transaction and should be covered with
    DB retry decorator.

    Closes-Bug: #1777965
    Closes-Bug: #1771293
    Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/695351

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/695352

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/695352
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3ec7aed8a367ba2a44fed300b08797e02fc47e06
Submitter: Zuul
Branch: stable/stein

commit 3ec7aed8a367ba2a44fed300b08797e02fc47e06
Author: Oleg Bondarev <email address hidden>
Date: Thu Sep 19 16:11:06 2019 +0400

    Set DB retry for quota_enforcement pecan_wsgi hook

    The hook starts a DB transaction and should be covered with
    DB retry decorator.

    Closes-Bug: #1777965
    Closes-Bug: #1771293
    Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7
    (cherry picked from commit ab286bcdaccb788ab9df3186e0605e93a9b10bbc)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/695351
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ff810fcd80af8054763b0751e1309944d39bbc81
Submitter: Zuul
Branch: stable/train

commit ff810fcd80af8054763b0751e1309944d39bbc81
Author: Oleg Bondarev <email address hidden>
Date: Thu Sep 19 16:11:06 2019 +0400

    Set DB retry for quota_enforcement pecan_wsgi hook

    The hook starts a DB transaction and should be covered with
    DB retry decorator.

    Closes-Bug: #1777965
    Closes-Bug: #1771293
    Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7
    (cherry picked from commit ab286bcdaccb788ab9df3186e0605e93a9b10bbc)

tags: added: in-stable-train
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/698002

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/698002
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=22250e783b194b9d188268c0d301581d73bd430d
Submitter: Zuul
Branch: stable/rocky

commit 22250e783b194b9d188268c0d301581d73bd430d
Author: Oleg Bondarev <email address hidden>
Date: Thu Sep 19 16:11:06 2019 +0400

    Set DB retry for quota_enforcement pecan_wsgi hook

    The hook starts a DB transaction and should be covered with
    DB retry decorator.

    For Rocky backport had to import neutron_lib.db for retry_db_errors.

    Closes-Bug: #1777965
    Closes-Bug: #1771293
    Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7
    (cherry picked from commit ab286bcdaccb788ab9df3186e0605e93a9b10bbc)
    (cherry picked from commit 3ec7aed8a367ba2a44fed300b08797e02fc47e06)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/699193

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/699193
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2084585b81649b2afa747e24f036d54e5e16e90e
Submitter: Zuul
Branch: stable/queens

commit 2084585b81649b2afa747e24f036d54e5e16e90e
Author: Oleg Bondarev <email address hidden>
Date: Thu Sep 19 16:11:06 2019 +0400

    Set DB retry for quota_enforcement pecan_wsgi hook

    The hook starts a DB transaction and should be covered with
    DB retry decorator.

    For Rocky backport had to import neutron_lib.db for retry_db_errors.

    Closes-Bug: #1777965
    Closes-Bug: #1771293
    Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7
    (cherry picked from commit ab286bcdaccb788ab9df3186e0605e93a9b10bbc)
    (cherry picked from commit 3ec7aed8a367ba2a44fed300b08797e02fc47e06)
    (cherry picked from commit 22250e783b194b9d188268c0d301581d73bd430d)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/699938

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 13.0.6

This issue was fixed in the openstack/neutron 13.0.6 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 14.0.4

This issue was fixed in the openstack/neutron 14.0.4 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 15.0.1

This issue was fixed in the openstack/neutron 15.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/pike)

Reviewed: https://review.opendev.org/699938
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8e88ae8dc32239a21985d331ec25641f0a4757a5
Submitter: Zuul
Branch: stable/pike

commit 8e88ae8dc32239a21985d331ec25641f0a4757a5
Author: Oleg Bondarev <email address hidden>
Date: Thu Sep 19 16:11:06 2019 +0400

    Set DB retry for quota_enforcement pecan_wsgi hook

    The hook starts a DB transaction and should be covered with
    DB retry decorator.

    For Rocky backport had to import neutron_lib.db for retry_db_errors.

    Closes-Bug: #1777965
    Closes-Bug: #1771293
    Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7
    (cherry picked from commit ab286bcdaccb788ab9df3186e0605e93a9b10bbc)
    (cherry picked from commit 3ec7aed8a367ba2a44fed300b08797e02fc47e06)
    (cherry picked from commit 22250e783b194b9d188268c0d301581d73bd430d)
    (cherry picked from commit 2084585b81649b2afa747e24f036d54e5e16e90e)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.0.0.0b1

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.

tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/740977

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/740977
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers