Create port get quota related DBDeadLock

Bug #1777965 reported by LIU Yulong
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Undecided
Oleg Bondarev

Bug Description

ENV:
Neutron stable/queens (12.0.1)
CentOS 7 (3.10.0-514.26.2.el7.x86_64)
Ceph v10.2.9 Jewel

Exception:
2018-06-20 14:21:52.070 140217 ERROR oslo_middleware.catch_errors DBDeadlock: (pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded; try restarting transaction') [SQL: u'UPDATE quotausages SET dirty=%(dirty)s WHERE quotausages.project_id = %(quotausages_project_id)s AND quotausages.resource = %(quotausages_resource)s'] [parameters: {'quotausages_project_id': u'f4ff15be8de443b78baf21640d93132b', 'dirty': 1, 'quotausages_resource': u'port'}] (Background on this error at: http://sqlalche.me/e/2j85)

API req and resp:
req:
2018-06-20 14:21:35.997 140217 DEBUG neutron.api.v2.base [req-1622426f-193c-439a-ac8e-09f9cd1809e5 03570241e4ea4f2d8e52e48eabc73f8e f4ff15be8de443b78baf21640d93132b - default default] Request body: {u'port': {u'network_id': u'014d37e6-1d99-42de-8023-0859c0721ddc', u'tenant_id': u'f4ff15be8de443b78baf21640d93132b', u'device_id': u'ccdc82d7-cf7e-473d-8721-b09a91a5f10f', u'admin_state_up': True}} prepare_request_body /usr/lib/python2.7/site-packages/neutron/api/v2/base.py:690

500 resp:
2018-06-20 14:21:52.075 140217 INFO neutron.wsgi [req-1622426f-193c-439a-ac8e-09f9cd1809e5 03570241e4ea4f2d8e52e48eabc73f8e f4ff15be8de443b78baf21640d93132b - default default] 10.129.169.147 "POST /v2.0/ports HTTP/1.1" status: 500 len: 399 time: 16.0932069

LOG:
http://paste.openstack.org/show/723982/

How to reproduce:
Use nova boot with --min_count 300 to create 1core1g test VM.

Revision history for this message
LIU Yulong (dragon889) wrote :

Similar bug:
Deadlock with quota when deleting port
https://bugs.launchpad.net/neutron/+bug/1771293

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/683128

Changed in neutron:
assignee: nobody → Oleg Bondarev (obondarev)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/683128
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ab286bcdaccb788ab9df3186e0605e93a9b10bbc
Submitter: Zuul
Branch: master

commit ab286bcdaccb788ab9df3186e0605e93a9b10bbc
Author: Oleg Bondarev <email address hidden>
Date: Thu Sep 19 16:11:06 2019 +0400

    Set DB retry for quota_enforcement pecan_wsgi hook

    The hook starts a DB transaction and should be covered with
    DB retry decorator.

    Closes-Bug: #1777965
    Closes-Bug: #1771293
    Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/695351

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/695352

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/695352
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3ec7aed8a367ba2a44fed300b08797e02fc47e06
Submitter: Zuul
Branch: stable/stein

commit 3ec7aed8a367ba2a44fed300b08797e02fc47e06
Author: Oleg Bondarev <email address hidden>
Date: Thu Sep 19 16:11:06 2019 +0400

    Set DB retry for quota_enforcement pecan_wsgi hook

    The hook starts a DB transaction and should be covered with
    DB retry decorator.

    Closes-Bug: #1777965
    Closes-Bug: #1771293
    Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7
    (cherry picked from commit ab286bcdaccb788ab9df3186e0605e93a9b10bbc)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/695351
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ff810fcd80af8054763b0751e1309944d39bbc81
Submitter: Zuul
Branch: stable/train

commit ff810fcd80af8054763b0751e1309944d39bbc81
Author: Oleg Bondarev <email address hidden>
Date: Thu Sep 19 16:11:06 2019 +0400

    Set DB retry for quota_enforcement pecan_wsgi hook

    The hook starts a DB transaction and should be covered with
    DB retry decorator.

    Closes-Bug: #1777965
    Closes-Bug: #1771293
    Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7
    (cherry picked from commit ab286bcdaccb788ab9df3186e0605e93a9b10bbc)

tags: added: in-stable-train
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/698002

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/698002
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=22250e783b194b9d188268c0d301581d73bd430d
Submitter: Zuul
Branch: stable/rocky

commit 22250e783b194b9d188268c0d301581d73bd430d
Author: Oleg Bondarev <email address hidden>
Date: Thu Sep 19 16:11:06 2019 +0400

    Set DB retry for quota_enforcement pecan_wsgi hook

    The hook starts a DB transaction and should be covered with
    DB retry decorator.

    For Rocky backport had to import neutron_lib.db for retry_db_errors.

    Closes-Bug: #1777965
    Closes-Bug: #1771293
    Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7
    (cherry picked from commit ab286bcdaccb788ab9df3186e0605e93a9b10bbc)
    (cherry picked from commit 3ec7aed8a367ba2a44fed300b08797e02fc47e06)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/699193

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/699193
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2084585b81649b2afa747e24f036d54e5e16e90e
Submitter: Zuul
Branch: stable/queens

commit 2084585b81649b2afa747e24f036d54e5e16e90e
Author: Oleg Bondarev <email address hidden>
Date: Thu Sep 19 16:11:06 2019 +0400

    Set DB retry for quota_enforcement pecan_wsgi hook

    The hook starts a DB transaction and should be covered with
    DB retry decorator.

    For Rocky backport had to import neutron_lib.db for retry_db_errors.

    Closes-Bug: #1777965
    Closes-Bug: #1771293
    Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7
    (cherry picked from commit ab286bcdaccb788ab9df3186e0605e93a9b10bbc)
    (cherry picked from commit 3ec7aed8a367ba2a44fed300b08797e02fc47e06)
    (cherry picked from commit 22250e783b194b9d188268c0d301581d73bd430d)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/699938

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 13.0.6

This issue was fixed in the openstack/neutron 13.0.6 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 14.0.4

This issue was fixed in the openstack/neutron 14.0.4 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 15.0.1

This issue was fixed in the openstack/neutron 15.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/pike)

Reviewed: https://review.opendev.org/699938
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8e88ae8dc32239a21985d331ec25641f0a4757a5
Submitter: Zuul
Branch: stable/pike

commit 8e88ae8dc32239a21985d331ec25641f0a4757a5
Author: Oleg Bondarev <email address hidden>
Date: Thu Sep 19 16:11:06 2019 +0400

    Set DB retry for quota_enforcement pecan_wsgi hook

    The hook starts a DB transaction and should be covered with
    DB retry decorator.

    For Rocky backport had to import neutron_lib.db for retry_db_errors.

    Closes-Bug: #1777965
    Closes-Bug: #1771293
    Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7
    (cherry picked from commit ab286bcdaccb788ab9df3186e0605e93a9b10bbc)
    (cherry picked from commit 3ec7aed8a367ba2a44fed300b08797e02fc47e06)
    (cherry picked from commit 22250e783b194b9d188268c0d301581d73bd430d)
    (cherry picked from commit 2084585b81649b2afa747e24f036d54e5e16e90e)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.0.0.0b1

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.

tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/740977

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/740977
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers