Cinder

Deadlocked queries retry forever

Bug #1738292 reported by Mohammed Naser
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Medium
Sean McGinnis

Bug Description

The `_retry_on_deadlock` function in the SQLalchemy DB API retries forever with a 0.1s sleep which in rare cases can lead for it to be stuck and deadlocking forever.

It seems that oslo_db provides a wrapper to resolve this:

@oslo_db_api.wrap_db_retry(max_retries=5, retry_on_deadlock=True)

Sean McGinnis (sean-mcginnis)
Changed in cinder:
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/528107

Changed in cinder:
assignee: nobody → Sean McGinnis (sean-mcginnis)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/528107
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=3aae507a2a90c522e78760e53d2c525d3c6b1f88
Submitter: Zuul
Branch: master

commit 3aae507a2a90c522e78760e53d2c525d3c6b1f88
Author: Sean McGinnis <email address hidden>
Date: Thu Dec 14 16:28:28 2017 -0600

    Switch to oslo_db retry decorator

    Our sqlalchemy code uses a custom retry decorator that does not
    limit the number of retries. Since this code was written, oslo_db
    now has a retry decorator better used for this purpose that limits
    the number of retry attempts.

    Change-Id: If39a81bbea26c14604b151a51d8e2bbe1bb2f110
    Closes-bug: #1738292

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/528442

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/pike)

Reviewed: https://review.openstack.org/528442
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=82c9d001606c50e9647464ee8619ce1bc060ddff
Submitter: Zuul
Branch: stable/pike

commit 82c9d001606c50e9647464ee8619ce1bc060ddff
Author: Sean McGinnis <email address hidden>
Date: Thu Dec 14 16:28:28 2017 -0600

    Switch to oslo_db retry decorator

    Our sqlalchemy code uses a custom retry decorator that does not
    limit the number of retries. Since this code was written, oslo_db
    now has a retry decorator better used for this purpose that limits
    the number of retry attempts.

    Closes-bug: #1738292
    (cherry picked from commit 3aae507a2a90c522e78760e53d2c525d3c6b1f88)

    Conflicts:
      cinder/db/sqlalchemy/api.py

    Change-Id: If39a81bbea26c14604b151a51d8e2bbe1bb2f110

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 12.0.0.0b3

This issue was fixed in the openstack/cinder 12.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 11.1.0

This issue was fixed in the openstack/cinder 11.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.