Deadlock updating image properties in glance-registry daemon

[Sean Dague]

Glance Registry is now regularly deadlocking when trying to update image properties. This causes gate failures.

Example Log: http://logs.openstack.org/33/100933/3/gate/gate-tempest-dsvm-full/e21d71a/logs/screen-g-reg.txt.gz#_2014-07-09_00_27_38_261

Logstash query: message:'Deadlock found when trying to get lock; try restarting transaction' AND tags:screen-g-reg.txt

[Sean Dague (sdague)]

90 fails in the last 7 days. This is a pretty major issue.

[Vish Ishaya (vishvananda)]

It looks like _update_image_properties needs a @retry_on_deadlock error and it needs to be put on update_image_properties.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/106764

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/106764
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=149074976c0a33bd67a4bfcabb6d2e65dac1310d
Submitter: Jenkins
Branch: master

commit 149074976c0a33bd67a4bfcabb6d2e65dac1310d
Author: Flavio Percoco <email address hidden>
Date: Mon Jul 14 15:18:17 2014 +0200

    Add a `_retry_on_deadlock` decorator

    This patch adds a `_retry_on_deadlock` decorator from Nova. The aim is
    to retry db transactions that had failed because of race-conditions.
    Although this is not the best and most ideal fix - it'd be better to get
    rid of the race condition - the fix does adds a guard against unseen
    reace conditions for some of the functions in the database API.

    I'll contribute this patch to oslo.db but in the meantime, I'd prefer to
    let it land in Glance and then clean it up once it's in `oslo.db`.

    Change-Id: Iea1fa94a7c8690c874859b1b1e9fd1cdf29fed21
    Fixes-bug: #1339735

[Matt Riedemann (mriedem)]

Looks like this is still an issue:

http://logs.openstack.org/08/114008/3/check/check-grenade-dsvm/ae4dd30/logs/new/screen-g-reg.txt.gz?level=TRACE