DB Deadlock when creating a replacement resource

[Zane Bitter]

Seen once in the tests:

ERROR heat.engine.check_resource [None req-ec3bc8b0-5bbe-4638-8cef-946e55cd415f demo None] Unexpected exception in resource check.: DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'UPDATE resource SET status=%(status)s, atomic_key=%(atomic_key)s, replaced_by=%(replaced_by)s WHERE resource.atomic_key = %(atomic_key_1)s AND resource.id = %(id_1)s AND resource.engine_id IS NULL'] [parameters: {'status': 'COMPLETE', 'atomic_key': 10, 'replaced_by': 1266, u'atomic_key_1': 9, u'id_1': 1233}]

http://logs.openstack.org/64/521064/3/check/heat-functional-convg-mysql-lbaasv2/f96c950/logs/screen-h-eng.txt.gz?#_Nov_17_15_10_13_618946

This is in the new, atomic make_replacement code from the fix for bug 1727128 (https://review.openstack.org/#/c/514888/) - the part where we update the old resource with it's replacement.

It's not at all clear what other transaction could be taking DB locks in a different order, so we might have to settle for retrying the transaction if this happens.

[Zane Bitter (zaneb)]

The test that failed was heat_integrationtests.functional.test_resource_group.ResourceGroupUpdatePolicyTest.test_resource_group_update_replace, but this is at least failing in a different place from bug 1635504, where the failure occurred when creating resources (likely during the initial stack create/update, not creating a replacement, judging by the log snippet that's there).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/521170

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/521170
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=e919b723bc6f6604d3165d143de51715fbf057e7
Submitter: Zuul
Branch: master

commit e919b723bc6f6604d3165d143de51715fbf057e7
Author: Zane Bitter <email address hidden>
Date: Fri Nov 17 14:14:32 2017 -0500

    Retry on DB deadlock when updating resource

    Updating a row in the DB using UPDATE...WHERE... seems to be prone to
    deadlocks. We already fixed added retries for updates of the Stack table
    (bug 1578615); do the same with the Resource table.

    Change-Id: I3928f1acc3c354711a27523215784a8cf8767581
    Closes-Bug: #1732969

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 10.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/628413

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/628413
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=da8972d5420054065b62e100e3cee33a52a491c3
Submitter: Zuul
Branch: stable/pike

commit da8972d5420054065b62e100e3cee33a52a491c3
Author: Zane Bitter <email address hidden>
Date: Fri Nov 17 14:14:32 2017 -0500

    Retry on DB deadlock when updating resource

    Updating a row in the DB using UPDATE...WHERE... seems to be prone to
    deadlocks. We already fixed added retries for updates of the Stack table
    (bug 1578615); do the same with the Resource table.

    Depends-On: https://review.openstack.org/#/c/622853/
    Change-Id: I3928f1acc3c354711a27523215784a8cf8767581
    Closes-Bug: #1732969
    (cherry picked from commit e919b723bc6f6604d3165d143de51715fbf057e7)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 9.0.7 release.