test_purge.PurgeTest.test_purge fails with db deadlock intermittently

Bug #1721695 reported by Rabi Mishra
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Undecided
Zane Bitter

Bug Description

http://logs.openstack.org/50/499550/8/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2-py35-ubuntu-xenial/76a4f91/console.html#_2017-10-06_05_04_04_392369
2017-10-06 05:04:04.352557 | 2017-10-06 05:04:04.352 | Captured traceback:
2017-10-06 05:04:04.355486 | 2017-10-06 05:04:04.354 | ~~~~~~~~~~~~~~~~~~~
2017-10-06 05:04:04.358367 | 2017-10-06 05:04:04.357 | b'Traceback (most recent call last):'
2017-10-06 05:04:04.361009 | 2017-10-06 05:04:04.360 | b' File "/opt/stack/new/heat/heat_integrationtests/functional/test_purge.py", line 37, in test_purge'
2017-10-06 05:04:04.364191 | 2017-10-06 05:04:04.363 | b' processutils.execute(cmd, shell=True)'
2017-10-06 05:04:04.366847 | 2017-10-06 05:04:04.366 | b' File "/usr/local/lib/python3.5/dist-packages/oslo_concurrency/processutils.py", line 419, in execute'
2017-10-06 05:04:04.369504 | 2017-10-06 05:04:04.369 | b' cmd=sanitized_cmd)'
2017-10-06 05:04:04.371776 | 2017-10-06 05:04:04.371 | b'oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.'
2017-10-06 05:04:04.374840 | 2017-10-06 05:04:04.374 | b'Command: heat-manage purge_deleted 0'
2017-10-06 05:04:04.377388 | 2017-10-06 05:04:04.376 | b'Exit code: 1'
.....

2017-10-06 05:04:04.447649 | 2017-10-06 05:04:04.446 | b'Stderr: "ERROR: (pymysql.err.InternalError) (1213, \'Deadlock found when trying to get lock; try restarting transaction\') [SQL: \'DELETE FROM resource_data WHERE resource_data.resource_id IN (SELECT resource.id \\\\nFROM resource \\\\nWHERE resource.stack_id IN (%(stack_id_1)s, %(stack_id_2)s, %(stack_id_3)s, %(stack_id_4)s, %(stack_id_5)s, %(stack_id_6)s, %(stack_id_7)s, %(stack_id_8)s, %(stack_id_9)s, %(stack_id_10)s, %(stack_id_11)s, %(stack_id_12)s, %(stack_id_13)s, %(stack_id_14)s, %(stack_id_15)s, %(stack_id_16)s, %(stack_id_17)s, %(stack_id_18)s, %(stack_id_19)s, %(stack_id_20)s))\'] [parameters: {\'stack_id_15\': \'818b2f7a-9049-4a9c-8cee-c9d14555adcd\', \'stack_id_13\': \'80e93a83-ecbe-4faa-964d-a4277b453a30\', \'stack_id_4\': \'7757a722-ef70-40bf-b409-a036ac6cca96\', \'stack_id_19\': \'872e9480-7818-420f-ba3b-0740c08ae521\', \'stack_id_17\': \'841d0222-17ef-4384-8e57-ab5f8725443f\', \'stack_id_5\': \'776d9240-8d83-4dc5-a63f-c67ebca20896\', \'stack_id_14\': \'80ea4aa1-fe4b-41e9-aa49-9652480c1c44\', \'stack_id_1\': \'74dc2488-fe42-4e0e-8fff-616c03b56545\', \'stack_id_16\': \'81f94260-cedb-48c1-8cc4-0902cb0b8566\', \'stack_id_12\': \'7fdc3e19-524

Tags: gate-failure
Zane Bitter (zaneb)
tags: added: gate-failure
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/565531

Changed in heat:
assignee: nobody → Zane Bitter (zaneb)
status: New → In Progress
Rico Lin (rico-lin)
Changed in heat:
milestone: none → no-priority-tag-bugs
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/565531
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=c06a0e469bc611b0be72d10d6335b93b97c5afbb
Submitter: Zuul
Branch: master

commit c06a0e469bc611b0be72d10d6335b93b97c5afbb
Author: Zane Bitter <email address hidden>
Date: Tue May 1 12:11:33 2018 -0400

    Retry on deadlock in purge_deleted

    Change-Id: I6dc7429b2564ab07dd66bf0cf0c8657ac1928508
    Closes-Bug: #1721695

Changed in heat:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 11.0.0.0b2

This issue was fixed in the openstack/heat 11.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.