HTTP 500 on delete_server_metadata_item (DB deadlock)

[Attila Fazekas]

2013-09-16 14:03:40.584 | ======================================================================
2013-09-16 14:03:40.585 | FAIL: tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_delete_server_metadata_item[gate]
2013-09-16 14:03:40.585 | tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_delete_server_metadata_item[gate]
2013-09-16 14:03:40.585 | ----------------------------------------------------------------------
2013-09-16 14:03:40.585 | _StringException: Empty attachments:
2013-09-16 14:03:40.585 | stderr
2013-09-16 14:03:40.585 | stdout
2013-09-16 14:03:40.586 |
2013-09-16 14:03:40.586 | pythonlogging:'': {{{
2013-09-16 14:03:40.586 | 2013-09-16 13:51:05,920 Request: PUT http://127.0.0.1:8774/v2/f89374309acf4ae59b87735dffc6f395/servers/d927cfc5-a27d-4fc5-aea6-04407701a8c9/metadata
2013-09-16 14:03:40.586 | 2013-09-16 13:51:06,137 Response Status: 200
2013-09-16 14:03:40.586 | 2013-09-16 13:51:06,137 Nova request id: req-17714898-35be-482d-ab15-868ebc4b7e72
2013-09-16 14:03:40.586 | 2013-09-16 13:51:06,137 Request: DELETE http://127.0.0.1:8774/v2/f89374309acf4ae59b87735dffc6f395/servers/d927cfc5-a27d-4fc5-aea6-04407701a8c9/metadata/key1
2013-09-16 14:03:40.587 | 2013-09-16 13:51:06,289 Response Status: 500
2013-09-16 14:03:40.587 | 2013-09-16 13:51:06,290 Nova request id: req-025fd39f-66a5-426e-bfac-fa0d3764d574
2013-09-16 14:03:40.587 | }}}
2013-09-16 14:03:40.587 |
2013-09-16 14:03:40.587 | Traceback (most recent call last):
2013-09-16 14:03:40.587 | File "tempest/api/compute/servers/test_server_metadata.py", line 141, in test_delete_server_metadata_item
2013-09-16 14:03:40.587 | 'key1')
2013-09-16 14:03:40.588 | File "tempest/services/compute/json/servers_client.py", line 267, in delete_server_metadata_item
2013-09-16 14:03:40.588 | (str(server_id), key))
2013-09-16 14:03:40.588 | File "tempest/common/rest_client.py", line 266, in delete
2013-09-16 14:03:40.588 | return self.request('DELETE', url, headers)
2013-09-16 14:03:40.588 | File "tempest/common/rest_client.py", line 394, in request
2013-09-16 14:03:40.588 | resp, resp_body)
2013-09-16 14:03:40.588 | File "tempest/common/rest_client.py", line 480, in _error_checker
2013-09-16 14:03:40.589 | raise exceptions.ComputeFault(message)
2013-09-16 14:03:40.589 | ComputeFault: Got compute fault
2013-09-16 14:03:40.589 | Details: The server has either erred or is incapable of performing the requested operation.

=======
Console
=======
http://logs.openstack.org/22/46722/2/check/gate-tempest-devstack-vm-full/fd3664e/console.html

===================
DB deadlock message
===================
http://logs.openstack.org/22/46722/2/check/gate-tempest-devstack-vm-full/fd3664e/logs/screen-n-api.txt.gz#_2013-09-16_13_51_06_281

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/46964

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/46964
Committed: http://github.com/openstack/nova/commit/efe441fbab46813f00d383f1dbec473f645b2ef6
Submitter: Jenkins
Branch: master

commit efe441fbab46813f00d383f1dbec473f645b2ef6
Author: Russell Bryant <email address hidden>
Date: Tue Sep 17 10:32:28 2013 -0400

    Retry on deadlock in instance_metadata_delete

    Add the retry_on_deadlock decorator to the instance_metadata_delete
    function in the sqlalchemy db API. A tempest test failure shows that it
    would benefit from it.

    Change-Id: I39547fbbbabc310b78493df0dc055bc320995770
    Closes-bug: #1226076