HTTP 500 on update_server_metadata (DB deadlock)

[Simon Pasquier]

Tempest console
=============

2013-12-18 10:32:34.092 | ======================================================================
2013-12-18 10:32:34.094 | FAIL: tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_update_server_metadata[gate]
2013-12-18 10:32:34.095 | tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_update_server_metadata[gate]
2013-12-18 10:32:34.096 | ----------------------------------------------------------------------

http://logs.openstack.org/32/62832/1/check/check-tempest-dsvm-full/2e0941c/console.html#_2013-12-18_10_32_34_092

Nova API log with deadlock DB message
===============================

http://logs.openstack.org/22/46722/2/check/gate-tempest-devstack-vm-full/fd3664e/logs/screen-n-api.txt.gz#_2013-09-16_13_51_06_281

[Joe Gordon (jogo)]

logstash query: message:"Deadlock found when trying to get lock" AND filename:"logs/screen-n-api.txt"

This has only been seen once in the last 2 weeks

[Joe Gordon (jogo)]

make that three times

[Russell Bryant (russellb)]

In the traceback we see:

  2013-09-16 13:51:06.281 22659 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 4504, in instance_metadata_delete

and in nova.db.sqlalchemy.api we have a decorator, _retry_on_deadlock() that is supposed to catch these errors and retry. We have the decorator set on this function, so I'm honestly not sure why we're seeing this.

[Russell Bryant (russellb)]

Further inspection of the backtrace confirms that the retry_on_deadlock decorator is not actually there on whatever version is being tested here, though it is there in master

[Russell Bryant (russellb)]

a newer backtrace shows a different failure for a method without the decorator yet:

http://logs.openstack.org/32/62832/1/check/check-tempest-dsvm-full/2e0941c/logs/screen-n-api.txt.gz?level=TRACE

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/62952

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/62952
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3e64cac1a1dad9fca6d1ab4a9d9913560130951d
Submitter: Jenkins
Branch: master

commit 3e64cac1a1dad9fca6d1ab4a9d9913560130951d
Author: Russell Bryant <email address hidden>
Date: Wed Dec 18 13:39:57 2013 -0500

    Retry on deadlock in instance_metadata_update

    THe instance_metadata_update() method in the sqlalchemy db API may hit a
    DBDeadlock, as shown in the bug report. Apply the necessary decorator
    that will have the method retry in that case.

    Change-Id: Ice3f13857ba8f4ee1d1d2dc06cef293e6d17daca
    Closes-bug: #1262154