Deadlock found when trying to get lock; try restarting transaction

Bug #1343613 reported by Davanum Srinivas (DIMS)
26
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Pavel Kholkin
tempest
Invalid
Undecided
Unassigned

Bug Description

Example URL:
http://logs.openstack.org/31/107131/1/gate/gate-grenade-dsvm/d019d8e/logs/old/screen-n-api.txt.gz?level=ERROR#_2014-07-17_20_59_37_031

Logstash query(?):
message:"Deadlock found when trying to get lock; try restarting transaction" AND loglevel:"ERROR" AND build_status:"FAILURE"

32 hits in 48 hours.

Tags: db testing
Tracy Jones (tjones-i)
tags: added: db testing
Revision history for this message
Jay Pipes (jaypipes) wrote :

Pavel has a partial fix for the underlying problem here: https://review.openstack.org/#/c/106367/

Basically, the use of SELECT FOR UPDATE can be combined into a single UPDATE statement and reduce (most likely eliminate) the chance of deadlock exposure.

It's a partial fix because the allocate also would need to be fixed.

Changed in nova:
assignee: nobody → Pavel Kholkin (pkholkin)
importance: Undecided → Critical
importance: Critical → Medium
status: New → Triaged
Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/106367
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b8fe7b13cff3170cb1b87c5a26fbc8a8ec9422cb
Submitter: Jenkins
Branch: master

commit b8fe7b13cff3170cb1b87c5a26fbc8a8ec9422cb
Author: pkholkin <email address hidden>
Date: Fri Jul 11 16:54:38 2014 +0400

    Optimize db.floating_ip_deallocate

    'select for update' and 'update' operations
    were combined into a single 'update' operation

    problems with 'select for update' relate to
    spec https://review.openstack.org/#/c/97310/1

    now 'floating_ip_deallocate' function returns
    number of updated rows in db

    Partial-Bug: #1343613

    Change-Id: I60f5a8f3e1541983dea1589783927107c00c5fa4

Revision history for this message
Matthew Treinish (treinish) wrote :

I'm not seeing how this is a bug in tempest, besides the fact that a tempest run triggered it. Can you elaborate this is a bug in tempest?

Changed in tempest:
status: New → Incomplete
Changed in nova:
status: In Progress → Confirmed
Revision history for this message
Joe Gordon (jogo) wrote :

not seeing any hits for this in nova logs

Changed in nova:
status: Confirmed → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → kilo-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: kilo-1 → 2015.1.0
Revision history for this message
Yaroslav Lobankov (ylobankov) wrote :

It looks like this issue is not related to Tempest. As Matthew said, tempest runs just triggered it. So moving the bug status to "Invalid".

Changed in tempest:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.