glance db migrations deadlock against precise mysql server

Bug #932466 reported by Adam Gandelman
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Glance
Fix Released
Low
Adam Gandelman
glance (Ubuntu)
Fix Released
High
Adam Gandelman

Bug Description

Running 'glance db_sync' against an external mysql server (mysql-server-5.5 5.5.17-4ubuntu6) glance appears to deadlock in 003_add_disk_format.py. A little bit of tracing reveals the call to 'disk_format.create(images)' on line 111 never returns. Meanwhile, 'show processlist' shows:

| glance | 10.252.28.238:60409 | glance | Sleep | 26 | | NULL |
| 5837 | glance | 10.252.28.238:60410 | glance | Query | 26 | Waiting for table metadata lock | ALTER TABLE images ADD disk_format VARCHAR(20) |

I've confirmed that pointing at an oneiric mysql server (mysql-server-5.1 5.1.58-1ubuntu1) does not run into the same problems. Its probably worth noting that mysql 5.5 ships with a different configuration, most notably enabling InnoDB by default, tho I could not reproduce on oneiric using InnoDB.

FWIW, I've also found that this only becomes a problem if I've deployed everything once, automated using Juju, then manually 'drop database glance ; create database glance;' and attempt to rerun the migrations. After its been dropped and recreated, it deadlocks 100% of the time.

Changed in glance (Ubuntu):
assignee: nobody → Adam Gandelman (gandelman-a)
importance: Undecided → High
status: New → In Progress
summary: - glance db migrations fail against precise mysql server
+ glance db migrations deadlock against precise mysql server
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/4176

Changed in glance:
assignee: nobody → Adam Gandelman (gandelman-a)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/4176
Committed: http://github.com/openstack/glance/commit/b90575d8e9b8001b1dc7ae5cfb4cc77e74151b69
Submitter: Jenkins
Branch: master

commit b90575d8e9b8001b1dc7ae5cfb4cc77e74151b69
Author: Adam Gandelman <email address hidden>
Date: Tue Feb 14 17:01:59 2012 -0800

    003_add_disk_format.py: Avoid deadlock in upgrade

    The connection used to query existing images needs to be closed
    earlier to avoid deadlocking later calls thru sqlalchemy.

    Fixes bug 932466

    Update: Update my email in Authors

    Update: Revert Authors email to original, add git commit email
            to .mailamp.

    Change-Id: Ic248fdfe3933437928f0b393d8cde993b96bf2cb

Changed in glance:
status: In Progress → Fix Committed
Jay Pipes (jaypipes)
Changed in glance:
milestone: none → essex-4
importance: Undecided → Low
Changed in glance (Ubuntu):
status: In Progress → Fix Released
Thierry Carrez (ttx)
Changed in glance:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in glance:
milestone: essex-4 → 2012.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.