OpenStack Image Registry and Delivery Service (Glance)

glance db migrations deadlock against precise mysql server

Reported by Adam Gandelman on 2012-02-15
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Glance
Low
Adam Gandelman
glance (Ubuntu)
High
Adam Gandelman

Bug Description

Running 'glance db_sync' against an external mysql server (mysql-server-5.5 5.5.17-4ubuntu6) glance appears to deadlock in 003_add_disk_format.py. A little bit of tracing reveals the call to 'disk_format.create(images)' on line 111 never returns. Meanwhile, 'show processlist' shows:

| glance | 10.252.28.238:60409 | glance | Sleep | 26 | | NULL |
| 5837 | glance | 10.252.28.238:60410 | glance | Query | 26 | Waiting for table metadata lock | ALTER TABLE images ADD disk_format VARCHAR(20) |

I've confirmed that pointing at an oneiric mysql server (mysql-server-5.1 5.1.58-1ubuntu1) does not run into the same problems. Its probably worth noting that mysql 5.5 ships with a different configuration, most notably enabling InnoDB by default, tho I could not reproduce on oneiric using InnoDB.

FWIW, I've also found that this only becomes a problem if I've deployed everything once, automated using Juju, then manually 'drop database glance ; create database glance;' and attempt to rerun the migrations. After its been dropped and recreated, it deadlocks 100% of the time.

Changed in glance (Ubuntu):
assignee: nobody → Adam Gandelman (gandelman-a)
importance: Undecided → High
status: New → In Progress
summary: - glance db migrations fail against precise mysql server
+ glance db migrations deadlock against precise mysql server

Fix proposed to branch: master
Review: https://review.openstack.org/4176

Changed in glance:
assignee: nobody → Adam Gandelman (gandelman-a)
status: New → In Progress

Reviewed: https://review.openstack.org/4176
Committed: http://github.com/openstack/glance/commit/b90575d8e9b8001b1dc7ae5cfb4cc77e74151b69
Submitter: Jenkins
Branch: master

commit b90575d8e9b8001b1dc7ae5cfb4cc77e74151b69
Author: Adam Gandelman <email address hidden>
Date: Tue Feb 14 17:01:59 2012 -0800

    003_add_disk_format.py: Avoid deadlock in upgrade

    The connection used to query existing images needs to be closed
    earlier to avoid deadlocking later calls thru sqlalchemy.

    Fixes bug 932466

    Update: Update my email in Authors

    Update: Revert Authors email to original, add git commit email
            to .mailamp.

    Change-Id: Ic248fdfe3933437928f0b393d8cde993b96bf2cb

Changed in glance:
status: In Progress → Fix Committed
Jay Pipes (jaypipes) on 2012-02-15
Changed in glance:
milestone: none → essex-4
importance: Undecided → Low
Changed in glance (Ubuntu):
status: In Progress → Fix Released
Thierry Carrez (ttx) on 2012-02-29
Changed in glance:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in glance:
milestone: essex-4 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers