OpenStack Image Registry and Delivery Service (Glance)

Glance's auto-recovery of db connections is incompatible with newer sqlalchemy

Reported by Adam Gandelman on 2012-03-29
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
High
Adam Gandelman
glance (Ubuntu)
Undecided
Unassigned

Bug Description

Commit https://review.openstack.org/#change,5889 reverted the use of an event listener in favor or wrapping certain sqlalchemy functions in a function that would catch errors and retry at an interval. This appears to have worked fine for sqlalchemy 0.6.8 (oneiric), however, the assumptions that are being made about the internals of sqlalchemy are no longer valid when using sqlalchemy 0.7.4 (precise). The fix is no longer valid for precise/sqlalchemy 0.7.4, to confirm, on precise:

glance -f index
service mysql restart
glance -f index

...results in the original Bug #954971 again. It appears, internally, sqlalchemy is now calling conn.execute() where it was calling session.execute(). The exceptions that now bubble up are those raised by the backend dialect in-use, and not SQLA proper.

I propose Glance re-instates the event listener for the catching this specific error, given:

- it works well
- its portable across both new and old versions of SQLA
- puts most of the work on the library for assuring consistent database connection over time
- has already been implemented across most other projects to fix this same issue

Fix proposed to branch: master
Review: https://review.openstack.org/5938

Changed in glance:
assignee: nobody → Adam Gandelman (gandelman-a)
status: New → In Progress

Reviewed: https://review.openstack.org/5938
Committed: http://github.com/openstack/glance/commit/0dc093b98faf8e256ddc48f9dafb4d061662f0e3
Submitter: Jenkins
Branch: master

commit 0dc093b98faf8e256ddc48f9dafb4d061662f0e3
Author: Adam Gandelman <email address hidden>
Date: Wed Mar 28 17:08:13 2012 -0700

    Add MySQLPingListener() back

    The use of wrap_db() to catch low-level problems in the connection
    pool appears to be incompatible with new versions of sqlalchemy
    (0.7.4/precise). This re-instates the use of a event listener to
    catch the error of 'mysql has gone away'. Keeps the wrapping
    logic in place for use before the connection pool has been constructed,
    specifically to allow glance to retry its initial database connection on
    service startup.

    Fixes bug 967887

    Change-Id: Ia732377d3404104b8acf7fe4ada164ec7ecf705f

Changed in glance:
status: In Progress → Fix Committed
Adam Gandelman (gandelman-a) wrote :

Bug #954971 was marked as 'Low' upstream, so I'm not sure this is going to be fixed in the Essex release. We should consider carrying the fix in Precise.

Thierry Carrez (ttx) on 2012-04-03
tags: added: essex-rc-potential
Thierry Carrez (ttx) on 2012-04-03
Changed in glance:
importance: Undecided → High
Thierry Carrez (ttx) on 2012-04-03
Changed in glance:
milestone: none → essex-rc3
tags: removed: essex-rc-potential

Reviewed: https://review.openstack.org/6167
Committed: http://github.com/openstack/glance/commit/f5ed968136311da032989eab71a5b69e99d15add
Submitter: Jenkins
Branch: milestone-proposed

commit f5ed968136311da032989eab71a5b69e99d15add
Author: Adam Gandelman <email address hidden>
Date: Wed Mar 28 17:08:13 2012 -0700

    Add MySQLPingListener() back

    The use of wrap_db() to catch low-level problems in the connection
    pool appears to be incompatible with new versions of sqlalchemy
    (0.7.4/precise). This re-instates the use of a event listener to
    catch the error of 'mysql has gone away'. Keeps the wrapping
    logic in place for use before the connection pool has been constructed,
    specifically to allow glance to retry its initial database connection on
    service startup.

    Fixes bug 967887

    Change-Id: Ia732377d3404104b8acf7fe4ada164ec7ecf705f

Changed in glance:
status: Fix Committed → Fix Released
Changed in glance (Ubuntu):
status: New → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in glance:
milestone: essex-rc3 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers