Glance's auto-recovery of db connections is incompatible with newer sqlalchemy

[Adam Gandelman]

Commit https://review.openstack.org/#change,5889 reverted the use of an event listener in favor or wrapping certain sqlalchemy functions in a function that would catch errors and retry at an interval. This appears to have worked fine for sqlalchemy 0.6.8 (oneiric), however, the assumptions that are being made about the internals of sqlalchemy are no longer valid when using sqlalchemy 0.7.4 (precise). The fix is no longer valid for precise/sqlalchemy 0.7.4, to confirm, on precise:

glance -f index
service mysql restart
glance -f index

...results in the original Bug #954971 again. It appears, internally, sqlalchemy is now calling conn.execute() where it was calling session.execute(). The exceptions that now bubble up are those raised by the backend dialect in-use, and not SQLA proper.

I propose Glance re-instates the event listener for the catching this specific error, given:

- it works well
- its portable across both new and old versions of SQLA
- puts most of the work on the library for assuring consistent database connection over time
- has already been implemented across most other projects to fix this same issue

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/5938

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/5938
Committed: http://github.com/openstack/glance/commit/0dc093b98faf8e256ddc48f9dafb4d061662f0e3
Submitter: Jenkins
Branch: master

commit 0dc093b98faf8e256ddc48f9dafb4d061662f0e3
Author: Adam Gandelman <email address hidden>
Date: Wed Mar 28 17:08:13 2012 -0700

    Add MySQLPingListener() back

    The use of wrap_db() to catch low-level problems in the connection
    pool appears to be incompatible with new versions of sqlalchemy
    (0.7.4/precise). This re-instates the use of a event listener to
    catch the error of 'mysql has gone away'. Keeps the wrapping
    logic in place for use before the connection pool has been constructed,
    specifically to allow glance to retry its initial database connection on
    service startup.

    Fixes bug 967887

    Change-Id: Ia732377d3404104b8acf7fe4ada164ec7ecf705f

[Adam Gandelman (gandelman-a)]

Bug #954971 was marked as 'Low' upstream, so I'm not sure this is going to be fixed in the Essex release. We should consider carrying the fix in Precise.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/6167

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/6167
Committed: http://github.com/openstack/glance/commit/f5ed968136311da032989eab71a5b69e99d15add
Submitter: Jenkins
Branch: milestone-proposed

commit f5ed968136311da032989eab71a5b69e99d15add
Author: Adam Gandelman <email address hidden>
Date: Wed Mar 28 17:08:13 2012 -0700

    Add MySQLPingListener() back

    The use of wrap_db() to catch low-level problems in the connection
    pool appears to be incompatible with new versions of sqlalchemy
    (0.7.4/precise). This re-instates the use of a event listener to
    catch the error of 'mysql has gone away'. Keeps the wrapping
    logic in place for use before the connection pool has been constructed,
    specifically to allow glance to retry its initial database connection on
    service startup.

    Fixes bug 967887

    Change-Id: Ia732377d3404104b8acf7fe4ada164ec7ecf705f