DB2 disconnect not handled pessimistically

Bug #1231657 reported by David Peraza on 2013-09-26
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Glance
Undecided
David Peraza

Bug Description

Only MySQL server disconnect is being handled pessimistically by adding a pool checkout listener to the sqlalchemy engine.
Since DB2 connection pool is not handled pessimistically, we get the following error after a temporarily disconnect of DB2 server that is already up and running:

2013-09-15 13:49:48.274 1013 TRACE nova.api.openstack OperationalError: (OperationalError) ibm_db_dbi::OperationalError: [IBM][CLI Driver] SQL30081N A communication error has been detected. Communication protocol being used: "TCP/IP". Communication API being used: "SOCKETS". Location where the error was detected: "172.17.35.52". Communication function detecting the error: "send". Protocol specific error code(s): "110", "*", "*". SQLSTATE=08001 SQLCODE=-30081

To avoid this error DB2 needs to be included in the logic to handle the connections pessimistically.

To recreate in glance:
Run glance index and make sure it is working
Restart DB2 server
Run glance index again, you will see error above.

Tags: db Edit Tag help

Fix proposed to branch: master
Review: https://review.openstack.org/48523

Changed in glance:
assignee: nobody → David Peraza (dperaza)
status: New → In Progress
Brant Knudson (blk-u) on 2013-09-26
Changed in keystone:
assignee: nobody → Brant Knudson (blk-u)
Dolph Mathews (dolph) on 2013-09-26
Changed in keystone:
status: New → Triaged
importance: Undecided → Medium
Matt Riedemann (mriedem) on 2013-09-27
Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)

Fix proposed to branch: master
Review: https://review.openstack.org/48733

Changed in oslo:
assignee: nobody → David Peraza (dperaza)
status: New → In Progress
Jay Bryant (jsbryant) on 2013-09-27
Changed in cinder:
assignee: nobody → Jay Bryant (jsbryant)
status: New → Confirmed
Matt Riedemann (mriedem) on 2013-09-28
tags: added: db
Changed in nova:
status: New → Triaged
Julien Danjou (jdanjou) on 2013-09-30
Changed in ceilometer:
status: New → Triaged
importance: Undecided → Medium
Dolph Mathews (dolph) on 2013-09-30
tags: added: havana-backport-potential
James Carey (jecarey) on 2013-10-01
Changed in neutron:
assignee: nobody → James Carey (jecarey)

Related fix proposed to branch: master
Review: https://review.openstack.org/49271

Changed in keystone:
status: Triaged → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/49790

Changed in nova:
status: Triaged → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/49854

Changed in cinder:
status: Confirmed → In Progress
JunJie Nan (nanjj) on 2013-10-08
Changed in heat:
assignee: nobody → Jun Jie Nan (nanjj)
Thierry Carrez (ttx) on 2013-10-08
Changed in keystone:
milestone: none → havana-rc2
tags: removed: havana-backport-potential

Reviewed: https://review.openstack.org/49272
Committed: http://github.com/openstack/keystone/commit/f18dbd7e74303b7e0af97439ed935038e2231feb
Submitter: Jenkins
Branch: master

commit f18dbd7e74303b7e0af97439ed935038e2231feb
Author: Brant Knudson <email address hidden>
Date: Thu Oct 3 15:35:19 2013 -0500

    Handle DB2 disconnect

    If the DB2 server was restarted, database connections would fail,
    requiring a restart of the Keystone server.

    This change adds an on-checkout handler for DB2 similar to the
    MySQL handler.

    Change-Id: Ie53d9606edac408401961fc6526bb7c99baa836a
    Closes-Bug: #1231657

Changed in keystone:
status: In Progress → Fix Committed

Fix proposed to branch: master
Review: https://review.openstack.org/50552

Changed in heat:
status: New → In Progress
Thierry Carrez (ttx) on 2013-10-09
Changed in keystone:
status: Fix Committed → Fix Released
Liang Chen (cbjchen) on 2013-10-11
Changed in oslo:
status: In Progress → Fix Committed
status: Fix Committed → In Progress
Thierry Carrez (ttx) on 2013-10-17
Changed in keystone:
milestone: havana-rc2 → 2013.2
Changed in neutron:
status: New → Triaged
importance: Undecided → Medium

Reviewed: https://review.openstack.org/48733
Committed: http://github.com/openstack/oslo-incubator/commit/bb4d7a2534f6087d828e22784dc263522d309592
Submitter: Jenkins
Branch: master

commit bb4d7a2534f6087d828e22784dc263522d309592
Author: David Peraza <email address hidden>
Date: Wed Dec 4 09:15:28 2013 -0600

    Enables db2 server disconnects to be handled pessimistically

    A checkout listener can be added to sqlalchemy to be able to
    handle database disconnect pessimistically and avoid an error in
    the event of a DB restart or simply a temp connection issue due
    to network glitches. At the moment only mysql is enabled with
    this feature. This patch enables db2 as well.

    Partial-Bug: #1231657
    Change-Id: I5563360b43c4b648f5b035099b0e1d2e71d32503

Fix proposed to branch: master
Review: https://review.openstack.org/61602

Changed in ceilometer:
assignee: nobody → David Peraza (dperaza)
status: Triaged → In Progress

Reviewed: https://review.openstack.org/50552
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=cb0a2a059132721e8483e1090766611009deafad
Submitter: Jenkins
Branch: master

commit cb0a2a059132721e8483e1090766611009deafad
Author: JUN JIE NAN <email address hidden>
Date: Thu Dec 12 15:23:46 2013 +0800

    Enables db2 server disconnects to be handled pessimistically

    A checkout listener can be added to sqlalchemy to be able to handle
    database disconnect pessimistically and avoid an error in the event of
    a DB restart or simply a temp connection issue due to network
    glitches. At the moment only mysql is enabled with this feature. This
    patch enables db2 as well.

    Sync oslo db.sqlalchemy from commit:
    bb4d7a2534f6087d828e22784dc263522d309592

    Partial-Bug: #1231657

    Change-Id: Ib89de07f004975528baffdfd19aae689b089a324

Reviewed: https://review.openstack.org/61602
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=2c199a0388e2f456b9e28057c523c51163189d0d
Submitter: Jenkins
Branch: master

commit 2c199a0388e2f456b9e28057c523c51163189d0d
Author: David Peraza <email address hidden>
Date: Wed Dec 11 17:00:10 2013 -0600

    Oslo sync to recover from db2 server disconnects

    A checkout listener can be added to sqlalchemy to be able to
    handle database disconnect pessimistically and avoid an error in
    the event of a DB restart or simply a temp connection issue due
    to network glitches. At the moment only mysql is enabled with
    this feature. This patch enables db2 as well.

    Oslo version in change I5563360b43c4b648f5b035099b0e1d2e71d32503

    Partial-Bug: #1231657
    Change-Id: Idf8b3b11a06790813fed218bdbaf130554f80fa3

JunJie Nan (nanjj) on 2014-01-09
Changed in heat:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/65513
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=922f4a28c9498ee450c095868cce587cddcf033a
Submitter: Jenkins
Branch: master

commit 922f4a28c9498ee450c095868cce587cddcf033a
Author: David Peraza <email address hidden>
Date: Wed Jan 8 12:45:13 2014 -0600

    Oslo sync to recover from db2 server disconnects

    A checkout listener can be added to sqlalchemy to be able to
    handle database disconnect pessimistically and avoid an error in
    the event of a DB restart or simply a temp connection issue due
    to network glitches. At the moment only mysql is enabled with
    this feature. This patch enables db2 as well.

    Oslo version in change I5563360b43c4b648f5b035099b0e1d2e71d32503

    Partial-Bug: #1231657
    Change-Id: I218ef9836f2e0f135897007d2e53354eb1248422

Thierry Carrez (ttx) on 2014-01-22
Changed in heat:
milestone: none → icehouse-2
status: Fix Committed → Fix Released
Jay Bryant (jsbryant) on 2014-02-06
Changed in cinder:
importance: Undecided → Medium
gordon chung (chungg) wrote :

fix merged in ceilometer: https://review.openstack.org/#/c/61602/

Changed in ceilometer:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2014-03-27
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-04-17
Changed in heat:
milestone: icehouse-2 → 2014.1
Jay Bryant (jsbryant) wrote :

The fix for Cinder has been pulled in with our latest Oslo DB Sync: https://review.openstack.org/#/c/77125/

Changed in cinder:
milestone: none → juno-2
status: In Progress → Fix Committed
Changed in oslo:
status: In Progress → Fix Committed
Changed in oslo:
importance: Undecided → Medium
Thierry Carrez (ttx) on 2014-07-29
Changed in oslo:
milestone: none → juno-2
status: Fix Committed → Fix Released
Changed in cinder:
status: Fix Committed → Fix Released
Sean Dague (sdague) on 2014-09-17
no longer affects: keystone
no longer affects: heat
no longer affects: oslo-incubator
no longer affects: cinder
no longer affects: ceilometer
no longer affects: nova
Matt Riedemann (mriedem) wrote :

This was fixed in oslo-incubator:

https://review.openstack.org/#/c/48733/

And that's in oslo.db now and nova is using oslo so yeah, fixed for nova.

no longer affects: neutron
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers