DB2 disconnect not handled pessimistically

Bug #1231657 reported by David Peraza
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Glance
Undecided
David Peraza

Bug Description

Only MySQL server disconnect is being handled pessimistically by adding a pool checkout listener to the sqlalchemy engine.
Since DB2 connection pool is not handled pessimistically, we get the following error after a temporarily disconnect of DB2 server that is already up and running:

2013-09-15 13:49:48.274 1013 TRACE nova.api.openstack OperationalError: (OperationalError) ibm_db_dbi::OperationalError: [IBM][CLI Driver] SQL30081N A communication error has been detected. Communication protocol being used: "TCP/IP". Communication API being used: "SOCKETS". Location where the error was detected: "172.17.35.52". Communication function detecting the error: "send". Protocol specific error code(s): "110", "*", "*". SQLSTATE=08001 SQLCODE=-30081

To avoid this error DB2 needs to be included in the logic to handle the connections pessimistically.

To recreate in glance:
Run glance index and make sure it is working
Restart DB2 server
Run glance index again, you will see error above.

Tags: db Edit Tag help
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/48523

Changed in glance:
assignee: nobody → David Peraza (dperaza)
status: New → In Progress
Brant Knudson (blk-u)
Changed in keystone:
assignee: nobody → Brant Knudson (blk-u)
Dolph Mathews (dolph)
Changed in keystone:
status: New → Triaged
importance: Undecided → Medium
Matt Riedemann (mriedem)
Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo-incubator (master)

Fix proposed to branch: master
Review: https://review.openstack.org/48733

Changed in oslo:
assignee: nobody → David Peraza (dperaza)
status: New → In Progress
Jay Bryant (jsbryant)
Changed in cinder:
assignee: nobody → Jay Bryant (jsbryant)
status: New → Confirmed
Matt Riedemann (mriedem)
tags: added: db
Changed in nova:
status: New → Triaged
Julien Danjou (jdanjou)
Changed in ceilometer:
status: New → Triaged
importance: Undecided → Medium
Dolph Mathews (dolph)
tags: added: havana-backport-potential
James Carey (jecarey)
Changed in neutron:
assignee: nobody → James Carey (jecarey)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to keystone (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/49270

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/49271

Changed in keystone:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/49272

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to keystone (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/49273

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/49790

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/49854

Changed in cinder:
status: Confirmed → In Progress
JunJie Nan (nanjj)
Changed in heat:
assignee: nobody → Jun Jie Nan (nanjj)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/50455

Thierry Carrez (ttx)
Changed in keystone:
milestone: none → havana-rc2
tags: removed: havana-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/49272
Committed: http://github.com/openstack/keystone/commit/f18dbd7e74303b7e0af97439ed935038e2231feb
Submitter: Jenkins
Branch: master

commit f18dbd7e74303b7e0af97439ed935038e2231feb
Author: Brant Knudson <email address hidden>
Date: Thu Oct 3 15:35:19 2013 -0500

    Handle DB2 disconnect

    If the DB2 server was restarted, database connections would fail,
    requiring a restart of the Keystone server.

    This change adds an on-checkout handler for DB2 similar to the
    MySQL handler.

    Change-Id: Ie53d9606edac408401961fc6526bb7c99baa836a
    Closes-Bug: #1231657

Changed in keystone:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/50552

Changed in heat:
status: New → In Progress
Thierry Carrez (ttx)
Changed in keystone:
status: Fix Committed → Fix Released
Liang Chen (cbjchen)
Changed in oslo:
status: In Progress → Fix Committed
status: Fix Committed → In Progress
Thierry Carrez (ttx)
Changed in keystone:
milestone: havana-rc2 → 2013.2
Changed in neutron:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (master)

Reviewed: https://review.openstack.org/48733
Committed: http://github.com/openstack/oslo-incubator/commit/bb4d7a2534f6087d828e22784dc263522d309592
Submitter: Jenkins
Branch: master

commit bb4d7a2534f6087d828e22784dc263522d309592
Author: David Peraza <email address hidden>
Date: Wed Dec 4 09:15:28 2013 -0600

    Enables db2 server disconnects to be handled pessimistically

    A checkout listener can be added to sqlalchemy to be able to
    handle database disconnect pessimistically and avoid an error in
    the event of a DB restart or simply a temp connection issue due
    to network glitches. At the moment only mysql is enabled with
    this feature. This patch enables db2 as well.

    Partial-Bug: #1231657
    Change-Id: I5563360b43c4b648f5b035099b0e1d2e71d32503

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/61602

Changed in ceilometer:
assignee: nobody → David Peraza (dperaza)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/50552
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=cb0a2a059132721e8483e1090766611009deafad
Submitter: Jenkins
Branch: master

commit cb0a2a059132721e8483e1090766611009deafad
Author: JUN JIE NAN <email address hidden>
Date: Thu Dec 12 15:23:46 2013 +0800

    Enables db2 server disconnects to be handled pessimistically

    A checkout listener can be added to sqlalchemy to be able to handle
    database disconnect pessimistically and avoid an error in the event of
    a DB restart or simply a temp connection issue due to network
    glitches. At the moment only mysql is enabled with this feature. This
    patch enables db2 as well.

    Sync oslo db.sqlalchemy from commit:
    bb4d7a2534f6087d828e22784dc263522d309592

    Partial-Bug: #1231657

    Change-Id: Ib89de07f004975528baffdfd19aae689b089a324

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/61602
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=2c199a0388e2f456b9e28057c523c51163189d0d
Submitter: Jenkins
Branch: master

commit 2c199a0388e2f456b9e28057c523c51163189d0d
Author: David Peraza <email address hidden>
Date: Wed Dec 11 17:00:10 2013 -0600

    Oslo sync to recover from db2 server disconnects

    A checkout listener can be added to sqlalchemy to be able to
    handle database disconnect pessimistically and avoid an error in
    the event of a DB restart or simply a temp connection issue due
    to network glitches. At the moment only mysql is enabled with
    this feature. This patch enables db2 as well.

    Oslo version in change I5563360b43c4b648f5b035099b0e1d2e71d32503

    Partial-Bug: #1231657
    Change-Id: Idf8b3b11a06790813fed218bdbaf130554f80fa3

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/65513

JunJie Nan (nanjj)
Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/65513
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=922f4a28c9498ee450c095868cce587cddcf033a
Submitter: Jenkins
Branch: master

commit 922f4a28c9498ee450c095868cce587cddcf033a
Author: David Peraza <email address hidden>
Date: Wed Jan 8 12:45:13 2014 -0600

    Oslo sync to recover from db2 server disconnects

    A checkout listener can be added to sqlalchemy to be able to
    handle database disconnect pessimistically and avoid an error in
    the event of a DB restart or simply a temp connection issue due
    to network glitches. At the moment only mysql is enabled with
    this feature. This patch enables db2 as well.

    Oslo version in change I5563360b43c4b648f5b035099b0e1d2e71d32503

    Partial-Bug: #1231657
    Change-Id: I218ef9836f2e0f135897007d2e53354eb1248422

Thierry Carrez (ttx)
Changed in heat:
milestone: none → icehouse-2
status: Fix Committed → Fix Released
Jay Bryant (jsbryant)
Changed in cinder:
importance: Undecided → Medium
Revision history for this message
gordon chung (chungg) wrote :

fix merged in ceilometer: https://review.openstack.org/#/c/61602/

Changed in ceilometer:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in heat:
milestone: icehouse-2 → 2014.1
Revision history for this message
Jay Bryant (jsbryant) wrote :

The fix for Cinder has been pulled in with our latest Oslo DB Sync: https://review.openstack.org/#/c/77125/

Changed in cinder:
milestone: none → juno-2
status: In Progress → Fix Committed
Changed in oslo:
status: In Progress → Fix Committed
Changed in oslo:
importance: Undecided → Medium
Thierry Carrez (ttx)
Changed in oslo:
milestone: none → juno-2
status: Fix Committed → Fix Released
Changed in cinder:
status: Fix Committed → Fix Released
Sean Dague (sdague)
no longer affects: keystone
no longer affects: heat
no longer affects: oslo-incubator
no longer affects: cinder
no longer affects: ceilometer
no longer affects: nova
Revision history for this message
Matt Riedemann (mriedem) wrote :

This was fixed in oslo-incubator:

https://review.openstack.org/#/c/48733/

And that's in oslo.db now and nova is using oslo so yeah, fixed for nova.

no longer affects: neutron
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers