Glance

DB2 disconnect not handled pessimistically

Bug #1231657 reported by David Peraza
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Glance
In Progress
Undecided
David Peraza

Bug Description

Only MySQL server disconnect is being handled pessimistically by adding a pool checkout listener to the sqlalchemy engine.
Since DB2 connection pool is not handled pessimistically, we get the following error after a temporarily disconnect of DB2 server that is already up and running:

2013-09-15 13:49:48.274 1013 TRACE nova.api.openstack OperationalError: (OperationalError) ibm_db_dbi::OperationalError: [IBM][CLI Driver] SQL30081N A communication error has been detected. Communication protocol being used: "TCP/IP". Communication API being used: "SOCKETS". Location where the error was detected: "172.17.35.52". Communication function detecting the error: "send". Protocol specific error code(s): "110", "*", "*". SQLSTATE=08001 SQLCODE=-30081

To avoid this error DB2 needs to be included in the logic to handle the connections pessimistically.

To recreate in glance:
Run glance index and make sure it is working
Restart DB2 server
Run glance index again, you will see error above.

Tags: db
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/48523

Changed in glance:
assignee: nobody → David Peraza (dperaza)
status: New → In Progress
Brant Knudson (blk-u)
Changed in keystone:
assignee: nobody → Brant Knudson (blk-u)
Dolph Mathews (dolph)
Changed in keystone:
status: New → Triaged
importance: Undecided → Medium
Matt Riedemann (mriedem)
Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
Revision history for this message
Matt Riedemann (mriedem) wrote :

The code is in oslo-incubator here:

https://github.com/openstack/oslo-incubator/blob/2013.2.b3/openstack/common/db/sqlalchemy/session.py#L604

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo-incubator (master)

Fix proposed to branch: master
Review: https://review.openstack.org/48733

Changed in oslo:
assignee: nobody → David Peraza (dperaza)
status: New → In Progress
Jay Bryant (jsbryant)
Changed in cinder:
assignee: nobody → Jay Bryant (jsbryant)
status: New → Confirmed
Matt Riedemann (mriedem)
tags: added: db
Changed in nova:
status: New → Triaged
Julien Danjou (jdanjou)
Changed in ceilometer:
status: New → Triaged
importance: Undecided → Medium
Dolph Mathews (dolph)
tags: added: havana-backport-potential
James Carey (jecarey)
Changed in neutron:
assignee: nobody → James Carey (jecarey)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to keystone (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/49270

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/49271

Changed in keystone:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/49272

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to keystone (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/49273

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/49790

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/49854

Changed in cinder:
status: Confirmed → In Progress
JunJie Nan (nanjj)
Changed in heat:
assignee: nobody → Jun Jie Nan (nanjj)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/50455

Thierry Carrez (ttx)
Changed in keystone:
milestone: none → havana-rc2
tags: removed: havana-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/49272
Committed: http://github.com/openstack/keystone/commit/f18dbd7e74303b7e0af97439ed935038e2231feb
Submitter: Jenkins
Branch: master

commit f18dbd7e74303b7e0af97439ed935038e2231feb
Author: Brant Knudson <email address hidden>
Date: Thu Oct 3 15:35:19 2013 -0500

    Handle DB2 disconnect

    If the DB2 server was restarted, database connections would fail,
    requiring a restart of the Keystone server.

    This change adds an on-checkout handler for DB2 similar to the
    MySQL handler.

    Change-Id: Ie53d9606edac408401961fc6526bb7c99baa836a
    Closes-Bug: #1231657

Changed in keystone:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/50552

Changed in heat:
status: New → In Progress
Thierry Carrez (ttx)
Changed in keystone:
status: Fix Committed → Fix Released
Liang Chen (cbjchen)
Changed in oslo:
status: In Progress → Fix Committed
status: Fix Committed → In Progress
Thierry Carrez (ttx)
Changed in keystone:
milestone: havana-rc2 → 2013.2
Mark McClain (markmcclain)
Changed in neutron:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (master)

Reviewed: https://review.openstack.org/48733
Committed: http://github.com/openstack/oslo-incubator/commit/bb4d7a2534f6087d828e22784dc263522d309592
Submitter: Jenkins
Branch: master

commit bb4d7a2534f6087d828e22784dc263522d309592
Author: David Peraza <email address hidden>
Date: Wed Dec 4 09:15:28 2013 -0600

    Enables db2 server disconnects to be handled pessimistically

    A checkout listener can be added to sqlalchemy to be able to
    handle database disconnect pessimistically and avoid an error in
    the event of a DB restart or simply a temp connection issue due
    to network glitches. At the moment only mysql is enabled with
    this feature. This patch enables db2 as well.

    Partial-Bug: #1231657
    Change-Id: I5563360b43c4b648f5b035099b0e1d2e71d32503

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/61602

Changed in ceilometer:
assignee: nobody → David Peraza (dperaza)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/50552
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=cb0a2a059132721e8483e1090766611009deafad
Submitter: Jenkins
Branch: master

commit cb0a2a059132721e8483e1090766611009deafad
Author: JUN JIE NAN <email address hidden>
Date: Thu Dec 12 15:23:46 2013 +0800

    Enables db2 server disconnects to be handled pessimistically

    A checkout listener can be added to sqlalchemy to be able to handle
    database disconnect pessimistically and avoid an error in the event of
    a DB restart or simply a temp connection issue due to network
    glitches. At the moment only mysql is enabled with this feature. This
    patch enables db2 as well.

    Sync oslo db.sqlalchemy from commit：
    bb4d7a2534f6087d828e22784dc263522d309592

    Partial-Bug: #1231657

    Change-Id: Ib89de07f004975528baffdfd19aae689b089a324

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/61602
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=2c199a0388e2f456b9e28057c523c51163189d0d
Submitter: Jenkins
Branch: master

commit 2c199a0388e2f456b9e28057c523c51163189d0d
Author: David Peraza <email address hidden>
Date: Wed Dec 11 17:00:10 2013 -0600

    Oslo sync to recover from db2 server disconnects

    A checkout listener can be added to sqlalchemy to be able to
    handle database disconnect pessimistically and avoid an error in
    the event of a DB restart or simply a temp connection issue due
    to network glitches. At the moment only mysql is enabled with
    this feature. This patch enables db2 as well.

    Oslo version in change I5563360b43c4b648f5b035099b0e1d2e71d32503

    Partial-Bug: #1231657
    Change-Id: Idf8b3b11a06790813fed218bdbaf130554f80fa3

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/65513

JunJie Nan (nanjj)
Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/65513
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=922f4a28c9498ee450c095868cce587cddcf033a
Submitter: Jenkins
Branch: master

commit 922f4a28c9498ee450c095868cce587cddcf033a
Author: David Peraza <email address hidden>
Date: Wed Jan 8 12:45:13 2014 -0600

    Oslo sync to recover from db2 server disconnects

    A checkout listener can be added to sqlalchemy to be able to
    handle database disconnect pessimistically and avoid an error in
    the event of a DB restart or simply a temp connection issue due
    to network glitches. At the moment only mysql is enabled with
    this feature. This patch enables db2 as well.

    Oslo version in change I5563360b43c4b648f5b035099b0e1d2e71d32503

    Partial-Bug: #1231657
    Change-Id: I218ef9836f2e0f135897007d2e53354eb1248422

Thierry Carrez (ttx)
Changed in heat:
milestone: none → icehouse-2
status: Fix Committed → Fix Released
Jay Bryant (jsbryant)
Changed in cinder:
importance: Undecided → Medium
Revision history for this message
gordon chung (chungg) wrote :

fix merged in ceilometer: https://review.openstack.org/#/c/61602/

Changed in ceilometer:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in heat:
milestone: icehouse-2 → 2014.1
Revision history for this message
Jay Bryant (jsbryant) wrote :

The fix for Cinder has been pulled in with our latest Oslo DB Sync: https://review.openstack.org/#/c/77125/

Changed in cinder:
milestone: none → juno-2
status: In Progress → Fix Committed
Viktor Serhieiev (vsergeyev)
Changed in oslo:
status: In Progress → Fix Committed
Roman Podoliaka (rpodolyaka)
Changed in oslo:
importance: Undecided → Medium
Thierry Carrez (ttx)
Changed in oslo:
milestone: none → juno-2
status: Fix Committed → Fix Released
Changed in cinder:
status: Fix Committed → Fix Released
Sean Dague (sdague)
no longer affects: keystone
no longer affects: heat
no longer affects: oslo-incubator
no longer affects: cinder
no longer affects: ceilometer
no longer affects: nova
Revision history for this message
Matt Riedemann (mriedem) wrote :

This was fixed in oslo-incubator:

https://review.openstack.org/#/c/48733/

And that's in oslo.db now and nova is using oslo so yeah, fixed for nova.

Eugene Nikanorov (enikanorov)
no longer affects: neutron
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.