restarting mysql, Keystone doesn't reconnect, AttributeError

Bug #1234873 reported by Brant Knudson on 2013-10-03
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
High
Brant Knudson

Bug Description

When restart the mysqld server that Keystone is connected to, Keystone operations fail with a 500 Internal Server Error.

To recreate, start with normal devstack. Run a keystone command:

$ keystone user-list
+----------------------------------+----------+---------+----------------------+
| id | name | enabled | email |
+----------------------------------+----------+---------+----------------------+
| 1fae4ded68444d5fbc646fce442eaff1 | admin | True | <email address hidden> |
| 3ff71f71063f4aeea0522d1261267861 | alt_demo | True | <email address hidden> |
...

Then restart mysqld:

$ sudo service mysql restart

Then run a keystone command again:

$ keystone user-list
Authorization Failed: An unexpected error prevented the server from fulfilling your request. 'module' object has no attribute 'warn' (HTTP 500)

The Keystone server log has:

  File "/opt/stack/keystone/keystone/common/sql/core.py", line 207, in mysql_on_checkout
    logging.warn(_('Got mysql server has gone away: %s'), e)
AttributeError: 'module' object has no attribute 'warn'

The expected behavior is that the Keystone server will reconnect properly and the command will work as normal.

Brant Knudson (blk-u) on 2013-10-03
Changed in keystone:
assignee: nobody → Brant Knudson (blk-u)
Changed in keystone:
status: New → In Progress
Dolph Mathews (dolph) wrote :

This doesn't affect grizzly, right?

Changed in keystone:
importance: Undecided → High
tags: added: havana-backport-potential
Brant Knudson (blk-u) wrote :

I don't think this affect grizzly. The same code is there: https://github.com/openstack/keystone/blob/stable/grizzly/keystone/common/sql/core.py#L204 , but there's also a logging.warn (https://github.com/openstack/keystone/blob/stable/grizzly/keystone/common/logging.py#L47). This must have been introduced with the change to oslo logging.

Reviewed: https://review.openstack.org/49270
Committed: http://github.com/openstack/keystone/commit/d973ea9a47a6f9892208722f69a0cd127b2f73dc
Submitter: Jenkins
Branch: master

commit d973ea9a47a6f9892208722f69a0cd127b2f73dc
Author: Brant Knudson <email address hidden>
Date: Tue Oct 1 16:26:18 2013 -0500

    Fix mysql checkout handler AttributeError

    If the mysql_on_checkout handler encountered an error due to
    disconnect, it would fail with an AttributeError,

     AttributeError: 'module' object has no attribute 'warn'

    This is because it used the logging module where the LOG logger
    should have been used.

    Change-Id: Iae8a9bd4aeee6e35508059c20b92d70c36232963
    Closes-Bug: #1234873

Changed in keystone:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2013-10-08
Changed in keystone:
milestone: none → havana-rc2
tags: removed: havana-backport-potential

Reviewed: https://review.openstack.org/50454
Committed: http://github.com/openstack/keystone/commit/fd5260d452d750ba2ad8e0f3cb77e658d4d56264
Submitter: Jenkins
Branch: milestone-proposed

commit fd5260d452d750ba2ad8e0f3cb77e658d4d56264
Author: Brant Knudson <email address hidden>
Date: Tue Oct 1 16:26:18 2013 -0500

    Fix mysql checkout handler AttributeError

    If the mysql_on_checkout handler encountered an error due to
    disconnect, it would fail with an AttributeError,

     AttributeError: 'module' object has no attribute 'warn'

    This is because it used the logging module where the LOG logger
    should have been used.

    Change-Id: Iae8a9bd4aeee6e35508059c20b92d70c36232963
    Closes-Bug: #1234873
    (cherry picked from commit d973ea9a47a6f9892208722f69a0cd127b2f73dc)

Changed in keystone:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-10-17
Changed in keystone:
milestone: havana-rc2 → 2013.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers