restarting mysql, Keystone doesn't reconnect, AttributeError

[Brant Knudson]

When restart the mysqld server that Keystone is connected to, Keystone operations fail with a 500 Internal Server Error.

To recreate, start with normal devstack. Run a keystone command:

$ keystone user-list
+----------------------------------+----------+---------+----------------------+
| id | name | enabled | email |
+----------------------------------+----------+---------+----------------------+
| 1fae4ded68444d5fbc646fce442eaff1 | admin | True | <email address hidden> |
| 3ff71f71063f4aeea0522d1261267861 | alt_demo | True | <email address hidden> |
...

Then restart mysqld:

$ sudo service mysql restart

Then run a keystone command again:

$ keystone user-list
Authorization Failed: An unexpected error prevented the server from fulfilling your request. 'module' object has no attribute 'warn' (HTTP 500)

The Keystone server log has:

  File "/opt/stack/keystone/keystone/common/sql/core.py", line 207, in mysql_on_checkout
    logging.warn(_('Got mysql server has gone away: %s'), e)
AttributeError: 'module' object has no attribute 'warn'

The expected behavior is that the Keystone server will reconnect properly and the command will work as normal.

[Dolph Mathews (dolph)]

This doesn't affect grizzly, right?

[Brant Knudson (blk-u)]

I don't think this affect grizzly. The same code is there: https://github.com/openstack/keystone/blob/stable/grizzly/keystone/common/sql/core.py#L204 , but there's also a logging.warn (https://github.com/openstack/keystone/blob/stable/grizzly/keystone/common/logging.py#L47). This must have been introduced with the change to oslo logging.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/49270
Committed: http://github.com/openstack/keystone/commit/d973ea9a47a6f9892208722f69a0cd127b2f73dc
Submitter: Jenkins
Branch: master

commit d973ea9a47a6f9892208722f69a0cd127b2f73dc
Author: Brant Knudson <email address hidden>
Date: Tue Oct 1 16:26:18 2013 -0500

    Fix mysql checkout handler AttributeError

    If the mysql_on_checkout handler encountered an error due to
    disconnect, it would fail with an AttributeError,

     AttributeError: 'module' object has no attribute 'warn'

    This is because it used the logging module where the LOG logger
    should have been used.

    Change-Id: Iae8a9bd4aeee6e35508059c20b92d70c36232963
    Closes-Bug: #1234873

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/50454

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/50454
Committed: http://github.com/openstack/keystone/commit/fd5260d452d750ba2ad8e0f3cb77e658d4d56264
Submitter: Jenkins
Branch: milestone-proposed

commit fd5260d452d750ba2ad8e0f3cb77e658d4d56264
Author: Brant Knudson <email address hidden>
Date: Tue Oct 1 16:26:18 2013 -0500

    Fix mysql checkout handler AttributeError

    If the mysql_on_checkout handler encountered an error due to
    disconnect, it would fail with an AttributeError,

     AttributeError: 'module' object has no attribute 'warn'

    This is because it used the logging module where the LOG logger
    should have been used.

    Change-Id: Iae8a9bd4aeee6e35508059c20b92d70c36232963
    Closes-Bug: #1234873
    (cherry picked from commit d973ea9a47a6f9892208722f69a0cd127b2f73dc)