catch panko errors when not available

Bug #1633592 reported by gordon chung
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Low
Hanxi Liu

Bug Description

this is all over newton and master logs

2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped from (pymysql.err.InternalError) (1049, u"Unknown database 'panko'")
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters return fn()
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters return _ConnectionFairy._checkout(self)
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters fairy = _ConnectionRecord.checkout(pool)
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters rec = pool._do_get()
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters self._dec_overflow()
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters compat.reraise(exc_type, exc_value, exc_tb)
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters return self._create_connection()
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters return _ConnectionRecord(self)
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters self.connection = self.__connect()
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters connection = self.__pool._invoke_creator(self)
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters return dialect.connect(*cargs, **cparams)
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters return self.dbapi.connect(*cargs, **cparams)
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/pymysql/__init__.py", line 90, in Connect
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters return Connection(*args, **kwargs)
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 688, in __init__
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters self.connect()
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 906, in connect
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters self._request_authentication()
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1114, in _request_authentication
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters auth_packet = self._read_packet()
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 981, in _read_packet
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters packet.check_error()
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 393, in check_error
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data)
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters raise errorclass(errno, errval)
2016-10-14 14:35:01.538 22163 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1049, u"Unknown database 'panko'")

Revision history for this message
gordon chung (chungg) wrote :

this be broke...

Changed in ceilometer:
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
gordon chung (chungg) wrote :

failing initially because panko is installed and configured after ceilometer is started.

no longer affects: panko
Changed in ceilometer:
importance: Critical → High
Revision history for this message
Hanxi Liu (hanxi-liu) wrote :

Has it fixed? I see the related patchs have been merged.

Revision history for this message
gordon chung (chungg) wrote :

fixed in gate but should hide error better

Changed in ceilometer:
importance: High → Low
gordon chung (chungg)
summary: - collector doesn't understand panko
+ catch panko errors when not available.
summary: - catch panko errors when not available.
+ catch panko errors when not available
Hanxi Liu (hanxi-liu)
Changed in ceilometer:
assignee: nobody → Hanxi Liu (hanxi-liu)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/415866

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/415866
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=322ccb0f4324246ac4d6d61c93c33a62f3075486
Submitter: Jenkins
Branch: master

commit 322ccb0f4324246ac4d6d61c93c33a62f3075486
Author: Hanxi Liu <email address hidden>
Date: Thu Dec 29 16:40:18 2016 +0800

    devstack: check for ceilometer+panko service

    Make sure panko is enabled before ceilometer in case
    potential issues occur.

    Change-Id: I7d27f6188557249789193cbb3a1928a2d7ee84b4
    Partial-Bug: #1633592

Revision history for this message
gordon chung (chungg) wrote :

closing, not much else we can do if the db itself doesn't exist.

Changed in ceilometer:
status: In Progress → Invalid
status: Invalid → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.