OpenStack Neutron API Charm

shared-db-relation-changed hook fails running migrations during mojo run (trusty-liberty)

Bug #1802115 reported by Chris MacNaughton on 2018-11-07

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Neutron API Charm	Won't Fix	Undecided	Unassigned
	OpenStack Nova Cloud Controller Charm	Won't Fix	Undecided	Unassigned

Bug Description

nova-cloud-controller first looses its connection to the database, and then fails to recreate a table -

2018-11-07 13:38:26 DEBUG shared-db-relation-changed 2018-11-07 13:38:26.637 22832 ERROR 216_havana [-] Exception while creating table.
...
2018-11-07 13:38:26 DEBUG shared-db-relation-changed 2018-11-07 13:38:26.646 22832 CRITICAL nova [-] DBConnectionError: (pymysql.err.OperationalError) (2013, "Lost connection to MySQL server during query (error(104, 'Connection reset by peer'))")

...
2018-11-07 13:38:26 INFO juju-log shared-db:32: Retrying 'migrate_nova_databases' 5 more times (delay=3)
2018-11-07 13:38:29 INFO juju-log shared-db:32: Migrating the nova database.
2018-11-07 13:38:31 DEBUG shared-db-relation-changed No handlers could be found for logger "oslo_config.cfg"
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped from (pymysql.err.InternalError) (1050, u"Table 'instances' already exists") [SQL: u"\nCREATE TABLE instances (\n\tcreated_at DATETIME, \n\tupdated_at DATETIME, \n\tdeleted_at DATETIME, \n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tinternal_id INTEGER, \n\tuser_id VARCHAR(255), \n\tproject_id VARCHAR(255), \n\timage_ref VARCHAR(255), \n\tkernel_id VARCHAR(255), \n\tramdisk_id VARCHAR(255), \n\tlaunch_index INTEGER, \n\tkey_name VARCHAR(255), \n\tkey_data MEDIUMTEXT, \n\tpower_state INTEGER, \n\tvm_state VARCHAR(255), \n\tmemory_mb INTEGER, \n\tvcpus INTEGER, \n\thostname VARCHAR(255), \n\thost VARCHAR(255), \n\tuser_data MEDIUMTEXT, \n\treservation_id VARCHAR(255), \n\tscheduled_at DATETIME, \n\tlaunched_at DATETIME, \n\tterminated_at DATETIME, \n\tdisplay_name VARCHAR(255), \n\tdisplay_description VARCHAR(255), \n\tavailability_zone VARCHAR(255), \n\tlocked BOOL, \n\tos_type VARCHAR(255), \n\tlaunched_on MEDIUMTEXT, \n\tinstance_type_id INTEGER, \n\tvm_mode VARCHAR(255), \n\tuuid VARCHAR(36), \n\tarchitecture VARCHAR(255), \n\troot_device_name VARCHAR(255), \n\taccess_ip_v4 VARCHAR(39), \n\taccess_ip_v6 VARCHAR(39), \n\tconfig_drive VARCHAR(255), \n\ttask_state VARCHAR(255), \n\tdefault_ephemeral_device VARCHAR(255), \n\tdefault_swap_device VARCHAR(255), \n\tprogress INTEGER, \n\tauto_disk_config BOOL, \n\tshutdown_terminate BOOL, \n\tdisable_terminate BOOL, \n\troot_gb INTEGER, \n\tephemeral_gb INTEGER, \n\tcell_name VARCHAR(255), \n\tnode VARCHAR(255), \n\tdeleted INTEGER, \n\tlocked_by ENUM('owner','admin'), \n\tcleaned INTEGER, \n\tPRIMARY KEY (id), \n\tCHECK (locked IN (0, 1)), \n\tCHECK (auto_disk_config IN (0, 1)), \n\tCHECK (shutdown_terminate IN (0, 1)), \n\tCHECK (disable_terminate IN (0, 1))\n)ENGINE=InnoDB CHARSET=utf8\n\n"]
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters context)
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 132, in execute
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query)
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 271, in _query
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q)
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 726, in query
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 861, in _read_query_result
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters result.read()
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1064, in read
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet()
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 826, in _read_packet
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters packet.check_error()
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 370, in check_error
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters raise_mysql_exception(self._data)
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 116, in raise_mysql_exception
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters _check_mysql_exception(errinfo)
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters raise InternalError(errno, errorvalue)
2018-11-07 13:38:32 DEBUG shared-db-relation-changed 2018-11-07 13:38:32.210 22911 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1050, u"Table 'instances' already exists")