Trove instance stays stuck in BUILD status due to access denied to metadata db in MySQL

[Mike Metral]

A new instance created by Trove never finishes the upstart job as the attempt to connect to the metadata's db gives it an access denied

/tmg/logfile.txt

2013-10-03 21:07:48.752 1338 INFO trove.db.sqlalchemy.session [-] Creating SQLAlchemy engine with args: {'pool_recycle': 3600, 'echo': False}
2013-10-03 21:07:50.771 1338 CRITICAL root [-] (OperationalError) (1045, "Access denied for user 'root'@'localhost' (using password: YES)") None None
2013-10-03 21:07:50.771 1338 TRACE root Traceback (most recent call last):
2013-10-03 21:07:50.771 1338 TRACE root File "/home/mike/trove/bin/trove-guestagent", line 56, in <module>
2013-10-03 21:07:50.771 1338 TRACE root get_db_api().configure_db(CONF)
2013-10-03 21:07:50.771 1338 TRACE root File "/home/mike/trove/trove/db/sqlalchemy/api.py", line 84, in configure_db
2013-10-03 21:07:50.771 1338 TRACE root session.configure_db(options)
2013-10-03 21:07:50.771 1338 TRACE root File "/home/mike/trove/trove/db/sqlalchemy/session.py", line 65, in configure_db
2013-10-03 21:07:50.771 1338 TRACE root mappers.map(_ENGINE, models)
2013-10-03 21:07:50.771 1338 TRACE root File "/home/mike/trove/trove/db/sqlalchemy/mappers.py", line 30, in map
2013-10-03 21:07:50.771 1338 TRACE root orm.mapper(models['instance'], Table('instances', meta, autoload=True))
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/sqlalchemy/schema.py", line 292, in __new__
2013-10-03 21:07:50.771 1338 TRACE root table._init(name, metadata, *args, **kw)
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/sqlalchemy/schema.py", line 353, in _init
2013-10-03 21:07:50.771 1338 TRACE root self._autoload(metadata, autoload_with, include_columns)
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/sqlalchemy/schema.py", line 377, in _autoload
2013-10-03 21:07:50.771 1338 TRACE root self, include_columns
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2274, in run_callable
2013-10-03 21:07:50.771 1338 TRACE root conn = self.contextual_connect()
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2340, in contextual_connect
2013-10-03 21:07:50.771 1338 TRACE root self.pool.connect(),
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 210, in connect
2013-10-03 21:07:50.771 1338 TRACE root return _ConnectionFairy(self).checkout()
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 371, in __init__
2013-10-03 21:07:50.771 1338 TRACE root rec = self._connection_record = pool._do_get()
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 697, in _do_get
2013-10-03 21:07:50.771 1338 TRACE root con = self._create_connection()
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 174, in _create_connection
2013-10-03 21:07:50.771 1338 TRACE root return _ConnectionRecord(self)
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 256, in __init__
2013-10-03 21:07:50.771 1338 TRACE root self.connection = self.__connect()
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 316, in __connect
2013-10-03 21:07:50.771 1338 TRACE root connection = self.__pool._creator()
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2013-10-03 21:07:50.771 1338 TRACE root return dialect.connect(*cargs, **cparams)
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 280, in connect
2013-10-03 21:07:50.771 1338 TRACE root return self.dbapi.connect(*cargs, **cparams)
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
2013-10-03 21:07:50.771 1338 TRACE root return Connection(*args, **kwargs)
2013-10-03 21:07:50.771 1338 TRACE root File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
2013-10-03 21:07:50.771 1338 TRACE root super(Connection, self).__init__(*args, **kwargs2)
2013-10-03 21:07:50.771 1338 TRACE root OperationalError: (OperationalError) (1045, "Access denied for user 'root'@'localhost' (using password: YES)") None None
2013-10-03 21:07:50.771 1338 TRACE root

Bug is associated with gate bug discovered today 10/2/13

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/49659

[Mike Metral (mike-metral)]

This bug is due to fix in devstack in commit: https://github.com/openstack-dev/devstack/commit/96ba6ec1bf0b7cc54f9968e4cc3aa80b8f2c368e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/49659
Committed: http://github.com/openstack/trove-integration/commit/82df245073429cf5b4747841494f46b75119f2ec
Submitter: Jenkins
Branch: master

commit 82df245073429cf5b4747841494f46b75119f2ec
Author: Mike Metral <email address hidden>
Date: Fri Oct 4 01:03:47 2013 +0000

    Sets correct MYSQL_HOST needed to run stack.sh

    Change-Id: I0bbb79551fd6fecb8a72688ed23da5223e333077
    Closes-Bug: #1235037