ISO: {"build_id": "2014-08-04_22-06-52", "ostf_sha": "a3fa823ea0e4e03beb637ae07a91adea82c33182", "build_number": "564", "auth_required": true, "api": "1.0", "nailgun_sha": "5d1e8e38ea3f2cd88ceca8e0927d53e9770598bc", "production": "docker", "fuelmain_sha": "6b05280621c23ab158e9943f88b6f71a22fd4c69", "astute_sha": "99a790ad1b7526cbbd5bf8add0cb2b4e503fccd4", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "25eb632c70fe0bb3b2c1053314b558a160e4af50"}
Steps To Reproduce:
1. Install OS(Ubuntu, Nova Network, Ceilometer, 3 Controller+Mongo, 2 Compute)
2. Launch "Ceilometer create, update, check, delete alarm" ostf test
3. Launch any ostf test(sanity tests for example)
Actual result:
This test fails on "6. Wait for 'alarm' alarm state.". And all ostf tests after this action will fail with reason "Keystone client is not available. Please, refer to OpenStack logs to fix this problem"
In keystone logs we can see following traceback:
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.223 16283 TRACE keystone.common.wsgi
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi Traceback (most recent call last):
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 212, in __call__
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi result = method(context, **params)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/controllers.py", line 98, in authenticate
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi context, auth)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/controllers.py", line 267, in _authenticate_local
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi username, CONF.identity.default_domain_id)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 189, in wrapper
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return f(self, *args, **kwargs)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 314, in get_user_by_name
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi driver = self._select_identity_driver(domain_id)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 256, in _select_identity_driver
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi self.assignment_api.get_domain(domain_id)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 936, in decorate
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi should_cache_fn)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 588, in get_or_create
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi async_creator) as value:
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 158, in __enter__
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return self._enter()
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 98, in _enter
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi generated = self._enter_create(createdtime)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 149, in _enter_create
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi created = self.creator()
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 565, in gen_value
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi created_value = creator()
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 932, in creator
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return fn(*arg, **kw)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/assignment/core.py", line 297, in get_domain
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return self.driver.get_domain(domain_id)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/assignment/backends/sql.py", line 464, in get_domain
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return self._get_domain(session, domain_id).to_dict()
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/assignment/backends/sql.py", line 457, in _get_domain
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi ref = session.query(Domain).get(domain_id)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 816, in get
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return loading.load_on_ident(self, key)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 226, in load_on_ident
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return q.one()
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2306, in one
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi ret = list(self)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2349, in __iter__
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return self._execute_and_instances(context)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2362, in _execute_and_instances
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi close_with_result=True)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2353, in _connection_from_session
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi **kw)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 799, in connection
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi close_with_result=close_with_result)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return self.transaction._connection_for_bind(engine)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi conn = bind.contextual_connect()
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi self.pool.connect(),
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 272, in connect
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return _ConnectionFairy(self).checkout()
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 425, in __init__
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi rec = self._connection_record = pool._do_get()
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 778, in _do_get
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi con = self._create_connection()
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 225, in _create_connection
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return _ConnectionRecord(self)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 318, in __init__
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi self.connection = self.__connect()
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 368, in __connect
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi connection = self.__pool._creator()
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 80, in connect
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return dialect.connect(*cargs, **cparams)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 283, in connect
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return self.dbapi.connect(*cargs, **cparams)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi return Connection(*args, **kwargs)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 193, in __init__
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi super(Connection, self).__init__(*args, **kwargs2)
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi OperationalError: (OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") None None
node-1.test.domain.local/keystone-all.log:2014-08-05 14:48:25.500 16283 TRACE keystone.common.wsgi
Deployment was finished with healthy galera cluster, according to logs: 05T14:05: 17.848575 node-5 ./remote/ node-5. test.domain. local/puppet- apply.log: 2014-08- 05T14:05: 17.848575+ 01:00 notice: Finished catalog run i 05T14:27: 53.636527 node-1 ./remote/ node-1. test.domain. local/nova- scheduler. log:2014- 08-05T14: 27:53.636527+ 01:00 err: 2014-08-05 13:27:53.468 up.drivers. db [-] model server went away
2014-08-
n 0.15 seconds
2014-08-
15641 ERROR nova.servicegro
Galera cluster issues started 22 min later