One thing I noticed is the following on controller-0:
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core [req-a910cef1-4202-41dd-a895-d5d04b3076ae - - - - -] Could not find user: ceilometer
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core Traceback (most recent call last):
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core File "/usr/lib/python2.7/site-packages/keystone/auth/plugins/core.py", line 173, in _validate_and_normalize_auth_data
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core user_name, domain_ref['id'])
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 124, in wrapped
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core __ret_val = __f(*args, **kwargs)
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 433, in wrapper
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core return f(self, *args, **kwargs)
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 443, in wrapper
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core return f(self, *args, **kwargs)
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 1053, in decorate
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core should_cache_fn)
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 657, in get_or_create
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core async_creator) as value:
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core File "/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py", line 158, in __enter__
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core return self._enter()
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core File "/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py", line 98, in _enter
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core generated = self._enter_create(createdtime)
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core File "/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py", line 149, in _enter_create
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core created = self.creator()
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 625, in gen_value
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core created_value = creator()
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 1049, in creator
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core return fn(*arg, **kw)
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 902, in get_user_by_name
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core ref = driver.get_user_by_name(user_name, domain_id)
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core File "/usr/lib/python2.7/site-packages/keystone/identity/backends/sql.py", line 253, in get_user_by_name
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core raise exception.UserNotFound(user_id=user_name)
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core UserNotFound: Could not find user: ceilometer
keystone/keystone.log:2016-10-19 10:49:03.523 16145 ERROR keystone.auth.plugins.core
keystone/keystone.log:2016-10-19 10:49:03.527 16145 WARNING keystone.common.wsgi [req-a910cef1-4202-41dd-a895-d5d04b3076ae - - - - -] Authorization failed. The request you have made requires authentication. from 172.17.0.13
Although in the CI job http://logs.openstack.org/68/388568/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/b8e1308/ we have the following:
2016-10-19 10:47:14.651913 | 2016-10-19 10:47:02 [ExtraConfig]: CREATE_COMPLETE state changed
2016-10-19 10:47:14.651994 | 2016-10-19 10:47:02 [overcloud-BlockStorageNodesPostDeployment-5cqyvvmdpldh]: CREATE_COMPLETE Stack CREATE completed successfully
2016-10-19 10:47:15.342118 | Stack overcloud CREATE_COMPLETE
2016-10-19 10:48:23.482117 | Service Unavailable (HTTP 503)
2016-10-19 10:48:23.544456 | #################
But the 503 gets returned *before* the ceilometer messages (unless there is a clock skew undercloud/overcloud). Not 100% sure that error is relevant to this failure, but the ceilometer exception is probably worth investigating anyway
One thing I noticed is the following on controller-0: keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core [req-a910cef1- 4202-41dd- a895-d5d04b3076 ae - - - - -] Could not find user: ceilometer keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core Traceback (most recent call last): keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core File "/usr/lib/ python2. 7/site- packages/ keystone/ auth/plugins/ core.py" , line 173, in _validate_ and_normalize_ auth_data keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core user_name, domain_ref['id']) keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core File "/usr/lib/ python2. 7/site- packages/ keystone/ common/ manager. py", line 124, in wrapped keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core __ret_val = __f(*args, **kwargs) keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core File "/usr/lib/ python2. 7/site- packages/ keystone/ identity/ core.py" , line 433, in wrapper keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core return f(self, *args, **kwargs) keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core File "/usr/lib/ python2. 7/site- packages/ keystone/ identity/ core.py" , line 443, in wrapper keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core return f(self, *args, **kwargs) keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core File "/usr/lib/ python2. 7/site- packages/ dogpile/ cache/region. py", line 1053, in decorate keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core should_cache_fn) keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core File "/usr/lib/ python2. 7/site- packages/ dogpile/ cache/region. py", line 657, in get_or_create keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core async_creator) as value: keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core File "/usr/lib/ python2. 7/site- packages/ dogpile/ core/dogpile. py", line 158, in __enter__ keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core return self._enter() keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core File "/usr/lib/ python2. 7/site- packages/ dogpile/ core/dogpile. py", line 98, in _enter keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core generated = self._enter_ create( createdtime) keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core File "/usr/lib/ python2. 7/site- packages/ dogpile/ core/dogpile. py", line 149, in _enter_create keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core created = self.creator() keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core File "/usr/lib/ python2. 7/site- packages/ dogpile/ cache/region. py", line 625, in gen_value keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core created_value = creator() keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core File "/usr/lib/ python2. 7/site- packages/ dogpile/ cache/region. py", line 1049, in creator keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core return fn(*arg, **kw) keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core File "/usr/lib/ python2. 7/site- packages/ keystone/ identity/ core.py" , line 902, in get_user_by_name keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core ref = driver. get_user_ by_name( user_name, domain_id) keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core File "/usr/lib/ python2. 7/site- packages/ keystone/ identity/ backends/ sql.py" , line 253, in get_user_by_name keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core raise exception. UserNotFound( user_id= user_name) keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core UserNotFound: Could not find user: ceilometer keystone. log:2016- 10-19 10:49:03.523 16145 ERROR keystone. auth.plugins. core keystone. log:2016- 10-19 10:49:03.527 16145 WARNING keystone. common. wsgi [req-a910cef1- 4202-41dd- a895-d5d04b3076 ae - - - - -] Authorization failed. The request you have made requires authentication. from 172.17.0.13
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
keystone/
Although in the CI job http:// logs.openstack. org/68/ 388568/ 1/check- tripleo/ gate-tripleo- ci-centos- 7-ovb-ha/ b8e1308/ we have the following: BlockStorageNod esPostDeploymen t-5cqyvvmdpldh] : CREATE_COMPLETE Stack CREATE completed successfully
2016-10-19 10:47:14.651913 | 2016-10-19 10:47:02 [ExtraConfig]: CREATE_COMPLETE state changed
2016-10-19 10:47:14.651994 | 2016-10-19 10:47:02 [overcloud-
2016-10-19 10:47:15.342118 | Stack overcloud CREATE_COMPLETE
2016-10-19 10:48:23.482117 | Service Unavailable (HTTP 503)
2016-10-19 10:48:23.544456 | #################
But the 503 gets returned *before* the ceilometer messages (unless there is a clock skew undercloud/ overcloud) . Not 100% sure that error is relevant to this failure, but the ceilometer exception is probably worth investigating anyway