14.04.5 provision failed on compute with keystone authentication error

[wenqing liang]

r4.0-10 mitaka provision of 14.04.5 openstack-/contrail-ha cluster failed on computes with:

2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi [req-e702969a-a602-4270-a5f6-2a1d99639b63 - - - - -] (pymysql.err.ProgrammingError) (1146, u"Table 'keystone.user' doesn't exist") [SQL: u'SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id \nFROM user INNER JOIN local_user ON user.id = local_user.user_id \nWHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s'] [parameters: {u'domain_id_1': 'default', u'name_1': u'admin'}]
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi Traceback (most recent call last):
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 249, in __call__
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi result = method(context, **params)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_log/versionutils.py", line 165, in wrapped
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi return func_or_cls(*args, **kwargs)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/controllers.py", line 100, in authenticate
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi context, auth)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/controllers.py", line 294, in _authenticate_local
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi username, CONF.identity.default_domain_id)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in wrapped
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi __ret_val = __f(*args, **kwargs)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 433, in wrapper
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi return f(self, *args, **kwargs)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 443, in wrapper
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi return f(self, *args, **kwargs)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1053, in decorate
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi should_cache_fn)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 657, in get_or_create
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi async_creator) as value:
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 158, in __enter__
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi return self._enter()
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 98, in _enter
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi generated = self._enter_create(createdtime)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 149, in _enter_create
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi created = self.creator()
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 625, in gen_value
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi created_value = creator()
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1049, in creator
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi return fn(*arg, **kw)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 902, in get_user_by_name
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi ref = driver.get_user_by_name(user_name, domain_id)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/sql.py", line 251, in get_user_by_name
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi user_ref = query.one()
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2693, in one
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi ret = list(self)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi return self._execute_and_instances(context)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi result = conn.execute(querycontext.statement, self._params)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi return meth(self, multiparams, params)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi compiled_sql, distilled_params
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi context)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi util.raise_from_cause(newraise, exc_info)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi reraise(type(exception), exception, tb=exc_tb)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi context)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi cursor.execute(statement, parameters)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 158, in execute
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi result = self._query(query)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 308, in _query
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi conn.query(q)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 820, in query
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1002, in _read_query_result
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi result.read()
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1285, in read
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi first_packet = self.connection._read_packet()
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 966, in _read_packet
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi packet.check_error()
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 394, in check_error
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi err.raise_mysql_exception(self._data)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi _check_mysql_exception(errinfo)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi raise errorclass(errno, errorvalue)
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'keystone.user' doesn't exist") [SQL: u'SELECT user.id AS user_id, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id \nFROM user INNER JOIN local_user ON user.id = local_user.user_id \nWHERE local_user.name = %(name_1)s AND local_user.domain_id = %(domain_id_1)s'] [parameters: {u'domain_id_1': 'default', u'name_1': u'admin'}]
2017-05-22 20:05:27.599 22542 ERROR keystone.common.wsgi

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31998
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31998
Committed: http://github.com/Juniper/contrail-puppet/commit/6abcd9d91c9b8df50ebd8001fc0bfebae8bbe815
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 6abcd9d91c9b8df50ebd8001fc0bfebae8bbe815
Author: Dheeraj Gautam <email address hidden>
Date: Mon May 22 23:03:07 2017 -0700

fix 14.04.5 openstack ha issues.

Partial-Bug: #1692703

1. tested with 14.04.5 provisioning and checked mysql access is proper and
created glance image.

Change-Id: I77cad60c99a003592e6b9b4a750b17c767cad348

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/32028
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/32031
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/32028
Committed: http://github.com/Juniper/contrail-puppet/commit/9546907319eed872a5a513399c4e2f1511b7f49f
Submitter: Zuul (<email address hidden>)
Branch: master

commit 9546907319eed872a5a513399c4e2f1511b7f49f
Author: Dheeraj Gautam <email address hidden>
Date: Mon May 22 23:03:07 2017 -0700

fix 14.04.5 openstack ha issues.

Partial-Bug: #1692703

1. tested with 14.04.5 provisioning and checked mysql access is proper and
created glance image.

Change-Id: I77cad60c99a003592e6b9b4a750b17c767cad348
(cherry picked from commit 6abcd9d91c9b8df50ebd8001fc0bfebae8bbe815)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/32031
Committed: http://github.com/Juniper/contrail-puppet/commit/ec88248f70fd97129439788637c6056ae3205e64
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit ec88248f70fd97129439788637c6056ae3205e64
Author: Dheeraj Gautam <email address hidden>
Date: Tue May 23 14:23:56 2017 -0700

fix 16.04 galera sync and keystone startup

Closes-Bug: #1692703

testing:
1. reimaged/provisoined 16.04 and provisioning was successful.
issued, tenant list and created network.

changes:
1. remove contrail-hamon service (for now)
2. galera setup fails for install_mysql_db for 16.04, don't call that on 16.04
3. check if /etc/keystone/ssl before trying to copy files.
4. fix keystone virtual-host config for apache to listen on ports.
5. ensure to disable keystone site (default site as part of packakge)
6. use "systemdctl enable mysql" instead of update-rc.d

Change-Id: Ibebf7d5d41ec54bff64c1c75cc96cef554986dac

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/32070
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/32070
Committed: http://github.com/Juniper/contrail-puppet/commit/b031147c8f4f5b9084c5315f7be4caedfcb2e15e
Submitter: Zuul (<email address hidden>)
Branch: master

commit b031147c8f4f5b9084c5315f7be4caedfcb2e15e
Author: Dheeraj Gautam <email address hidden>
Date: Tue May 23 14:23:56 2017 -0700

fix 16.04 galera sync and keystone startup

Closes-Bug: #1692703

testing:
1. reimaged/provisoined 16.04 and provisioning was successful.
issued, tenant list and created network.

changes:
1. remove contrail-hamon service (for now)
2. galera setup fails for install_mysql_db for 16.04, don't call that on 16.04
3. check if /etc/keystone/ssl before trying to copy files.
4. fix keystone virtual-host config for apache to listen on ports.
5. ensure to disable keystone site (default site as part of packakge)
6. use "systemdctl enable mysql" instead of update-rc.d

Conflicts:
 contrail/environment/modules/contrail/manifests/ha_config.pp

Change-Id: Ibebf7d5d41ec54bff64c1c75cc96cef554986dac