SM:R3.0.3 build62kilo: HA: test_svc_creation_with_heat fails with HTTPInternalServerError: ERROR: Remote error: AuthorizationFailure Authorization failed: Bad Gateway (HTTP 502) (HTTP 502)

[sundarkh]

SM:R3.0.3 build62kilo: HA: test_svc_creation_with_heat fails with HTTPInternalServerError: ERROR: Remote error: AuthorizationFailure Authorization failed: Bad Gateway (HTTP 502) (HTTP 502)

1) Provision a Openstack HA setup using SM
2) test_svc_creation_with_heat fails with following trace

Traceback (most recent call last):
File "tcutils/wrappers.py", line 73, in wrapper
result = function(self, *args, **kwargs)
File "scripts/heat/test_heat.py", line 61, in test_svc_creation_with_heat
right_net_fix, r_hs_obj = self.config_vn(stack_name='right_net')
File "scripts/heat/base.py", line 104, in config_vn
vn_hs_obj = self.config_heat_obj(stack_name, template, env)
File "scripts/heat/base.py", line 118, in config_heat_obj
inputs=self.inputs, stack_name=stack_name, project_fq_name=self.inputs.project_fq_name, template=template, env=env))
File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 679, in useFixture
reraise(*exc_info)
File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 666, in useFixture
fixture.setUp()
File "/contrail-test/fixtures/heat_test.py", line 120, in setUp
stack_obj = self.heat_client_obj.stacks.create(**fields)
File "/usr/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 134, in create
data=kwargs, headers=headers)
File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 266, in json_request
resp = self._http_request(url, method, **kwargs)
File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 221, in _http_request
raise exc.from_response(resp)
HTTPInternalServerError: ERROR: Remote error: AuthorizationFailure Authorization failed: Bad Gateway (HTTP 502) (HTTP 502)

 heat stack-list
+----+------------+--------------+---------------+
| id | stack_name | stack_status | creation_time |
+----+------------+--------------+---------------+
+----+------------+--------------+---------------+
root@nodeg21:~#

[sundarkh (sundar-kh)]

In Juno, the error is

<class 'heatclient.exc.HTTPInternalServerError'>: ERROR: Remote error: DBConnectionError (OperationalError) (2006, 'MySQL server has gone away') 'SELECT stack.status_reason AS stack_status_reason, stack.created_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.backup AS stack_backup, stack.updated_at AS stack_updated_at \nFROM stack \nWHERE stack.deleted_at IS NULL AND stack.owner_id IS NULL AND stack.tenant = %s ORDER BY stack.created_at DESC, stack.id DESC' ('137b98f5d1974edb81700d97a4c423b0',)

[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', u' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 69, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 499, in list_stacks\n return [api.format_stack(stack) for stack in stacks]\n', u' File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 268, in load_all\n show_deleted, show_nested) or []\n', u' File "/usr/lib/python2.7/dist-packages/heat/db/api.py", line 130, in stack_get_all\n show_deleted, show_nested)\n', u' File "/usr/lib/python2.7/dist-packages/heat/db/sqlalchemy/api.py", line 368, in stack_get_all\n marker, sort_dir, filters).all()\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2300, in all\n return list(self)\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2412, in __iter__\n return self._execute_and_instances(context)\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2427, in _execute_and_instances\n result = conn.execute(querycontext.statement, self._params)\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute\n return meth(self, multiparams, params)\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection\n return connection._execute_clauseelement(self, multiparams, params)\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement\n compiled_sql, distilled_params\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", li...

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24416
Submitter: Dheeraj Gautam (<email address hidden>)

[sundarkh (sundar-kh)]

with build 65 Kilo
------------------

2016-09-25 23:43:35.914 5744 INFO eventlet.wsgi.server [-] (5744) accepted ('10.204.217.57', 57148)
2016-09-25 23:43:36.789 5744 INFO eventlet.wsgi.server [-] (5744) accepted ('10.204.217.73', 33943)
2016-09-25 23:43:36.830 5744 INFO eventlet.wsgi.server [-] (5744) accepted ('10.204.217.61', 38013)
2016-09-25 23:43:37.004 5744 ERROR heat.common.wsgi [req-06fecaa3-ea50-4cae-82aa-b2bca0d21cc9 f31f654eb5684c25bb8118fc58989f2c debfc53d661247bb87a14cccfbd8ce78] Unexpected error occurred serving API: Remote error: Remote error: AuthorizationFailure Authorization failed: Bad Gateway (HTTP 502) (HTTP 502)
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/heat/common/context.py", line 300, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 685, in create_stack\n stack.store()\n', u' File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 465, in store\n trust_ctx = keystone.create_trust_context()\n', u' File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 233, in create_trust_context\n trustee_user_id = self.admin_client.auth_ref.user_id\n', u' File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 122, in admin_client\n c.authenticate()\n', u' File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner\n return func(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 503, in authenticate\n resp = self.get_raw_token_from_identity_service(**kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/client.py", line 281, in get_raw_token_from_identity_service\n _(\'Authorization failed: %s\') % e)\n', u'AuthorizationFailure: Authorization failed: Bad Gateway (HTTP 502) (HTTP 502)\n']. None
None.
2016-09-25 23:43:37.004 5744 TRACE heat.common.wsgi Traceback (most recent call last):
2016-09-25 23:43:37.004 5744 TRACE heat.common.wsgi File "/usr/lib/python2.7/dist-packages/heat/common/wsgi.py", line 670, in __call__
2016-09-25 23:43:37.004 5744 TRACE heat.common.wsgi request, **action_args)
2016-09-25 23:43:37.004 5744 TRACE heat.common.wsgi File "/usr/lib/python2.7/dist-packages/heat/common/wsgi.py", line 744, in dispatch
2016-09-25 23:43:37.004 5744 TRACE heat.common.wsgi return method(*args, **kwargs)
2016-09-25 23:43:37.004 5744 TRACE heat.common.wsgi File "/usr/l...

[sundarkh (sundar-kh)]

#2 is observed in Juno
Description and #4 are observed in Kilo

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24416
Committed: http://github.org/Juniper/contrail-puppet/commit/01b0ec922d4fd558bcf799bc70b28a2413bb692e
Submitter: Zuul
Branch: R3.0

commit 01b0ec922d4fd558bcf799bc70b28a2413bb692e
Author: Dheeraj Gautam <email address hidden>
Date: Fri Sep 23 14:14:44 2016 -0700

SM-HEAT: Additional database config for heat HA

Closes-Bug: #1626447

Fixes 3.0.3 as per https://review.opencontrail.org/#/c/19415 for the case of
heat HA.

Change-Id: I493ff0b6599b8346a029d2a5e175a7891f7ad1da

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25595
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25595
Committed: http://github.org/Juniper/contrail-puppet/commit/a383bf4b3e2e0cc5e9178838b8614a58dad19891
Submitter: Zuul
Branch: R3.1

commit a383bf4b3e2e0cc5e9178838b8614a58dad19891
Author: Dheeraj Gautam <email address hidden>
Date: Wed Nov 2 00:25:51 2016 -0700

SM-Heat: add heat HA params.

Closes-Bug: #1626447

Fixes as per https://review.opencontrail.org/#/c/19415 for the case of
heat HA.

Change-Id: I6320d0cbd1cca851093fde6593f05848dff632fb