Juniper Openstack

SMR3.1build18:creation of SVC with heat fails with authorisation failure

Series trunk
Bug #1609683

Bug #1609683 reported by sundarkh on 2016-08-04

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R3.0.3.x	Won't Fix	High	Dheeraj Gautam
R3.1	Fix Committed	High	Dheeraj Gautam	Juniper Openstack r3.1.1.0
Trunk	Fix Committed	High	Dheeraj Gautam	Juniper Openstack r3.2.0.0-fcs "r3.2.0.0"

Bug Description

SMR3.1build14:creation of SVC with heat fails with authorisation failure The resource could not be found. (HTTP 404)

Provisioned a node with SM R3.1 build 14 kilo

try creating a svc with heat, it fails with authorisation failure The resource could not be found. (HTTP 404)

Heat Logs while trying to create SVC

Heat-engine.log

34069 2016-08-03 22:00:48.109 5769 DEBUG keystoneclient.auth.identity.v3 [req-5e24424a-2d08-4923-aaee-bb2f1be3f941 0eac51d15cc544159fc4387cd8e47352 e128a056db4b44708172b186ebcba2b5] Making authentication request to http://10.204.217.77:5000/auth/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3.py:125
  34070 2016-08-03 22:00:48.116 5769 DEBUG keystoneclient.session [req-5e24424a-2d08-4923-aaee-bb2f1be3f941 0eac51d15cc544159fc4387cd8e47352 e128a056db4b44708172b186ebcba2b5] Request returned failure status: 404 request /usr/lib/python2.7/dist-packages/keystoneclient/session.py:388
  34071 2016-08-03 22:00:48.118 5769 ERROR oslo_messaging.rpc.dispatcher [req-5e24424a-2d08-4923-aaee-bb2f1be3f941 0eac51d15cc544159fc4387cd8e47352 e128a056db4b44708172b186ebcba2b5] Exception during message handling: Authorization failed: The resource could not be found. (HTTP 404) (HTTP 404)
  34072 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  34073 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  34074 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher executor_callback))
  34075 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  34076 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher executor_callback)
  34077 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  34078 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
  34079 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
  34080 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
  34081 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/common/context.py", line 300, in wrapped
  34082 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher return func(self, ctx, *args, **kwargs)
  34083 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 685, in create_stack
  34084 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher stack.store()
  34085 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
  34086 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
  34087 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 462, in store
  34088 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher trust_ctx = keystone.create_trust_context()
  34089 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 233, in create_trust_context
  34090 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher trustee_user_id = self.admin_client.auth_ref.user_id

34091 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 122, in admin_client
  34092 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher c.authenticate()
  34093 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner
  34094 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher return func(*args, **kwargs)
  34095 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 503, in authenticate
  34096 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher resp = self.get_raw_token_from_identity_service(**kwargs)
  34097 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/client.py", line 281, in get_raw_token_from_identity_service
  34098 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher _('Authorization failed: %s') % e)
  34099 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher AuthorizationFailure: Authorization failed: The resource could not be found. (HTTP 404) (HTTP 404)
  34100 2016-08-03 22:00:48.118 5769 TRACE oslo_messaging.rpc.dispatcher
  34101 2016-08-03 22:00:48.125 5769 DEBUG oslo_messaging._drivers.amqp [req-5e24424a-2d08-4923-aaee-bb2f1be3f941 0eac51d15cc544159fc4387cd8e47352 e128a056db4b44708172b186ebcba2b5] Pool creating new connection create /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:86
  34102

Heat-api.log
2016-08-03 22:31:10.996 1792 DEBUG oslo_messaging._drivers.amqpdriver [req-27116b0d-6a4e-4116-bbfa-fb23495d3003 881bfc13fc504b9f894b95cbb68cc589 3810d898ed3e46efb3307cf32f43df41] MSG_ID is c4583183382e45889999a9683b60db24 _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2016-08-03 22:31:10.997 1792 DEBUG oslo_messaging._drivers.amqp [req-27116b0d-6a4e-4116-bbfa-fb23495d3003 881bfc13fc504b9f894b95cbb68cc589 3810d898ed3e46efb3307cf32f43df41] UNIQUE_ID is 906013f3bf174679ab19ae7cf7563d3b. _add_unique_id /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:252
2016-08-03 22:31:11.045 1792 ERROR heat.common.wsgi [req-27116b0d-6a4e-4116-bbfa-fb23495d3003 881bfc13fc504b9f894b95cbb68cc589 3810d898ed3e46efb3307cf32f43df41] Unexpected error occurred serving API: Remote error: Remote error: AuthorizationFailure Authorization failed: The resource could not be found. (HTTP 404) (HTTP 404)
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/heat/common/context.py", line 300, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 685, in create_stack\n stack.store()\n', u' File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 462, in store\n trust_ctx = keystone.create_trust_context()\n', u' File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 233, in create_trust_context\n trustee_user_id = self.admin_client.auth_ref.user_id\n', u' File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 122, in admin_client\n c.authenticate()\n', u' File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner\n return func(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 503, in authenticate\n resp = self.get_raw_token_from_identity_service(**kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/client.py", line 281, in get_raw_token_from_identity_service\n _(\'Authorization failed: %s\') % e)\n', u'AuthorizationFailure: Authorization failed: The resource could not be found. (HTTP 404) (HTTP 404)\n']. None
None.
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi Traceback (most recent call last):
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi File "/usr/lib/python2.7/dist-packages/heat/common/wsgi.py", line 670, in __call__
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi request, **action_args)
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi File "/usr/lib/python2.7/dist-packages/heat/common/wsgi.py", line 744, in dispatch
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi return method(*args, **kwargs)
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi File "/usr/lib/python2.7/dist-packages/heat/api/openstack/v1/util.py", line 38, in handle_stack_method
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi return handler(controller, req, **kwargs)
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi File "/usr/lib/python2.7/dist-packages/heat/api/openstack/v1/stacks.py", line 300, in create

2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi retry=self.retry)
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi timeout=timeout, retry=retry)
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi retry=retry)
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi raise result
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi RemoteError: Remote error: AuthorizationFailure Authorization failed: The resource could not be found. (HTTP 404) (HTTP 404)
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/heat/common/context.py", line 300, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 685, in create_stack\n stack.store()\n', u' File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 462, in store\n trust_ctx = keystone.create_trust_context()\n', u' File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 233, in create_trust_context\n trustee_user_id = self.admin_client.auth_ref.user_id\n', u' File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 122, in admin_client\n c.authenticate()\n', u' File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner\n return func(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 503, in authenticate\n resp = self.get_raw_token_from_identity_service(**kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/client.py", line 281, in get_raw_token_from_identity_service\n _(\'Authorization failed: %s\') % e)\n', u'AuthorizationFailure: Authorization failed: The resource could not be found. (HTTP 404) (HTTP 404)\n'].
2016-08-03 22:31:11.045 1792 TRACE heat.common.wsgi
2016-08-03 22:31:11.046

Tags:

sundarkh (sundar-kh) on 2016-08-04

Changed in juniperopenstack:
milestone:	none → r3.1.0.0-fcs

sundarkh (sundar-kh) on 2016-08-04

Changed in juniperopenstack:
milestone:	r3.1.0.0-fcs → none

Revision history for this message

Praneet Bachheti (praneetb) wrote on 2016-08-04:

auth_uri needs to be fixed.

Changed in juniperopenstack:
assignee:	Praneet Bachheti (praneetb) → Nitish Krishna Kaveri (nitishk)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-08-04: [Review update] R3.1

Review in progress for https://review.opencontrail.org/22903
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-08-05:

Review in progress for https://review.opencontrail.org/22903
Submitter: Dheeraj Gautam (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-08-05: A change has been merged

Reviewed: https://review.opencontrail.org/22903
Committed: http://github.org/Juniper/contrail-puppet/commit/e37857560a4ab0b9713adee2b2df7d74ae18b361
Submitter: Zuul
Branch: R3.1

commit e37857560a4ab0b9713adee2b2df7d74ae18b361
Author: nitishkrishna <email address hidden>
Date: Thu Aug 4 14:21:14 2016 -0700

Closes-Bug: #1609683 - Heat authentication failures in All-in-one SM Provision
Closes-Bug: #1607978 - Heat bind host misconfigured in HA
Closes-Bug: #1606771 - Central Keystone provision broken by nova_config dependency

Three issues:
1. In single node setups, heat wasn't working in multi interface setups as the bind host was wrong
In HA and single node setups, heat stack-create wasn't working because of wrong auth_uri
2. Added some missing/misconfigured params after comparing with fab provisioned heat
3. Central Keystone provision got into dependency cycle due to addition of notify on nova_config.
This has been replaced with exec to do nova-compute restart

Patch 2: Comma missing

Change-Id: Id9e20db5d68b9608359e6ffb2957874980bd7353

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-08-05: [Review update] master

Review in progress for https://review.opencontrail.org/22979
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message

sundarkh (sundar-kh) wrote on 2016-08-06: Re: SMR3.1build14:creation of SVC with heat fails with authorisation failure The resource could not be found. (HTTP 404)

#11

Download full text (4.4 KiB)

Seen in BUild 17

Seen in BUild 17

2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher [req-fccc4a83-0f7e-4771-b925-cda3de78247c 523d549113a44501a3687ea6a54a7b8d a30c0e6322de4512a086c63816a34ebb] Exception during message handling: The resource could not be found. (HTTP 404) (HTTP 404)
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
Broadcast message from root@nodeg37oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-08-(unknown) at 6:45 ...ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
The system is going down for power off NOW!saging.rpc.dispatcher     return f(*args, **kwargs)
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/heat/common/context.py", line 308, in wrapped
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher     return func(self, ctx, *args, **kwargs)
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 729, in create_stack
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher     stack_id = stack.store()
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 524, in store
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher     trust_ctx = keystone.create_trust_context()
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 210, in create_trust_context
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher     self.session)
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 352, in get_user_id
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher     return self.get_access(session).user_id
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher     self.auth_ref = self.get_auth_ref(session)
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher     authenticated=False, log=False, **rkwargs)
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 501, in post
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher     return self.request(url, 'POST', **kwargs)
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 401, in request
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher     raise exceptions.from_response(resp, method, url)
2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher NotFound: The resource could not be found. (HTTP 404)2016-08-06 03:40:27.557 6808 ERROR oslo_messaging.rpc.dispatcher

Revision history for this message

sundarkh (sundar-kh) wrote on 2016-08-06:

#12

Traceback (most recent call last):
  File "tcutils/wrappers.py", line 73, in wrapper
    result = function(self, *args, **kwargs)
  File "scripts/heat/test_heat.py", line 61, in test_svc_creation_with_heat
    right_net_fix, r_hs_obj = self.config_vn(stack_name='right_net')
  File "scripts/heat/base.py", line 103, in config_vn
    vn_hs_obj = self.config_heat_obj(stack_name, template, env)
  File "scripts/heat/base.py", line 117, in config_heat_obj
    inputs=self.inputs, stack_name=stack_name, project_fq_name=self.inputs.project_fq_name, template=template, env=env))
  File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 679, in useFixture
    reraise(*exc_info)
  File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 666, in useFixture
    fixture.setUp()
  File "/contrail-test/fixtures/heat_test.py", line 120, in setUp
    stack_obj = self.heat_client_obj.stacks.create(**fields)
  File "/usr/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 136, in create
    data=kwargs, headers=headers)
  File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 287, in post
    return self.client_request("POST", url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 277, in client_request
    resp, body = self.json_request(method, url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 266, in json_request
    resp = self._http_request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 221, in _http_request
    raise exc.from_response(resp)
HTTPInternalServerError: ERROR: Remote error: NotFound The resource could not be found. (HTTP 404)
[u'

Revision history for this message

sundarkh (sundar-kh) wrote on 2016-08-06:

#13

Please ignore the previous comment, this trace was on build 15

Revision history for this message

sundarkh (sundar-kh) wrote on 2016-08-08:

#14

Download full text (6.5 KiB)

With build 17 issue is different from what seen in build 15
------------------------------------------------------------

With R3.1 build 17 mitaka, svc creation using heat test cases fails with following trace

Traceback (most recent call last):

File "tcutils/wrappers.py", line 73, in wrapper

result = function(self, *args, **kwargs)

File "scripts/heat/test_heat.py", line 61, in test_svc_creation_with_heat

right_net_fix, r_hs_obj = self.config_vn(stack_name='right_net')

File "scripts/heat/base.py", line 103, in config_vn

vn_hs_obj = self.config_heat_obj(stack_name, template, env)

File "scripts/heat/base.py", line 117, in config_heat_obj

inputs=self.inputs, stack_name=stack_name, project_fq_name=self.inputs.project_fq_name, template=template, env=env))

File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 679, in useFixture

reraise(*exc_info)

File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 666, in useFixture

fixture.setUp()

File "/contrail-test/fixtures/heat_test.py", line 120, in setUp

stack_obj = self.heat_client_obj.stacks.create(**fields)

File "/usr/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 162, in create

data=kwargs, headers=headers)

File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 281, in post

return self.client_request("POST", url, **kwargs)

File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 271, in client_request

resp, body = self.json_request(method, url, **kwargs)

File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 260, in json_request

resp = self._http_request(url, method, **kwargs)

File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 215, in _http_request

raise exc.from_response(resp)

HTTPInternalServerError: ERROR: Authorization failed.

Heat-api.log
-------------------
essaging/_drivers/amqpdriver.py:302
2016-08-06 23:54:36.334 15872 DEBUG heat.common.serializers [req-281ec965-6f3d-45bb-97be-c2461fb6718a - ctest-TestHeatv2-67516137 - 3cc68fecba8c4ef7b3179aef9f3e54ea 3cc68fecba8c4ef7b3179aef9f3e54ea] JSON response : {"explanation": "The server has either erred or is incapable of performing the requested operation.", "code": 500, "error": {"message": "Authorization failed.", "traceback": "Traceback (most recent call last):\n\n File \"/usr/lib/python2.7/dist-packages/heat/common/context.py\", line 329, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File \"/usr/lib/python2.7/dist-packages/heat/engine/service.py\", line 802, in create_stack\n stack_id = stack.store()\n\n File \"/usr/lib/python2.7/dist-packages/osprofiler/profiler.py\", line 117, in wrapper\n return f(*args, **kwargs)\n\n File \"/usr/lib/python2.7/dist-packages/heat/engine/stack.py\", line 597, in store\n trust_ctx = keystone.create_trust_context()\n\n File \"/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py\", line 201, in create_trust_context\n raise exception.AuthorizationFailure()\n\nAuthorizationFailure: Authorization failed.\n", "type": "Authorization...

With build 17 issue is different from what seen in build 15
------------------------------------------------------------

With R3.1 build 17 mitaka, svc creation using heat test cases fails with following trace
 
Traceback (most recent call last):
 
  File "tcutils/wrappers.py", line 73, in wrapper
 
    result = function(self, *args, **kwargs)
 
  File "scripts/heat/test_heat.py", line 61, in test_svc_creation_with_heat
 
    right_net_fix, r_hs_obj = self.config_vn(stack_name='right_net')
 
  File "scripts/heat/base.py", line 103, in config_vn
 
    vn_hs_obj = self.config_heat_obj(stack_name, template, env)
 
  File "scripts/heat/base.py", line 117, in config_heat_obj
 
    inputs=self.inputs, stack_name=stack_name, project_fq_name=self.inputs.project_fq_name, template=template, env=env))
 
  File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 679, in useFixture
 
    reraise(*exc_info)
 
  File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 666, in useFixture
 
    fixture.setUp()
 
  File "/contrail-test/fixtures/heat_test.py", line 120, in setUp
 
    stack_obj = self.heat_client_obj.stacks.create(**fields)
 
  File "/usr/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 162, in create
 
    data=kwargs, headers=headers)
 
  File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 281, in post
 
    return self.client_request("POST", url, **kwargs)
 
  File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 271, in client_request
 
    resp, body = self.json_request(method, url, **kwargs)
 
  File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 260, in json_request
 
    resp = self._http_request(url, method, **kwargs)
 
  File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 215, in _http_request
 
    raise exc.from_response(resp)
 
HTTPInternalServerError: ERROR: Authorization failed.
 
Heat-api.log
-------------------
essaging/_drivers/amqpdriver.py:302
2016-08-06 23:54:36.334 15872 DEBUG heat.common.serializers [req-281ec965-6f3d-45bb-97be-c2461fb6718a - ctest-TestHeatv2-67516137 - 3cc68fecba8c4ef7b3179aef9f3e54ea 3cc68fecba8c4ef7b3179aef9f3e54ea] JSON response : {"explanation": "The server has either erred or is incapable of performing the requested operation.", "code": 500, "error": {"message": "Authorization failed.", "traceback": "Traceback (most recent call last):\n\n  File \"/usr/lib/python2.7/dist-packages/heat/common/context.py\", line 329, in wrapped\n    return func(self, ctx, *args, **kwargs)\n\n  File \"/usr/lib/python2.7/dist-packages/heat/engine/service.py\", line 802, in create_stack\n    stack_id = stack.store()\n\n  File \"/usr/lib/python2.7/dist-packages/osprofiler/profiler.py\", line 117, in wrapper\n    return f(*args, **kwargs)\n\n  File \"/usr/lib/python2.7/dist-packages/heat/engine/stack.py\", line 597, in store\n    trust_ctx = keystone.create_trust_context()\n\n  File \"/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py\", line 201, in create_trust_context\n    raise exception.AuthorizationFailure()\n\nAuthorizationFailure: Authorization failed.\n", "type": "AuthorizationFailure"}, "title": "Internal Server Error"} to_json /usr/lib/python2.7/dist-packages/heat/common/serializers.py:40
2016-08-06 23:54:36.335 15872 INFO eventlet.wsgi.server [req-281ec965-6f3d-45bb-97be-c2461fb6718a - ctest-TestHeatv2-67516137 - 3cc68fecba8c4ef7b3179aef9f3e54ea 3cc68fecba8c4ef7b3179aef9f3e54ea] 172.17.0.2 - - [06/Aug/2016 23:54:36] "POST /v1/1a39d39d3e424dd1a3c0413715b2fa97/stacks HTTP/1.1" 500 1222 1.418896
 
root@nodeg37:~# keystone role-list
/usr/lib/python2.7/dist-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
  'python-keystoneclient.', DeprecationWarning)
/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py:145: DeprecationWarning: Constructing an instance of the keystoneclient.v2_0.client.Client class without a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py:147: DeprecationWarning: Using the 'tenant_name' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_name' argument instead
  super(Client, self).__init__(**kwargs)
/usr/lib/python2.7/dist-packages/debtcollector/renames.py:45: DeprecationWarning: Using the 'tenant_id' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_id' argument instead
  return f(*args, **kwargs)
/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py:371: DeprecationWarning: Constructing an HTTPClient instance without using a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
 'the 2.0.0 release.', DeprecationWarning)
/usr/lib/python2.7/dist-packages/keystoneclient/session.py:140: DeprecationWarning: keystoneclient.session.Session is deprecated as of the 2.1.0 release in favor of keystoneauth1.session.Session. It will be removed in future releases.
  DeprecationWarning)
/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py:56: DeprecationWarning: keystoneclient auth plugins are deprecated as of the 2.1.0 release in favor of keystoneauth1 plugins. They will be removed in future releases.
  'in future releases.', DeprecationWarning)
+----------------------------------+------------------+
|                id                |       name       |
+----------------------------------+------------------+
| 1a82badc54a741afb61cd33c4b54e348 |  ResellerAdmin   |
| 9fe2ff9ee4384b1894a90878d3e92bab |     _member_     |
| 2f781d9d085242a7a9c59a3fbc8dfa11 |      admin       |
| 8d1e3313711049d2aa85d3453ce148de | heat_stack_owner |
| 77d2ba343e19428ca9be62481c30a3a9 | heat_stack_user  |
+----------------------------------+------------------+
root@nodeg37:~# heat stack-list
+----+------------+--------------+---------------+--------------+
| id | stack_name | stack_status | creation_time | updated_time |
+----+------------+--------------+---------------+--------------+
+----+------------+--------------+---------------+--------------+
root@nodeg37:~#
 
~          
The above mentioned trace with build 17 is slightly different than the one mentioned in  1609683 (which was :  AuthorizationFailure: Authorization failed: The resource could not be found. (HTTP 404) (HTTP 404))

Revision history for this message

Nitish Krishna Kaveri (nitishk) wrote on 2016-08-09: Re: SMR3.1build18:creation of SVC with heat fails with authorisation failure in Mitaka

#15

issue seen in r3.1 18 Mitaka but not Liberty

Command issued:
heat --debug stack-create mystack -f /root/virtual_network_with_subnet.yaml -e /root/virtual_network_with_subnet.env

summary:

- SMR3.1build14:creation of SVC with heat fails with authorisation
- failure The resource could not be found. (HTTP 404)
+ SMR3.1build18:creation of SVC with heat fails with authorisation
+ failure in Mitaka

Revision history for this message

Abhay Joshi (abhayj) wrote on 2016-08-09:

#16

We can release note since it is not working for mitaka alone, as we continue to look into it.

tags:

added: releasenote

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-08-10: [Review update] master

#17

Review in progress for https://review.opencontrail.org/22979
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-08-11: A change has been merged

#19

Reviewed: https://review.opencontrail.org/22979
Committed: http://github.org/Juniper/contrail-puppet/commit/0781bd299aa7f17d5c2f3c15bba0ca851e56855e
Submitter: Zuul
Branch: master

commit 0781bd299aa7f17d5c2f3c15bba0ca851e56855e
Author: nitishkrishna <email address hidden>
Date: Thu Aug 4 14:21:14 2016 -0700

Patch 2: Comma missing

Patch 3: Conflicts
Change-Id: Id9e20db5d68b9608359e6ffb2957874980bd7353

Revision history for this message

sundarkh (sundar-kh) wrote on 2016-08-12: Re: SMR3.1build18:creation of SVC with heat fails with authorisation failure in Mitaka

#20

Issue seen with kilo build 22 as well

Traceback (most recent call last):

File "tcutils/wrappers.py", line 73, in wrapper

result = function(self, *args, **kwargs)

File "tcutils/util.py", line 967, in wrapper

return f(self, *func_args, **func_kwargs)

File "scripts/heat/test_heat.py", line 103, in test_public_access_thru_svc_w_fip

right_net_fix, r_hs_obj = self.config_vn(stack_name='right_net')

File "scripts/heat/base.py", line 103, in config_vn

vn_hs_obj = self.config_heat_obj(stack_name, template, env)

File "scripts/heat/base.py", line 117, in config_heat_obj

inputs=self.inputs, stack_name=stack_name, project_fq_name=self.inputs.project_fq_name, template=template, env=env))

File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 679, in useFixture

reraise(*exc_info)

File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 666, in useFixture

fixture.setUp()

File "/contrail-test/fixtures/heat_test.py", line 120, in setUp

stack_obj = self.heat_client_obj.stacks.create(**fields)

File "/usr/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 162, in create

data=kwargs, headers=headers)

File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 281, in post

return self.client_request("POST", url, **kwargs)

File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 271, in client_request

resp, body = self.json_request(method, url, **kwargs)

File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 260, in json_request

resp = self._http_request(url, method, **kwargs)

File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 215, in _http_request

raise exc.from_response(resp)

HTTPInternalServerError: ERROR: Authorization failed.

Revision history for this message

Praneet Bachheti (praneetb) wrote on 2016-09-01:

#21

keystone issue in server manager. Dheeraj has the patch to fix it.

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-09-06: [Review update] R3.1

#22

Review in progress for https://review.opencontrail.org/23929
Submitter: Dheeraj Gautam (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-09-07: [Review update] master

#26

Review in progress for https://review.opencontrail.org/23942
Submitter: Dheeraj Gautam (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-09-07: A change has been merged

#28

Reviewed: https://review.opencontrail.org/23929
Committed: http://github.org/Juniper/contrail-puppet/commit/7e85152254a68009e93de724ae59d6a96f67f107
Submitter: Zuul
Branch: R3.1

commit 7e85152254a68009e93de724ae59d6a96f67f107
Author: Dheeraj Gautam <email address hidden>
Date: Tue Sep 6 12:11:20 2016 -0700

SM-Mitaka: bootstrap keystone for default domain id

Closes-Bug: #1609683

1. enabled bootstrap for keystone domain. This is borrowed from mitaka
puppet-keystone module.
2. remove code for sync-ing default_domain_id among HA.

Change-Id: I5264893e78b8bec4d26b59a4feefcc085334b47a

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-09-07: [Review update] master

#29

Review in progress for https://review.opencontrail.org/23942
Submitter: Dheeraj Gautam (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-09-08: A change has been merged

#31

Reviewed: https://review.opencontrail.org/23942
Committed: http://github.org/Juniper/contrail-puppet/commit/e1bff262dab997f3b482a77dbabdf26d7f611fa5
Submitter: Zuul
Branch: master

commit e1bff262dab997f3b482a77dbabdf26d7f611fa5
Author: Dheeraj Gautam <email address hidden>
Date: Tue Sep 6 12:11:20 2016 -0700

SM-Mitaka: bootstrap keystone for default domain id

Closes-Bug: #1609683

1. enabled bootstrap for keystone domain. This is borrowed from mitaka
puppet-keystone module.
2. remove code for sync-ing default_domain_id among HA.

Conflicts:
contrail/environment/modules/contrail/manifests/profile/openstack/keystone.pp

Change-Id: I5264893e78b8bec4d26b59a4feefcc085334b47a

Revision history for this message

sundarkh (sundar-kh) wrote on 2017-02-23: Re: SMR3.1build18:creation of SVC with heat fails with authorisation failure in Mitaka

#32

Download full text (5.7 KiB)

R3.0.3.x Juno/Kilo - Seeing n Openstack HA setup only

reimage is successful

Provision also gets completed;

But,

1) Keystone is pointing to one of the control node ip , and not the external vip

VIP address in not available in keystone end-point list

syslog during provision of openstack nodes

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.