SM:R3.2 build 16: Creation of heat stack fails with keystone authentication
1) Provision a node with R3.2 liberty SM build 16
2) Try to create a heat stack using
heat -v -d stack-create test -f /usr/lib/python2.7/dist-packages/contrail_heat/new_templates/virtual_network_with_subnet.yaml -e /usr/lib/python2.7/dist-packages/contrail_heat/new_envs/virtual_network_with_subnet.env
3) Observe that it fails
heatclient.exc.HTTPInternalServerError: ERROR: Remote error: NotFound The resource could not be found. (HTTP 404)
[u'
4) /etc/heat/heat.conf has
plugin_dirs=/usr/lib/python2.7/dist-packages/vnc_api/gen/heat/resources,/usr/lib/python2.7/dist-packages/contrail_heat/resources
5) keystone service-list
id | name | type | description |
+----------------------------------+------------+----------------+----------------------------------+
| d2d5dd623f424786ba7cef4314efc576 | ceilometer | metering | Openstack Metering Service |
| 6ec76ca5d189443d95fe91ff76d73ec2 | cinder | volume | Cinder Service |
| 6bf3d2488ba749b6bed1995e7a67466d | cinderv2 | volumev2 | Cinder Service v2 |
| 1919a99e85e447a088b4478a109016e7 | glance | image | OpenStack Image Service |
| a7f40f5204774df88cadbb22e3fcfbc2 | heat | orchestration | Openstack Orchestration Service |
| b0bb040c787d43a9b163971c6e8177d1 | heat-cfn | cloudformation | Openstack Cloudformation Service |
| f9cca5e3672242de996b1f7fddd1a934 | keystone | identity | OpenStack Identity Service |
| 9e17fd3a1b464537b831cf853a835a2a | neutron | network | Neutron Networking Service |
| 09e4ee380fe7493998e51bc862b782c7 | nova | compute | Openstack Compute Service |
| 26588b79087c4584bee0093aa7c26d98 | nova_ec2 | ec2 | EC2 Service |
| c6ff8544f3134d208d92c00b9be2820b | novav3 | computev3 | Openstack Compute Service v3 |
+----------------------------------+------------+----------------+----------------------------------+
6) ps -aux | grep heat
root 7257 0.0 0.0 10460 936 pts/8 S+ 04:15 0:00 grep --color=auto heat
heat 10626 0.0 0.0 268164 62340 ? S Dec15 0:01 /usr/bin/python /usr/bin/heat-api-cfn
heat 10628 0.7 0.1 294496 88576 ? S Dec15 2:17 /usr/bin/python /usr/bin/heat-engine
heat 10630 1.4 0.1 279776 74164 ? S Dec15 4:39 /usr/bin/python /usr/bin/heat-api
heat 10834 1.5 0.1 325860 106660 ? S Dec15 4:50 /usr/bin/python /usr/bin/heat-engine
heat 10835 1.5 0.1 325472 106188 ? S Dec15 4:48 /usr/bin/python /usr/bin/heat-engine
heat 10836 1.5 0.1 325720 106508 ? S Dec15 4:48 /usr/bin/python /usr/bin/heat-engine
heat 10837 1.5 0.1 324908 105616 ? S Dec15 4:49 /usr/bin/python /usr/bin/heat-engine
root@nodeh8:~#
Heat engine log
---------------
] Making authentication request to http://10.204.217.112:5000/auth/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3/base.py:188
2016-12-16 03:54:03.586 10834 DEBUG keystoneclient.session [req-86ff4eb3-8492-47b4-9c1f-6636e3fc5c80 ac64bbb1ef6645b482622db280eea812 753c118a86fe4bc6af41fe07f9b9fa73] Request returned failure status: 404 request /usr/lib/python2.7/dist-packages/keystoneclient/session.py:400
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher [req-86ff4eb3-8492-47b4-9c1f-6636e3fc5c80 ac64bbb1ef6645b482622db280eea812 753c118a86fe4bc6af41fe07f9b9fa73] Exception during message handling: The resource could not be found. (HTTP 404) (HTTP 404)
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/common/context.py", line 308, in wrapped
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher return func(self, ctx, *args, **kwargs)
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 729, in create_stack
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher stack_id = stack.store()
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 524, in store
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher trust_ctx = keystone.create_trust_context()
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 210, in create_trust_context
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher self.session)
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 352, in get_user_id
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher return self.get_access(session).user_id
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher self.auth_ref = self.get_auth_ref(session)
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher authenticated=False, log=False, **rkwargs)
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 501, in post
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher return self.request(url, 'POST', **kwargs)
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher return func(*args, **kwargs)
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 401, in request
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher raise exceptions.from_response(resp, method, url)
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher NotFound: The resource could not be found. (HTTP 404)
2016-12-16 03:54:03.592 10834 ERROR oslo_messaging.rpc.dispatcher
2016-12-16 03:54:03.615 10834 ERROR oslo_messaging._drivers.common [req-86ff4eb3-8492-47b4-9c1f-6636e3fc5c80 ac64bbb1ef6645b482622db280eea812 753c118a86fe4bc6af41fe07f9b9fa73] Returning exception The resource could not be found. (HTTP 404) to caller
2016-12-16 03:54:03.617 10834 ERROR oslo_messaging._drivers.common [req-86ff4eb3-8492-47b4-9c1f-6636e3fc5c80 ac64bbb1ef6645b482622db280eea812 753c118a86fe4bc6af41fe07f9b9fa73] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/heat/common/context.py", line 308, in wrapped\n return func(self, ctx, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 729, in create_stack\n stack_id = stack.store()\n', ' File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 524, in store\n trust_ctx = keystone.create_trust_context()\n', ' File "/usr/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 210, in create_trust_context\n self.session)\n', ' File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 352, in get_user_id\n return self.get_access(session).user_id\n', ' File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 240, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', ' File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref\n authenticated=False, log=False, **rkwargs)\n', ' File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 501, in post\n return self.request(url, \'POST\', **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner\n return func(*args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 401, in request\n raise exceptions.from_response(resp, method, url)\n', 'NotFound: The resource could not be found. (HTTP 404)\n']
Notes:
-------
Issue seen with kilo, liberty .
Creation of heat stack works fine in Mitaka
Review in progress for https:/
Submitter: Dheeraj Gautam (<email address hidden>)