Cluster creation Error : EndpointNotFound: Could not find requested endpoint in Service Catalog.

Bug #1656247 reported by sven mark
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Sahara
Invalid
Undecided
sven mark

Bug Description

sahara error log as follow:

Error during operating on cluster (reason: Could not find requested endpoint in Service Catalog.)
Traceback (most recent call last):
  File "/opt/stack/sahara/sahara/service/ops.py", line 193, in wrapper
    f(cluster_id, *args, **kwds)
  File "/opt/stack/sahara/sahara/service/ops.py", line 286, in _provision_cluster
    INFRA.create_cluster(cluster)
  File "/opt/stack/sahara/sahara/service/heat/heat_engine.py", line 63, in create_cluster
    self._launch_instances(cluster, target_count, CREATE_STAGES)
  File "/opt/stack/sahara/sahara/service/heat/heat_engine.py", line 230, in _launch_instances
    cluster, target_count, update_stack, disable_rollback)
  File "/opt/stack/sahara/sahara/utils/cluster_progress_ops.py", line 139, in handler
    add_fail_event(instance, e)
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/sahara/sahara/utils/cluster_progress_ops.py", line 136, in handler
    value = func(*args, **kwargs)
  File "/opt/stack/sahara/sahara/service/heat/heat_engine.py", line 218, in _create_instances
    disable_rollback=disable_rollback)
  File "/opt/stack/sahara/sahara/service/heat/templates.py", line 207, in instantiate
    b.execute_with_retries(h.client().stacks.create, **kwargs)
  File "/opt/stack/sahara/sahara/utils/openstack/base.py", line 113, in execute_with_retries
    raise e
EndpointNotFound: Could not find requested endpoint in Service Catalog.

cluster status is Error,and the resean is Could not find requested endpoint in Service Catalog.

sven mark (svenmark)
Changed in sahara:
assignee: nobody → sven mark (svenmark)
Revision history for this message
sven mark (svenmark) wrote :

resean :lack endpoint_filter in the /usr/lib/python2.7/site-packages/keystoneauth1/session.py 463
patch:
/usr/lib/python2.7/site-packages/heatclient/common/http.py 308 add:
self.service_type='orchestration'
self.interface='publicURL'

Revision history for this message
Luigi Toscano (ltoscano) wrote :

Could you please share more details about the environment? How was openstack (and sahara) installed and configured, which version, on which platform...? The more the better.

Revision history for this message
Vitalii Gridnev (vgridnev) wrote :

You can configure endpoint_type for heat if needed in sahara.conf file: https://github.com/openstack/sahara/blob/master/sahara/utils/openstack/heat.py#L34

Changed in sahara:
status: New → Incomplete
Revision history for this message
sven mark (svenmark) wrote :

isntall: openstack is installed by devstack on centos 7 1503-1
heat and sahara is configed by default
version is the master brach

Revision history for this message
Vitalii Gridnev (vgridnev) wrote :

And again, what is the default configuration? As I see, you probably did not created endpoints, or you did not said sahara to use publicURL as endpoint type (by configuration file). Please provide **detailed** description of the issue, because everything works perfectly on my environments.

Thanks.

Revision history for this message
sven mark (svenmark) wrote :

[keystone_authtoken]
admin_password = ******
admin_user = sahara
admin_tenant_name = service
memcached_servers = **********
signing_dir = /var/cache/sahara
cafile = /opt/stack/data/ca-bundle.pem
auth_uri = http://******/identity
project_domain_name = Default
project_name = service
user_domain_name = Default
password = ******
username = sahara
auth_url = http://******/identity_admin
auth_type = password

[DEFAULT]
periodic_coordinator_backend_url = kazoo://127.0.0.1:2181
logging_exception_prefix = %(color)s%(asctime)s.%(msecs)03d TRACE %(name)s %(instance)s
logging_debug_format_suffix = from (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d
logging_default_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s
logging_context_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [%(request_id)s %(user_name)s %(project_name)s%(color)s] %(instance)s%(color)s%(message)s
use_syslog = False
use_neutron = true
plugins = vanilla,cdh,ambari,mapr,spark,storm,fake
debug = True
transport_url = rabbit://stackrabbit:ubuntu@******:5672/

[database]
connection = mysql+pymysql://root:ubuntu@******/sahara?charset=utf8

[neutron]
endpoint_type = admin

[heat]
endpoint_type = admin

[cinder]
endpoint_type = admin

[nova]
endpoint_type = admin

[swift]
endpoint_type = admin

[keystone]
endpoint_type = admin

[glance]
endpoint_type = admin

Revision history for this message
sven mark (svenmark) wrote :

here is the stack:
/usr/lib/python2.7/site-packages/futurist/_green.py(70)__call__()
-> self.work.run()
  /usr/lib/python2.7/site-packages/futurist/_utils.py(46)run()
-> result = self.fn(*self.args, **self.kwargs)
  /usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py(155)_process_incoming()
-> res = self.dispatcher.dispatch(message)
  /usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py(225)dispatch()
-> return self._do_dispatch(endpoint, method, ctxt, args)
  /usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py(195)_do_dispatch()
-> result = func(ctxt, **new_args)
  /opt/stack/sahara/sahara/service/ops.py(130)wrapped()
-> return func(self, *args, **kwargs)
  /opt/stack/sahara/sahara/service/ops.py(144)provision_cluster()
-> _provision_cluster(cluster_id)
  /opt/stack/sahara/sahara/service/ops.py(193)wrapper()
-> f(cluster_id, *args, **kwds)
  /opt/stack/sahara/sahara/service/ops.py(286)_provision_cluster()
-> INFRA.create_cluster(cluster)
  /opt/stack/sahara/sahara/service/heat/heat_engine.py(62)create_cluster()
-> self._launch_instances(cluster, target_count, CREATE_STAGES)
  /opt/stack/sahara/sahara/service/heat/heat_engine.py(229)_launch_instances()
-> cluster, target_count, update_stack, disable_rollback)
  /opt/stack/sahara/sahara/utils/cluster_progress_ops.py(136)handler()
-> value = func(*args, **kwargs)
  /opt/stack/sahara/sahara/service/heat/heat_engine.py(217)_create_instances()
-> disable_rollback=disable_rollback)
  /opt/stack/sahara/sahara/service/heat/templates.py(207)instantiate()
-> b.execute_with_retries(h.client().stacks.create, **kwargs)
  /opt/stack/sahara/sahara/utils/openstack/base.py(93)execute_with_retries()
-> return method(*args, **kwargs)
  /usr/lib/python2.7/site-packages/heatclient/v1/stacks.py(172)create()
-> data=kwargs, headers=headers)
  /usr/lib/python2.7/site-packages/keystoneauth1/adapter.py(222)post()
-> return self.request(url, 'POST', **kwargs)
  /usr/lib/python2.7/site-packages/heatclient/common/http.py(314)request()
-> **kwargs)
  /usr/lib/python2.7/site-packages/keystoneauth1/adapter.py(373)request()
-> resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  /usr/lib/python2.7/site-packages/keystoneauth1/adapter.py(141)request()
-> return self.session.request(url, method, **kwargs)
> /usr/lib/python2.7/site-packages/positional/__init__.py(101)inner()
-> return wrapped(*args, **kwargs)
  /usr/lib/python2.7/site-packages/keystoneauth1/session.py(463)request()
-> headers = kwargs.setdefault('headers', dict())

 if you read the code from /opt/stack/sahara/sahara/service/heat/templates.py(207)instantiate(),you will find that this bug has none busness with configuration file .

Revision history for this message
sven mark (svenmark) wrote : Re: [Bug 1656247] Re: Cluster creation Error : EndpointNotFound: Could not find requested endpoint in Service Catalog.
Download full text (5.4 KiB)

here is the stack:
/usr/lib/python2.7/site-packages/futurist/_green.py(70)__call__()
-> self.work.run()
  /usr/lib/python2.7/site-packages/futurist/_utils.py(46)run()
-> result = self.fn(*self.args, **self.kwargs)
  /usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py(155)_process_incoming()
-> res = self.dispatcher.dispatch(message)
  /usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py(225)dispatch()
-> return self._do_dispatch(endpoint, method, ctxt, args)
  /usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py(195)_do_dispatch()
-> result = func(ctxt, **new_args)
  /opt/stack/sahara/sahara/service/ops.py(130)wrapped()
-> return func(self, *args, **kwargs)
  /opt/stack/sahara/sahara/service/ops.py(144)provision_cluster()
-> _provision_cluster(cluster_id)
  /opt/stack/sahara/sahara/service/ops.py(193)wrapper()
-> f(cluster_id, *args, **kwds)
  /opt/stack/sahara/sahara/service/ops.py(286)_provision_cluster()
-> INFRA.create_cluster(cluster)
  /opt/stack/sahara/sahara/service/heat/heat_engine.py(62)create_cluster()
-> self._launch_instances(cluster, target_count, CREATE_STAGES)
  /opt/stack/sahara/sahara/service/heat/heat_engine.py(229)_launch_instances()
-> cluster, target_count, update_stack, disable_rollback)
  /opt/stack/sahara/sahara/utils/cluster_progress_ops.py(136)handler()
-> value = func(*args, **kwargs)
  /opt/stack/sahara/sahara/service/heat/heat_engine.py(217)_create_instances()
-> disable_rollback=disable_rollback)
  /opt/stack/sahara/sahara/service/heat/templates.py(207)instantiate()
-> b.execute_with_retries(h.client().stacks.create, **kwargs)
  /opt/stack/sahara/sahara/utils/openstack/base.py(93)execute_with_retries()
-> return method(*args, **kwargs)
  /usr/lib/python2.7/site-packages/heatclient/v1/stacks.py(172)create()
-> data=kwargs, headers=headers)
  /usr/lib/python2.7/site-packages/keystoneauth1/adapter.py(222)post()
-> return self.request(url, 'POST', **kwargs)
  /usr/lib/python2.7/site-packages/heatclient/common/http.py(314)request()
-> **kwargs)
  /usr/lib/python2.7/site-packages/keystoneauth1/adapter.py(373)request()
-> resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  /usr/lib/python2.7/site-packages/keystoneauth1/adapter.py(141)request()
-> return self.session.request(url, method, **kwargs)
> /usr/lib/python2.7/site-packages/positional/__init__.py(101)inner()
-> return wrapped(*args, **kwargs)
  /usr/lib/python2.7/site-packages/keystoneauth1/session.py(463)request()
-> headers = kwargs.setdefault('headers', dict())
 if you read the code from /opt/stack/sahara/sahara/service/heat/templates.py(207)instantiate(),you will find that this bug has none busness with configuration file

<email address hidden>

From: Vitaly Gridnev
Date: 2017-01-13 19:53
To: masw
Subject: [Bug 1656247] Re: Cluster creation Error : EndpointNotFound: Could not find requested endpoint in Service Catalog.
And again, what is the default configuration? As I see, you probably did
not created endpoints, or you did not said sahara to use publicURL as
endpoint type (by configuration file). Please provide **detailed**
description of the issue, bec...

Read more...

Revision history for this message
Vitalii Gridnev (vgridnev) wrote :

I Don't see bug that in the instantiate method, can you describe precisely?

Additionally how environment was installed? If that is devstack, please ensure that heat devstack plugin was included in your local.conf

Revision history for this message
sven mark (svenmark) wrote :
Download full text (3.4 KiB)

yes ,used devstack
I include sahara plugin in the local.conf
enable_plugin sahara git://git.openstack.org/openstack/sahara
enable_plugin sahara-dashboard git://git.openstack.org/openstack/sahara-dashboard

and heat plugin is enabled by
/opt/stack/sahara/devstack/settings 42
so it is not included in the local.conf file.

/opt/stack/sahara/sahara/service/heat/templates.py(207)instantiate()
the code like this
        if not update_existing:
            LOG.debug("Creating Heat stack with args: \n{args}"
                      .format(args=log_kwargs))
            b.execute_with_retries(h.client().stacks.create, **kwargs)
after b.execute_with_retries(h.client().stacks.create, **kwargs), endpoint_type and service_type is not set,and the url is /stack,so when http requests ,the endpoinbt exception happened.

<email address hidden>

From: Vitaly Gridnev
Date: 2017-01-16 14:21
To: masw
Subject: [Bug 1656247] Re: Cluster creation Error : EndpointNotFound: Could not find requested endpoint in Service Catalog.
I Don't see bug that in the instantiate method, can you describe
precisely?

Additionally how environment was installed? If that is devstack, please
ensure that heat devstack plugin was included in your local.conf

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1656247

Title:
  Cluster creation Error : EndpointNotFound: Could not find requested
  endpoint in Service Catalog.

Status in Sahara:
  Incomplete

Bug description:
  sahara error log as follow:

  Error during operating on cluster (reason: Could not find requested endpoint in Service Catalog.)
  Traceback (most recent call last):
    File "/opt/stack/sahara/sahara/service/ops.py", line 193, in wrapper
      f(cluster_id, *args, **kwds)
    File "/opt/stack/sahara/sahara/service/ops.py", line 286, in _provision_cluster
      INFRA.create_cluster(cluster)
    File "/opt/stack/sahara/sahara/service/heat/heat_engine.py", line 63, in create_cluster
      self._launch_instances(cluster, target_count, CREATE_STAGES)
    File "/opt/stack/sahara/sahara/service/heat/heat_engine.py", line 230, in _launch_instances
      cluster, target_count, update_stack, disable_rollback)
    File "/opt/stack/sahara/sahara/utils/cluster_progress_ops.py", line 139, in handler
      add_fail_event(instance, e)
    File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      self.force_reraise()
    File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      six.reraise(self.type_, self.value, self.tb)
    File "/opt/stack/sahara/sahara/utils/cluster_progress_ops.py", line 136, in handler
      value = func(*args, **kwargs)
    File "/opt/stack/sahara/sahara/service/heat/heat_engine.py", line 218, in _create_instances
      disable_rollback=disable_rollback)
    File "/opt/stack/sahara/sahara/service/heat/templates.py", line 207, in instantiate
      b.execute_with_retries(h.client().stacks.create, **kwargs)
    File "/opt/stack/sahara/sahara/utils/openstack/base.py", line 113, in execute_with_retries
    ...

Read more...

Revision history for this message
Vitalii Gridnev (vgridnev) wrote :
Revision history for this message
Vitalii Gridnev (vgridnev) wrote :

So, when we are creating client, it already has valid endpoint type

Revision history for this message
Vitalii Gridnev (vgridnev) wrote :

and enable heat plugin, please. This ([0]) doesn't deploys heat service, so you have to separately enable heat.

[0] https://github.com/openstack/sahara/blob/master/devstack/settings#L43

Changed in sahara:
status: Incomplete → Invalid
Revision history for this message
sven mark (svenmark) wrote :
Download full text (3.7 KiB)

look at here,from stacks.py(172) resp = self.client.post('/stacks',
                                data=kwargs, headers=headers)

the client is not heat.client(),so and endpointurl is not get,and endpoint_type and service_type is not set.

  /opt/stack/sahara/sahara/service/heat/templates.py(207)instantiate()
-> b.execute_with_retries(h.client().stacks.create, **kwargs)
  /opt/stack/sahara/sahara/utils/openstack/base.py(93)execute_with_retries()
-> return method(*args, **kwargs)
  /usr/lib/python2.7/site-packages/heatclient/v1/stacks.py(172)create()
-> data=kwargs, headers=headers)
  /usr/lib/python2.7/site-packages/keystoneauth1/adapter.py(222)post()
-> return self.request(url, 'POST', **kwargs)
  /usr/lib/python2.7/site-packages/heatclient/common/http.py(314)request()
-> **kwargs)
  /usr/lib/python2.7/site-packages/keystoneauth1/adapter.py(373)request()
-> resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  /usr/lib/python2.7/site-packages/keystoneauth1/adapter.py(141)request()
-> return self.session.request(url, method, **kwargs)
> /usr/lib/python2.7/site-packages/positional/__init__.py(101)inner()
-> return wrapped(*args, **kwargs)
  /usr/lib/python2.7/site-packages/keystoneauth1/session.py(463)request()
-> headers = kwargs.setdefault('headers', dict())

<email address hidden>

From: Vitaly Gridnev
Date: 2017-01-16 15:08
To: masw
Subject: [Bug 1656247] Re: Cluster creation Error : EndpointNotFound: Could not find requested endpoint in Service Catalog.
So, when we are creating client, it already has valid endpoint type

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1656247

Title:
  Cluster creation Error : EndpointNotFound: Could not find requested
  endpoint in Service Catalog.

Status in Sahara:
  Invalid

Bug description:
  sahara error log as follow:

  Error during operating on cluster (reason: Could not find requested endpoint in Service Catalog.)
  Traceback (most recent call last):
    File "/opt/stack/sahara/sahara/service/ops.py", line 193, in wrapper
      f(cluster_id, *args, **kwds)
    File "/opt/stack/sahara/sahara/service/ops.py", line 286, in _provision_cluster
      INFRA.create_cluster(cluster)
    File "/opt/stack/sahara/sahara/service/heat/heat_engine.py", line 63, in create_cluster
      self._launch_instances(cluster, target_count, CREATE_STAGES)
    File "/opt/stack/sahara/sahara/service/heat/heat_engine.py", line 230, in _launch_instances
      cluster, target_count, update_stack, disable_rollback)
    File "/opt/stack/sahara/sahara/utils/cluster_progress_ops.py", line 139, in handler
      add_fail_event(instance, e)
    File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      self.force_reraise()
    File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      six.reraise(self.type_, self.value, self.tb)
    File "/opt/stack/sahara/sahara/utils/cluster_progress_ops.py", line 136, in handler
      value = func(*args, **kwargs)
    File "/opt/stack/sahara/sahara/service/heat/heat_engine....

Read more...

Revision history for this message
sven mark (svenmark) wrote :

look at here,from /usr/lib/python2.7/site-packages/heatclient/v1/stacks.py(172)create()

  resp = self.client.post('/stacks',
                                data=kwargs, headers=headers)

the client is not heat.client(),so and endpointurl is not get,and endpoint_type and service_type is not set.

  /opt/stack/sahara/sahara/service/heat/templates.py(207)instantiate()
-> b.execute_with_retries(h.client().stacks.create, **kwargs)
  /opt/stack/sahara/sahara/utils/openstack/base.py(93)execute_with_retries()
-> return method(*args, **kwargs)
  /usr/lib/python2.7/site-packages/heatclient/v1/stacks.py(172)create()
-> data=kwargs, headers=headers)
  /usr/lib/python2.7/site-packages/keystoneauth1/adapter.py(222)post()
-> return self.request(url, 'POST', **kwargs)
  /usr/lib/python2.7/site-packages/heatclient/common/http.py(314)request()
-> **kwargs)
  /usr/lib/python2.7/site-packages/keystoneauth1/adapter.py(373)request()
-> resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  /usr/lib/python2.7/site-packages/keystoneauth1/adapter.py(141)request()
-> return self.session.request(url, method, **kwargs)
> /usr/lib/python2.7/site-packages/positional/__init__.py(101)inner()
-> return wrapped(*args, **kwargs)
  /usr/lib/python2.7/site-packages/keystoneauth1/session.py(463)request()
-> headers = kwargs.setdefault('headers', dict())

Changed in sahara:
status: Invalid → Incomplete
Revision history for this message
Vitalii Gridnev (vgridnev) wrote :

We are creating heat client using client method from utils.openstack.heat module. So, endpoint type is set. Please, ensure that you have the latest version of the heatclient, might be you have issue [0]. Since author of the bug has devstack environment, and the bug can't be reproduced on the gate (where we also have devstack) so I'm marking that bug as Invalid

[0] https://bugs.launchpad.net/python-heatclient/+bug/1643507

Changed in sahara:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.