EndpointNotFound error

[Jeremy Hanmer]

I get the following errors in my Apache log even though all of my endpoints appear to be correctly configured and horizon seems to work fine. This is with openstack-dashboard 2012.2+git201208121702~precise and python-keystoneclient 1:0.1.2+git201208061547~precise

[Mon Aug 13 18:53:41 2012] [error] unable to retrieve service catalog with token
[Mon Aug 13 18:53:41 2012] [error] Traceback (most recent call last):
[Mon Aug 13 18:53:41 2012] [error] File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 132, in _extract_service_catalog
[Mon Aug 13 18:53:41 2012] [error] endpoint_type='adminURL')
[Mon Aug 13 18:53:41 2012] [error] File "/usr/lib/python2.7/dist-packages/keystoneclient/service_catalog.py", line 62, in url_for
[Mon Aug 13 18:53:41 2012] [error] raise exceptions.EndpointNotFound('Endpoint not found.')
[Mon Aug 13 18:53:41 2012] [error] EndpointNotFound: Endpoint not found.

[Alex Gosse (agosse-rvbd)]

I saw this when I was trying to launch a new instance using devstack, which was running on a vmware fusion vm.

In another screen session, I found an error saying that there wasn't a host that could support the instance because my VM didn't have enough memory. I increased the memory size of my VM to 5 gigs, and that fixed the problem.

My problem and the error in this bug may not be related, but the above error didn't appear once I fixed the memory problem. To fair, I was just happy that my instances were launching, so I didn't look too hard for an error either.

[David Peraza (dperaza)]

I see the same problem running in Ubuntu 12.04, it does not seem to affect anything. But now I wonder since my snapshots show in CLI and not in the Dashboard

[Joseph Heck (heckj)]

Hey Alex, David -

Could you report on what you have configured in Keystone? It's generally expected that you've defined adminURL, publicURL, and internalURL on each service endpoint, but that's very dependent on how you set up and deployed keystone.

-joe

[zhangping (zhangping159)]

i have the same problem. this is my keystone configure file:

root@cloud-1:~# cat /etc/keystone/keystone.conf | grep -v ^#| grep -v ^$
[DEFAULT]
admin_token = password
bind_host = 0.0.0.0
public_port = 5000
admin_port = 35357
compute_port = 8774
verbose = True
debug = True
log_file = keystone.log
log_dir = /var/log/keystone
log_config = /etc/keystone/logging.conf
[sql]
connection = mysql://keystone:cloud1234@192.168.10.90:3306/keystone
idle_timeout = 200
[identity]
driver = keystone.identity.backends.sql.Identity
[catalog]
driver = keystone.catalog.backends.sql.Catalog
[token]
driver = keystone.token.backends.sql.Token
[policy]
driver = keystone.policy.backends.rules.Policy
[ec2]
driver = keystone.contrib.ec2.backends.sql.Ec2
[ssl]
[signing]
[ldap]
[filter:debug]
paste.filter_factory = keystone.common.wsgi:Debug.factory
[filter:token_auth]
paste.filter_factory = keystone.middleware:TokenAuthMiddleware.factory
[filter:admin_token_auth]
paste.filter_factory = keystone.middleware:AdminTokenAuthMiddleware.factory
[filter:xml_body]
paste.filter_factory = keystone.middleware:XmlBodyMiddleware.factory
[filter:json_body]
paste.filter_factory = keystone.middleware:JsonBodyMiddleware.factory
[filter:user_crud_extension]
paste.filter_factory = keystone.contrib.user_crud:CrudExtension.factory
[filter:crud_extension]
paste.filter_factory = keystone.contrib.admin_crud:CrudExtension.factory
[filter:ec2_extension]
paste.filter_factory = keystone.contrib.ec2:Ec2Extension.factory
[filter:s3_extension]
paste.filter_factory = keystone.contrib.s3:S3Extension.factory
[filter:url_normalize]
paste.filter_factory = keystone.middleware:NormalizingFilter.factory
[filter:stats_monitoring]
paste.filter_factory = keystone.contrib.stats:StatsMiddleware.factory
[filter:stats_reporting]
paste.filter_factory = keystone.contrib.stats:StatsExtension.factory
[app:public_service]
paste.app_factory = keystone.service:public_app_factory
[app:admin_service]
paste.app_factory = keystone.service:admin_app_factory
[pipeline:public_api]
pipeline = stats_monitoring url_normalize token_auth admin_token_auth xml_body json_body debug ec2_extension user_crud_extension public_service
[pipeline:admin_api]
pipeline = stats_monitoring url_normalize token_auth admin_token_auth xml_body json_body debug stats_reporting ec2_extension s3_extension crud_extension admin_service
[app:public_version_service]
paste.app_factory = keystone.service:public_version_app_factory
[app:admin_version_service]
paste.app_factory = keystone.service:admin_version_app_factory
[pipeline:public_version_api]
pipeline = stats_monitoring url_normalize xml_body public_version_service
[pipeline:admin_version_api]
pipeline = stats_monitoring url_normalize xml_body admin_version_service
[composite:main]
use = egg:Paste#urlmap
/v2.0 = public_api
/ = public_version_api
[composite:admin]
use = egg:Paste#urlmap
/v2.0 = admin_api
/ = admin_version_api

[Sina Sadeghi (sina-sa)]

Hello, please note this bug is still active and confirmed by multiple users!

[Tue Dec 11 08:11:01 2012] [error] DEBUG:openstack_auth.backend:Beginning user authentication for user "sina".
[Tue Dec 11 08:11:01 2012] [error] unable to retrieve service catalog with token
[Tue Dec 11 08:11:01 2012] [error] Traceback (most recent call last):
[Tue Dec 11 08:11:01 2012] [error] File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 132, in _extract_service_catalog
[Tue Dec 11 08:11:01 2012] [error] endpoint_type='adminURL')
[Tue Dec 11 08:11:01 2012] [error] File "/usr/lib/python2.7/dist-packages/keystoneclient/service_catalog.py", line 62, in url_for
[Tue Dec 11 08:11:01 2012] [error] raise exceptions.EndpointNotFound('Endpoint not found.')
[Tue Dec 11 08:11:01 2012] [error] EndpointNotFound: Endpoint not found.
[Tue Dec 11 08:11:01 2012] [error] ERROR:keystoneclient.v2_0.client:unable to retrieve service catalog with token
[Tue Dec 11 08:11:01 2012] [error] Traceback (most recent call last):
[Tue Dec 11 08:11:01 2012] [error] File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 132, in _extract_service_catalog
[Tue Dec 11 08:11:01 2012] [error] endpoint_type='adminURL')
[Tue Dec 11 08:11:01 2012] [error] File "/usr/lib/python2.7/dist-packages/keystoneclient/service_catalog.py", line 62, in url_for
[Tue Dec 11 08:11:01 2012] [error] raise exceptions.EndpointNotFound('Endpoint not found.')
[Tue Dec 11 08:11:01 2012] [error] EndpointNotFound: Endpoint not found.
[Tue Dec 11 08:11:01 2012] [error] DEBUG:openstack_auth.backend:Authentication completed for user "sina".

Yet, from the commandline of the same computer:

root@horizon:~# keystone catalog
Service: compute
+-------------+---------------------------------------------------------------------+
| Property | Value |
+-------------+---------------------------------------------------------------------+
| adminURL | https://nova.aptira.com:8774/v2/EDITFORSECURITY |
| id | b8c8611461a04392a3b5567b44cacbf8 |
| internalURL | https://nova.aptira.com:8774/v2/EDITFORSECURITY |
| publicURL | https://compute.aptira.com:8774/v2/EDITFORSECURITY |
| region | SY3 |
+-------------+---------------------------------------------------------------------+
Service: image
+-------------+-----------------------------------+
| Property | Value |
+-------------+-----------------------------------+
| adminURL | https://glance.aptira.com:9292/v1 |
| id | c548e52af1c94817a1285e774f717405 |
| internalURL | https://glance.aptira.com:9292/v1 |
| publicURL | https://glance.aptira.com:9292/v1 |
| region | SY3 |
+-------------+-----------------------------------+
Service: volume
+-------------+--------------------------------------------------------------------+
| Property | Value |
+-------------+---------------------------------------...

[Sina Sadeghi (sina-sa)]

To confirm what David Peraza wrote, this doesn't seem to affect anything, but it is a rather large error which constantly appears in the log, triggering monitoring scripts for operations to take action!

[Karel Striegel (karel-striegel)]

Could this also cause performance issues in the horizon dashboad ?

[Gabriel Hurley (gabriel-hurley)]

Could you check what version of keystoneclient you're using? If it's <0.2 I would suggest updating and seeing if that makes the logging go away. Previously there was a place where there was an expected EndpointNotFound (when receiving an unscoped token) which was being logged as an exception despite being perfectly acceptable. This was fixed and subsequently released in the latest keystoneclient.

[Sina Sadeghi (sina-sa)]

Gabriel, thanks for the info. We are running the python-keystoneclient in Ubuntu 12.04.1 LTS, using Ubuntu Cloud Archive, which is 0.1.3 it seems.

What process needs initiating to get the keystoneclient updated in UCA?

[Gabriel Hurley (gabriel-hurley)]

This bug *was* in keystoneclient. It was fixed in https://github.com/openstack/python-keystoneclient/commit/0dfc69806d3767a7f6be91d9df18f20d31866c91 and should be available in the latest version (0.2 I believe). Try updating your version of keystoneclient.

[Dolph Mathews (dolph)]

Agree with Gabriel above; AFAIK, this was fixed.

[Ricardo Contreras (rcontreras1989)]

This doesn't work for me! There is still a problem with the service_catalog!