Bug #1623327 “openstack orchestration service list fails to retu...” : Bugs : OpenStack Identity (keystone)

Brad Marshall (brad-marshall) on 2016-09-14

tags:

added: canonical-bootstack

Revision history for this message

Steve Martinelli (stevemar) wrote on 2016-09-14:

#1

I added heatclient since that's where the "openstack orchestration service list" comes from: https://github.com/openstack/python-heatclient/blob/master/setup.cfg#L36 -- heatclient is an OpenStackClient plugin, and the code is maintained there

Revision history for this message

Steve Martinelli (stevemar) wrote on 2016-09-14:

#2

Can you add the output of `openstack orchestration service list --debug` as well, so we can see what routes the command is using.

Revision history for this message

Brad Marshall (brad-marshall) wrote on 2016-09-14:

#3

To be clear, this isn't just happening for heat. Every other service that has $(tenant_id) in the endpoint isn't showing up in the catalog either. Cinder has empty endpoints, swift only has the admin endpoint listed and nova compute and heat orchestration is empty, when heat cloudformation has data.

Revision history for this message

Peter Sabaini (peter-sabaini) wrote on 2016-09-15:

#5

Download full text (13.0 KiB)

Steve, output of openstack orchestration service list --debug is similar to this:

START with options: ['orchestration', 'service', 'list', '--debug']
options: Namespace(access_token_endpoint='', auth_type='', auth_url='http://10.5.20.240:35357/v3', cacert='', cert='', client_id='', client_secret='***', cloud='', debug=True, default_domain='default', deferred_help=False, domain_id='', domain_name='admin_domain', endpoint='', identity_provider='', identity_provider_url='', insecure=None, interface='', key='', log_file=None, os_beta_command=False, os_compute_api_version='', os_identity_api_version='3', os_image_api_version='', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_volume_api_version='', password='***', project_domain_id='', project_domain_name='', project_id='', project_name='', protocol='', region_name='RegionOne', scope='', service_provider_endpoint='', timing=False, token='***', trust_id='', url='', user_domain_id='', user_domain_name='admin_domain', user_id='', username='admin', verbose_level=3, verify=None)
defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'key_manager_api_version': u'v1', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', u'orchestration_api_version': '1', u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://10.5.20.240:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'username': 'admin', 'user_domain_name': 'admin_domain', 'password': '***', 'auth_url': 'http://10.5.20.240:35357/v3', 'domain_name': 'admin_domain'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '***', 'cacert': None, u'key_manager_api_version': u'v1', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'username': 'admin', 'cert': None, u'secgroup_source': u'neutron', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 2, cmd group openstack.image.v2
volume API version 2, cmd group...

Steve, output of openstack orchestration service list --debug is similar to this:

START with options: ['orchestration', 'service', 'list', '--debug']
options: Namespace(access_token_endpoint='', auth_type='', auth_url='http://10.5.20.240:35357/v3', cacert='', cert='', client_id='', client_secret='***', cloud='', debug=True, default_domain='default', deferred_help=False, domain_id='', domain_name='admin_domain', endpoint='', identity_provider='', identity_provider_url='', insecure=None, interface='', key='', log_file=None, os_beta_command=False, os_compute_api_version='', os_identity_api_version='3', os_image_api_version='', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_volume_api_version='', password='***', project_domain_id='', project_domain_name='', project_id='', project_name='', protocol='', region_name='RegionOne', scope='', service_provider_endpoint='', timing=False, token='***', trust_id='', url='', user_domain_id='', user_domain_name='admin_domain', user_id='', username='admin', verbose_level=3, verify=None)
defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'key_manager_api_version': u'v1', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', u'orchestration_api_version': '1', u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://10.5.20.240:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'username': 'admin', 'user_domain_name': 'admin_domain', 'password': '***', 'auth_url': 'http://10.5.20.240:35357/v3', 'domain_name': 'admin_domain'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '***', 'cacert': None, u'key_manager_api_version': u'v1', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'username': 'admin', 'cert': None, u'secgroup_source': u'neutron', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 2, cmd group openstack.image.v2
volume API version 2, cmd group openstack.volume.v2
identity API version 3, cmd group openstack.identity.v3
object_store API version 1, cmd group openstack.object_store.v1
neutronclient API version 2, cmd group openstack.neutronclient.v2
orchestration API version 1, cmd group openstack.orchestration.v1
command: orchestration service list -> heatclient.osc.v1.service.ListService
Auth plugin password selected
auth_type: password
Using auth plugin: password
Using parameters {'username': 'admin', 'user_domain_name': 'admin_domain', 'domain_name': 'admin_domain', 'auth_url': 'http://10.5.20.240:35357/v3', 'password': '***', 'project_domain_id': 'default'}
Get auth_ref
REQ: curl -g -i -X GET http://10.5.20.240:35357/v3 -H "Accept: application/json" -H "User-Agent: python-openstackclient keystoneauth1/2.11.1 python-requests/2.11.0 CPython/2.7.6"
Starting new HTTP connection (1): 10.5.20.240
"GET /v3 HTTP/1.1" 200 250
RESP: [200] Date: Thu, 15 Sep 2016 07:26:06 GMT Server: Apache/2.4.7 (Ubuntu) Vary: X-Auth-Token X-Distribution: Ubuntu x-openstack-request-id: req-684cea58-13d9-47b7-8946-18615b3361bc Content-Length: 250 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json 
RESP BODY: {"version": {"status": "stable", "updated": "2016-04-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.6", "links": [{"href": "http://10.5.20.240:5000/v3/", "rel": "self"}]}}

Making authentication request to http://10.5.20.240:5000/v3/auth/tokens
Starting new HTTP connection (1): 10.5.20.240
"POST /v3/auth/tokens HTTP/1.1" 201 5371
{"token": {"domain": {"id": "07edd31a46284b6682ad1d4f1a74ef36", "name": "admin_domain"}, "methods": ["password"], "roles": [{"id": "76e847fdf28e4e62a133826c138aa89d", "name": "Admin"}], "expires_at": "2016-09-15T08:26:06.220515Z", "catalog": [{"endpoints": [], "type": "orchestration", "id": "0df730f7f5134d3eaa0145d8dedc048f", "name": "heat"}, {"endpoints": [{"url": "http://10.28.2.176:8000/v1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "00c8405b29174d888efa4906ca916133"}, {"url": "http://10.28.2.176:8000/v1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "cf19395dc841463687fd7bc54111dab5"}, {"url": "http://10.28.2.176:8000/v1", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "fa31f86b98ff4372bfa1e1f15e09290d"}], "type": "cloudformation", "id": "16e2686362d34307af86388c2c755d0c", "name": "heat-cfn"}, {"endpoints": [], "type": "compute", "id": "6055075011104edcbe7ef351eb9451f3", "name": "nova"}, {"endpoints": [{"url": "http://10.5.20.241:8080", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "3defad03a91c434d87139a56d719ed1f"}, {"url": "http://10.5.20.241:8080", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "7c70b479afe4459aba7bc3bbfdec9c67"}, {"url": "http://10.5.20.241:8080", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "81b2e877ced340418b0c4ba68c7f3f43"}], "type": "s3", "id": "68a11df3c93944b58c5d2012c3a987dc", "name": "s3"}, {"endpoints": [{"url": "http://10.5.20.240:5000/v3", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "510f68ebf82c4edf934bac7819ba4b8a"}, {"url": "http://10.5.20.240:35357/v3", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "c294a6f9af7e4666b905d741a3584c9f"}, {"url": "http://10.5.20.240:5000/v3", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "e4fa446028ec4217801ef6657e506ee0"}], "type": "identity", "id": "6f723cd1a14945b3be6d0676ce177821", "name": "keystone"}, {"endpoints": [{"url": "http://10.5.20.241:8080", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "53470d934f3947bfaa1d87c256e7c4f0"}], "type": "object-store", "id": "7d33d4eb86b143f2859931eace5babf0", "name": "swift"}, {"endpoints": [{"url": "http://10.5.20.243:8777", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "abeb4c7d9e3942fdac82094ff5cca3e8"}, {"url": "http://10.5.20.243:8777", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "dcff573bde754549a9e5bf927da24e6b"}, {"url": "http://10.5.20.243:8777", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "e9e9648fab9249d1a9b8b5152179979a"}], "type": "metering", "id": "84a9b297cef54e719bd788369b32a843", "name": "ceilometer"}, {"endpoints": [{"url": "http://10.5.20.247:9696", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "66cba65a80714e6ab19dbce2867f4b08"}, {"url": "http://10.5.20.247:9696", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "8aed6fc81fb54632895d1589aadc0ff7"}, {"url": "http://10.5.20.247:9696", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "9467121988814c61a42011b3a003106b"}], "type": "network", "id": "99c44d0486ab46b8ae261cfb9a452e10", "name": "neutron"}, {"endpoints": [], "type": "volume", "id": "b04b70ed881145e0933b4fdafe1ee4f4", "name": "cinder"}, {"endpoints": [{"url": "http://10.28.2.175", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "7652964c216c4d47a14080ee5b45f438"}, {"url": "http://10.28.2.175", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "7f270d1bca984620999dc07adec970ba"}, {"url": "http://10.28.2.175", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "c1027367e1bf4fb892aa8ebbefcbcc42"}], "type": "product-streams", "id": "bbf9f8530a9944fbbd75bde670d8e157", "name": "image-stream"}, {"endpoints": [{"url": "http://10.5.20.245:9292", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "4c3732ad6a76440f92013ebac49d7992"}, {"url": "http://10.5.20.245:9292", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "b33b1b5f56f84e8284996e267d599ca3"}, {"url": "http://10.5.20.245:9292", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "ba8ef497a6eb4680878bc97e0e20d067"}], "type": "image", "id": "d07d0850007c477083aec3556d11677b", "name": "glance"}, {"endpoints": [], "type": "volumev2", "id": "dfeee9c3b3904deb8b4a5bd32f10b88f", "name": "cinderv2"}], "user": {"domain": {"id": "07edd31a46284b6682ad1d4f1a74ef36", "name": "admin_domain"}, "id": "1d3dd7410c33419aac94508264038c89", "name": "admin"}, "audit_ids": ["hEjOydAGRUWrFVrA1x604A"], "issued_at": "2016-09-15T07:26:06.220538Z"}}
take_action(Namespace(columns=[], formatter='table', max_width=0, noindent=False, quote_mode='nonnumeric'))
Instantiating orchestration client: <class 'heatclient.v1.client.Client'>
public endpoint for orchestration service not found
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 387, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/cliff/display.py", line 100, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/heatclient/osc/v1/service.py", line 34, in take_action
    heat_client = self.app.client_manager.orchestration
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/common/clientmanager.py", line 51, in __get__
    self._handle = self.factory(instance)
  File "/usr/lib/python2.7/dist-packages/heatclient/osc/plugin.py", line 37, in make_client
    endpoint=instance.get_endpoint_for_service_type('orchestration'),
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/common/clientmanager.py", line 272, in get_endpoint_for_service_type
    endpoint_type=interface,
  File "/usr/local/lib/python2.7/dist-packages/wrapt/wrappers.py", line 561, in __call__
    args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/service_catalog.py", line 297, in url_for
    raise exceptions.EndpointNotFound(msg)
EndpointNotFound: public endpoint for orchestration service not found
clean_up ListService: public endpoint for orchestration service not found
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", line 119, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 267, in run
    result = self.run_subcommand(remainder)
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", line 164, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 387, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/cliff/display.py", line 100, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/heatclient/osc/v1/service.py", line 34, in take_action
    heat_client = self.app.client_manager.orchestration
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/common/clientmanager.py", line 51, in __get__
    self._handle = self.factory(instance)
  File "/usr/lib/python2.7/dist-packages/heatclient/osc/plugin.py", line 37, in make_client
    endpoint=instance.get_endpoint_for_service_type('orchestration'),
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/common/clientmanager.py", line 272, in get_endpoint_for_service_type
    endpoint_type=interface,
  File "/usr/local/lib/python2.7/dist-packages/wrapt/wrappers.py", line 561, in __call__
    args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/service_catalog.py", line 297, in url_for
    raise exceptions.EndpointNotFound(msg)
EndpointNotFound: public endpoint for orchestration service not found

END return value: 1

Revision history for this message

David Ames (thedac) wrote on 2016-09-15:

#7

Regarding heat, has the domain-setup action been run? This is required for heat
with keystone v3.

https://github.com/openstack/charm-heat/blob/master/README.md#usage
https://github.com/openstack/charm-heat/blob/master/actions.yaml#L5

@bradm Regarding catalogue entries note the difference between domain and
project auth:
http://pastebin.ubuntu.com/23183661/
http://pastebin.ubuntu.com/23183668/
Is there something more than that going on?

Revision history for this message

David Ames (thedac) wrote on 2016-09-15:

#8

Domain vs project auth has a similar effect on heat:

Revision history for this message

Brad Marshall (brad-marshall) wrote on 2016-09-16:

#9

@thedac, it appears you are right, I was trying with some project details set, but apparently not the right set. Is there any documentation available that details what is needed where? I took a look around and couldn't find anything obvious.

I can now get an output from both openstack orchestration service list and heat stack-list, so I take it that means that heat is working correctly with keystone v3.

Revision history for this message

Brad Marshall (brad-marshall) wrote on 2016-09-16:

#10

To be clear, I'm using the following variables:

OS_AUTH_URL=http://keystone:35357/v3
OS_USERNAME=username
OS_PASSWORD=password
OS_REGION_NAME=region
OS_USER_DOMAIN_NAME=user_domain
OS_PROJECT_NAME=user
OS_PROJECT_DOMAIN_NAME=default
API_VERSION=3
OS_IDENTITY_API_VERSION=3

And when doing domain auth, replace OS_PROJECT_NAME with OS_DOMAIN_NAME=user_domain and drop OS_PROJECT_DOMAIN_NAME.

Revision history for this message

Liam Young (gnuoy) wrote on 2016-09-16:

#11

Hi Brad, there is some info in the charm release notes https://wiki.ubuntu.com/OpenStack/OpenStackCharms/ReleaseNotes1604#Keystone_v3_API_support

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2016-09-22:

#12

The openstack deployment tool you used is not setting up the endpoints correctly for endpoints which require tenants in the path - if you used a particular deployment tool this bug should be raised there.

Changed in python-heatclient:
status:	New → Invalid

Revision history for this message

Billy Olsen (billy-olsen) wrote on 2016-12-12:

#13

Based on Brad's comment in #9, there were actions that were missing for the openstack orchestration service. I believe this to no longer be a valid bug, therefore I'm marking remaining tasks as invalid.

Changed in python-openstackclient:
status:	New → Invalid
Changed in keystone:
status:	New → Invalid

	Status	Importance	Assigned to
OpenStack Identity (keystone)	Invalid	Undecided	Unassigned
python-heatclient	Invalid	Undecided	Unassigned
python-openstackclient	Invalid	Undecided	Unassigned

OpenStack Identity (keystone)

openstack orchestration service list fails to return endpoint

Bug Description

Other bug subscribers

Remote bug watches