openstack catalog list error in multi region

[XiaodongPan]

In the multi region, openstack catalog list will hide same endpoints, as below:
[root@con01 openstackclient(keystone_admin)]$ openstack endpoint list
+----------------------------------+----------+--------------+--------------+
| ID | Region | Service Name | Service Type |
+----------------------------------+----------+--------------+--------------+
| 83e1de95e15a45dca96514a3f06ba19e | TestRG01 | cinder | volume |
| be1a015332d9434c88dcb8c88bbcb1e7 | TestRG02 | cinder1 | volume |
+----------------------------------+----------+--------------+--------------+

[root@con01 openstackclient(keystone_admin)]$ openstack catalog list
+------------+----------+-------------------------------------------------------------------------------+
| Name | Type | Endpoints |
+------------+----------+-------------------------------------------------------------------------------+
...
| cinder1 | volume | TestRG01 |
| | | publicURL: http://192.168.17.143:8776/v1/756f3606480a4b6d8ef7f72a2b1a73d6 |
| | | internalURL: http://192.168.17.143:8776/v1/756f3606480a4b6d8ef7f72a2b1a73d6 |
| | | adminURL: http://192.168.17.143:8776/v1/756f3606480a4b6d8ef7f72a2b1a73d6 |
| | | TestRG02 |
| | | publicURL: http://192.168.17.143:8776/v1/756f3606480a4b6d8ef7f72a2b1a73d6 |
| | | internalURL: http://192.168.17.143:8776/v1/756f3606480a4b6d8ef7f72a2b1a73d6 |
| | | adminURL: http://192.168.17.143:8776/v1/756f3606480a4b6d8ef7f72a2b1a73d6 |
| | | |
...
+------------+----------+-------------------------------------------------------------------------------+

the result of catalog list is error, there only left cinder1, cinder is out. It will cause cinder work error

[Steve Martinelli (stevemar)]

whats the output when you add --debug to the command? it should include the contents coming back from keystone, that'll help us in determining if the bug is in keystone (giving back less endpoints) or in openstackclient (in rendering the data)

[XiaodongPan (xdongp)]

I have find the reason of this bug, In the source file "master/keystone/token/providers/common.py", the function of "format_catalog", as below:
==============================
        for region, region_ref in catalog_ref.items():
            for service, service_ref in region_ref.items():
                new_service_ref = services.get(service, {})
                new_service_ref['name'] = service_ref.pop('name')
                new_service_ref['type'] = service
                new_service_ref['endpoints_links'] = []
                service_ref['region'] = region

                endpoints_ref = new_service_ref.get('endpoints', [])
                endpoints_ref.append(service_ref)

                new_service_ref['endpoints'] = endpoints_ref
                services[service] = new_service_ref

        return list(services.values())
===========================================
In the loop, the services dict use service type as key, when the type name are the same, but the service name are not the same, ex: region1: type=volume, name=cinder; region2: type=volume, name=cinder1. then the dict of services use "volume" as key, the last item will overwrite the previous value.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/490720

[Lance Bragstad (lbragstad)]

I attempted to recreate this with a test, but I was unable to. Another possible route for investigation might be looking at the client to make sure nothing strange is happening there. Please feel free to review or modify the test I've proposed [0] that attempts to recreate this.

[0] https://review.openstack.org/#/c/490720/

[Lance Bragstad (lbragstad)]

I attempted to recreate this manually without success. I used the following process;

1.) create a new regions so there are two in the deployment
2.) create a new service of the same type as an existing service but different name `keystone1`
3.) create an endpoint an associate it to the new service and region
4.) get the catalog

Additional information below:

$ openstack region list -f yaml
- Description: ''
  Parent Region: null
  Region: RegionOne
- Description: ''
  Parent Region: null
  Region: RegionTwo
$ openstack service list -f yaml
- ID: 379688e0b3df4d87b36b9eb3bca62168
  Name: keystone
  Type: identity
- ID: 4b278a66df2a4ec18395c7b586b10747
  Name: keystone1
  Type: identity
$ openstack endpoint list -f yaml
- Enabled: true
  ID: 0af66ca7532c4bcea8d33c350dd004c6
  Interface: admin
  Region: RegionOne
  Service Name: keystone
  Service Type: identity
  URL: http://192.168.122.160:35357/
- Enabled: true
  ID: 383440e8128b4c669d3ef791bcecf401
  Interface: public
  Region: RegionTwo
  Service Name: keystone1
  Service Type: identity
  URL: http://192.168.122.160:35357/
- Enabled: true
  ID: e0eed76efd2d49879a4db2532161105c
  Interface: public
  Region: RegionOne
  Service Name: keystone
  Service Type: identity
  URL: http://192.168.122.160:35357/
$ openstack catalog list -f yaml
- Endpoints: "RegionOne\n admin: http://192.168.122.160:35357/\nRegionOne\n public:\
    \ http://192.168.122.160:35357/\n"
  Name: keystone
  Type: identity
- Endpoints: "RegionTwo\n public: http://192.168.122.160:35357/\n"
  Name: keystone1
  Type: identity

[Lance Bragstad (lbragstad)]

I performed all the actions and tests in comments #4 and #5 against the v3 API. If there is something wrong with how I attempted to recreate the issue - please let me know. Marking this as Invalid for the time being. Please reopen if the issue resurfaces or if more information is available.