authentication code hangs when there are three or more admin keystone endpoints

[Chris Friesen]

I'm running stable/pike devstack, and I was playing around with what happens when there are many endpoints in multiple regions, and I stumbled over a scenario where the keystone authentication code hangs.

My original endpoint list looked like this:

ubuntu@devstack:/opt/stack/devstack$ openstack endpoint list
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
| 0a9979ebfdbf48ce91ccf4e2dd952c1a | RegionOne | kingbird | synchronization | True | internal | http://127.0.0.1:8118/v1.0 |
| 11d5507afe2a4eddb4f030695699114f | RegionOne | placement | placement | True | public | http://128.224.186.226/placement |
| 1e42cf139398405188755b7e00aecb4d | RegionOne | keystone | identity | True | admin | http://128.224.186.226/identity |
| 2daf99edecae4afba88bb58233595481 | RegionOne | glance | image | True | public | http://128.224.186.226/image |
| 2ece52e8bbb34d47b9bd5611f5959385 | RegionOne | kingbird | synchronization | True | admin | http://127.0.0.1:8118/v1.0 |
| 4835a089666a4b03bd2f499457ade6c2 | RegionOne | kingbird | synchronization | True | public | http://127.0.0.1:8118/v1.0 |
| 78e9fbc0a47642268eda3e3576920f37 | RegionOne | nova | compute | True | public | http://128.224.186.226/compute/v2.1 |
| 96a1e503dc0e4520a190b01f6a0cf79c | RegionOne | keystone | identity | True | public | http://128.224.186.226/identity |
| a1887dbc8c5e4af5b4a6dc5ce224b8ff | RegionOne | cinderv2 | volumev2 | True | public | http://128.224.186.226/volume/v2/$(project_id)s |
| b7d5938141694a4c87adaed5105ea3ab | RegionOne | cinder | volume | True | public | http://128.224.186.226/volume/v1/$(project_id)s |
| bb169382cbea4715964e4652acd48070 | RegionOne | nova_legacy | compute_legacy | True | public | http://128.224.186.226/compute/v2/$(project_id)s |
| e01c8d8e08874d61b9411045a99d4860 | RegionOne | neutron | network | True | public | http://128.224.186.226:9696/ |
| f94c96ed474249a29a6c0a1bb2b2e500 | RegionOne | cinderv3 | volumev3 | True | public | http://128.224.186.226/volume/v3/$(project_id)s |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+

I was able to successfully run the following python code:

from keystoneauth1 import loading
from keystoneauth1 import loading
from keystoneauth1 import session
from keystoneclient.v3 import client
loader = loading.get_plugin_loader("password")
auth = loader.load_from_options(username='admin',password='secret',project_name='admin',auth_url='http://128.224.186.226/identity')
sess = session.Session(auth=auth)
keystone = client.Client(session=sess)
keystone.services.list()

I then duplicated all of the endpoints in a new region "region2", and was able to run the python code. When I duplicated all the endpoints again in a new region "region3" (for a total of 39 endpoints) the python code hung at the final line.

Removing all the "region3" endpoints allowed the python code to work again.

During all of this the command "openstack endpoint list" worked fine.

Further testing seems to indicate that it is the third "admin" keystone endpoint that is causing the problem. I can add multiple "public" keystone endpoints, but three or more "admin" keystone endpoints cause the python code to hang.

[Lance Bragstad (lbragstad)]

It sounds like the client is having a hard time dealing with that specific data set. I'm not sure if the keystone service itself can do anything about that.

Adding python-keystoneclient to this bug report until we figure out exactly what's going on.

[wangxiyuan (wangxiyuan)]

I can't reproduce in my env(master branch). This is what i did:
test_create_endpoint.sh:

for i in compute volume image swift network identity ;
do
for j in RegionTwo RegionThree ;
do
for k in admin public internal ;
do
openstack endpoint create $i --region $j $k http://10.3.150.25/identity
done
done
done

kc_test.py:

from keystoneauth1 import loading
from keystoneauth1 import loading
from keystoneauth1 import session
from keystoneclient.v3 import client

loader = loading.get_plugin_loader("password")
auth = loader.load_from_options(username='admin',user_domain_name="default",password='root',project_name='admin',project_domain_name="default",auth_url='http://10.3.150.25/identity')
sess = session.Session(auth=auth)
keystone = client.Client(session=sess)
result = keystone.services.list()

print(result)

There are three region and 54 endpoints. But everything works well. Did I miss something?

[Chris Friesen (cbf123)]

Thanks for taking a look. I'll try and reproduce with current master when I get some time, probably not till next week.

[Lance Bragstad (lbragstad)]

Marking as Incomplete just until we get a bit more information. Thanks for attempting to recreate!

[Launchpad Janitor (janitor)]

[Expired for python-keystoneclient because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for OpenStack Identity (keystone) because there has been no activity for 60 days.]