authentication code hangs when there are three or more admin keystone endpoints

Bug #1724686 reported by Chris Friesen on 2017-10-18
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Undecided
Unassigned
python-keystoneclient
Undecided
Unassigned

Bug Description

I'm running stable/pike devstack, and I was playing around with what happens when there are many endpoints in multiple regions, and I stumbled over a scenario where the keystone authentication code hangs.

My original endpoint list looked like this:

ubuntu@devstack:/opt/stack/devstack$ openstack endpoint list
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
| 0a9979ebfdbf48ce91ccf4e2dd952c1a | RegionOne | kingbird | synchronization | True | internal | http://127.0.0.1:8118/v1.0 |
| 11d5507afe2a4eddb4f030695699114f | RegionOne | placement | placement | True | public | http://128.224.186.226/placement |
| 1e42cf139398405188755b7e00aecb4d | RegionOne | keystone | identity | True | admin | http://128.224.186.226/identity |
| 2daf99edecae4afba88bb58233595481 | RegionOne | glance | image | True | public | http://128.224.186.226/image |
| 2ece52e8bbb34d47b9bd5611f5959385 | RegionOne | kingbird | synchronization | True | admin | http://127.0.0.1:8118/v1.0 |
| 4835a089666a4b03bd2f499457ade6c2 | RegionOne | kingbird | synchronization | True | public | http://127.0.0.1:8118/v1.0 |
| 78e9fbc0a47642268eda3e3576920f37 | RegionOne | nova | compute | True | public | http://128.224.186.226/compute/v2.1 |
| 96a1e503dc0e4520a190b01f6a0cf79c | RegionOne | keystone | identity | True | public | http://128.224.186.226/identity |
| a1887dbc8c5e4af5b4a6dc5ce224b8ff | RegionOne | cinderv2 | volumev2 | True | public | http://128.224.186.226/volume/v2/$(project_id)s |
| b7d5938141694a4c87adaed5105ea3ab | RegionOne | cinder | volume | True | public | http://128.224.186.226/volume/v1/$(project_id)s |
| bb169382cbea4715964e4652acd48070 | RegionOne | nova_legacy | compute_legacy | True | public | http://128.224.186.226/compute/v2/$(project_id)s |
| e01c8d8e08874d61b9411045a99d4860 | RegionOne | neutron | network | True | public | http://128.224.186.226:9696/ |
| f94c96ed474249a29a6c0a1bb2b2e500 | RegionOne | cinderv3 | volumev3 | True | public | http://128.224.186.226/volume/v3/$(project_id)s |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+

I was able to successfully run the following python code:

from keystoneauth1 import loading
from keystoneauth1 import loading
from keystoneauth1 import session
from keystoneclient.v3 import client
loader = loading.get_plugin_loader("password")
auth = loader.load_from_options(username='admin',password='secret',project_name='admin',auth_url='http://128.224.186.226/identity')
sess = session.Session(auth=auth)
keystone = client.Client(session=sess)
keystone.services.list()

I then duplicated all of the endpoints in a new region "region2", and was able to run the python code. When I duplicated all the endpoints again in a new region "region3" (for a total of 39 endpoints) the python code hung at the final line.

Removing all the "region3" endpoints allowed the python code to work again.

During all of this the command "openstack endpoint list" worked fine.

Further testing seems to indicate that it is the third "admin" keystone endpoint that is causing the problem. I can add multiple "public" keystone endpoints, but three or more "admin" keystone endpoints cause the python code to hang.

Chris Friesen (cbf123) on 2017-10-18
summary: - authentication code hangs when there are many endpoints
+ authentication code hangs when there are three or more admin keystone
+ endpoints
description: updated
description: updated
Lance Bragstad (lbragstad) wrote :

It sounds like the client is having a hard time dealing with that specific data set. I'm not sure if the keystone service itself can do anything about that.

Adding python-keystoneclient to this bug report until we figure out exactly what's going on.

tags: added: office
tags: added: office-hours
removed: office
wangxiyuan (wangxiyuan) wrote :

I can't reproduce in my env(master branch). This is what i did:
test_create_endpoint.sh:

for i in compute volume image swift network identity ;
do
for j in RegionTwo RegionThree ;
do
for k in admin public internal ;
do
openstack endpoint create $i --region $j $k http://10.3.150.25/identity
done
done
done

kc_test.py:

from keystoneauth1 import loading
from keystoneauth1 import loading
from keystoneauth1 import session
from keystoneclient.v3 import client

loader = loading.get_plugin_loader("password")
auth = loader.load_from_options(username='admin',user_domain_name="default",password='root',project_name='admin',project_domain_name="default",auth_url='http://10.3.150.25/identity')
sess = session.Session(auth=auth)
keystone = client.Client(session=sess)
result = keystone.services.list()

print(result)

There are three region and 54 endpoints. But everything works well. Did I miss something?

Chris Friesen (cbf123) wrote :

Thanks for taking a look. I'll try and reproduce with current master when I get some time, probably not till next week.

Lance Bragstad (lbragstad) wrote :

Marking as Incomplete just until we get a bit more information. Thanks for attempting to recreate!

Changed in keystone:
status: New → Incomplete
Changed in python-keystoneclient:
status: New → Incomplete
Launchpad Janitor (janitor) wrote :

[Expired for python-keystoneclient because there has been no activity for 60 days.]

Changed in python-keystoneclient:
status: Incomplete → Expired
Launchpad Janitor (janitor) wrote :

[Expired for OpenStack Identity (keystone) because there has been no activity for 60 days.]

Changed in keystone:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers