Race condition between keystone and glance-api containers

Bug #1429104 reported by Ryan Hallisey
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
High
Charles Crouch

Bug Description

There is a race condition between the keystone and glance-api containers with regards to the user-create command in both keystone/start.sh and glance-api/start.sh. The race starts because both calls will reference the admin tenant and admin role, which crux will try to create if not present. Crux should not be run concurrently. Since it is being run concurrently, you can end up with two requests to create the admin tenant which will fail, causing the corresponding start.sh to exit and terminate the container.

The way I ran into this was I was using a 'docker run -d', which runs all the containers in the background. Often keystone would work, but the glance-api container would fail being unable to connect to keystone. Other times, both keystone and glance-api would fail because if glance won the race it was unable to find keystone.

 The solution is to let the keystone script create the admin tenant and role. We need a properly working wait_for command that will verify a few things.
1) ensure the admin tenant exists
2) ensure the admin role was created by calling an action like 'endpoint-list', which you must be admin to do

The reason this wasn't seen with kubernetes is because kube restarts the containers over and over again until they work.

Ryan Hallisey (rthall14)
Changed in kolla:
importance: Undecided → High
assignee: nobody → Charles Crouch (ccrouch)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.openstack.org/162358

Changed in kolla:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla (master)

Change abandoned by Steven Dake (<email address hidden>) on branch: master
Review: https://review.openstack.org/162358
Reason: Charles,

Thanks for the great commit log - it would be nice if everyone followed Charles lead on the great commit log work :)

As Ryan indicated, I asked him to cherry-pick this change into a different change, so it has already been merged. As a result I'm abandoning this change.

Note Ryan did add a "Co-authored-by" line and tried to change the committer in gerrit, but it wouldn't permit the submission. My apologies for you not getting "credit" in the repo.

Regards
-steve

Ryan Hallisey (rthall14)
Changed in kolla:
status: In Progress → Fix Committed
Steven Dake (sdake)
Changed in kolla:
milestone: none → liberty-1
Sam Yaple (s8m)
Changed in kolla:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.