Keystone token_flush cron job running at the same time in the controller node causes deadlock

[Haneef Ali]

Keystone token_flush is a cron job and it is scheduled to run at the same time in all the nodes. Running the same cron job at the same time causes both deadlock and transaction time out in other nodes. It is better to run the the job just in bootstrap node

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/141882

[Haneef Ali (haneef)]

Detail description of the defect.

Token flush does
     Select expiry date
     delete based on expiry date

Select will take READ lock and delete will take exclusive ROW lock in mysql. So if the same operation is running on all the controller nodes, then other nodes will be unnecessaily waiting for the lock since all of them expects LOCK for the same rows.

Easy way to figure to check is, to grep for transaction read time out message in all the controller nodes. You will always see read time out in all the controller node except one for a particular run

[Clint Byrum (clint-fewbar)]

https://review.openstack.org/#/c/142420/ is an alternative suggestion on how to solve this.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dan Prince (<email address hidden>) on branch: master
Review: https://review.openstack.org/141882