Race condition in multithreaded Apache/WSGI setup

Bug #1482271 reported by Alexander Makarov on 2015-08-06
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Medium
Alexander Makarov

Bug Description

When configured as an Apache WSGI module a race condition is possible during keystone cache initialization:
https://github.com/openstack/keystone/blob/a597a86b854215835a4d54885daeb161d7b0efb8/keystone/common/kvs/core.py#L240
The operation raises exception region.RegionAlreadyConfigured.
This is a result of the race condition involving global 'application' variable being initialized several times (1 per thread).
application is required to be global according to Paste Deploy documentation:
http://pythonpaste.org/deploy/
Apache modwsgi documentation suggests protecting global objects with thread locks:
http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading#Building_A_Portable_Application

Fix proposed to branch: master
Review: https://review.openstack.org/210001

Changed in keystone:
assignee: nobody → Alexander Makarov (amakarov)
status: New → In Progress
Dolph Mathews (dolph) wrote :

As a workaround, you can use Apache's prefork model.

Changed in keystone:
importance: Undecided → Medium
Boris Bobrov (bbobrov) wrote :

Alexander, what exception are you getting?

I'm getting this: http://paste.openstack.org/show/444161/

Alexander Makarov (amakarov) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/222173

Alex, I am new to this community and am unclear on how to replicate your bug using devstack. Could you give me some details on how you were able to replicate this bug. Thank you in advance.

Alexander Makarov (amakarov) wrote :

Tom, it's usually tricky to catch a race condition, and if you want to try it for this issue, please do the following:
- Install keystone in multithreaded mode using Apache mod_wsgi
- Repeat the following:
-- Restart Apache
-- Perform some operation involving token storage or caching in several processes in parallel
Occasionally you'll receive an error containing a message that cache region is already configured.
Good luck!

description: updated
Changed in keystone:
milestone: none → newton-1

Reviewed: https://review.openstack.org/222173
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=691d497885cf4d8b39bb6ddb384b7c027bb52f95
Submitter: Jenkins
Branch: master

commit 691d497885cf4d8b39bb6ddb384b7c027bb52f95
Author: Boris Bobrov <email address hidden>
Date: Thu Sep 3 16:05:55 2015 +0500

    Move region configuration to a critical section

    Cache initialization performed on the request handling
    after Apache wsgi module start raises exception
    region.RegionAlreadyConfigured on race condition
    in multithreaded mode.

    Change-Id: I65f85aedd5b087499b889540417b9502e050ce7c
    Closes-bug: 1482271

Changed in keystone:
status: In Progress → Fix Released

This issue was fixed in the openstack/keystone 10.0.0.0b1 development milestone.

Change abandoned by Alexander Makarov (<email address hidden>) on branch: master
Review: https://review.openstack.org/210001
Reason: in favor of https://review.openstack.org/#/c/222173/

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers