OSError raised on directory creation race

Bug #1649735 reported by Steve Baker on 2016-12-14
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
keystonemiddleware
Medium
Colleen Murphy

Bug Description

Doing a python import of keystonemiddleware.auth_token has the side-effect of creating a directory for token signing verification.

The problem is that keystonemiddleware.auth_token._signing_dir._verify_signing_dir has a race in checking for and creating the directory. If multiple heat-api processes are spawned at the same time then OSError is raised for creating a directory that already exists.

This method should tolerate OSError being raised for os.makedirs, then verify the directory regardless.

Here is a stack trace for heat-api demonstrating the issue. This maybe affects heat-api more than other services due to the timing of spawning the workers.

2015-08-10 13:34:14.833 8160 CRITICAL heat-api [-] OSError: [Errno 17] File exists: '/tmp/keystone-signing-heat'
2015-08-10 13:34:14.833 8160 TRACE heat-api Traceback (most recent call last):
2015-08-10 13:34:14.833 8160 TRACE heat-api File "/usr/bin/heat-api", line 58, in <module>
2015-08-10 13:34:14.833 8160 TRACE heat-api app = config.load_paste_app()
2015-08-10 13:34:14.833 8160 TRACE heat-api File "/usr/lib/python2.7/site-packages/heat/common/config.py", line 397, in load_paste_app
2015-08-10 13:34:14.833 8160 TRACE heat-api app = wsgi.paste_deploy_app(conf_file, app_name, cfg.CONF)
2015-08-10 13:34:14.833 8160 TRACE heat-api File "/usr/lib/python2.7/site-packages/heat/common/wsgi.py", line 898, in paste_deploy_app
2015-08-10 13:34:14.833 8160 TRACE heat-api return deploy.loadapp("config:%s" % paste_config_file, name=app_name)
2015-08-10 13:34:14.833 8160 TRACE heat-api File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2015-08-10 13:34:14.833 8160 TRACE heat-api return loadobj(APP, uri, name=name, **kw)
2015-08-10 13:34:14.833 8160 TRACE heat-api File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2015-08-10 13:34:14.833 8160 TRACE heat-api return context.create()
2015-08-10 13:34:14.833 8160 TRACE heat-api File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2015-08-10 13:34:14.833 8160 TRACE heat-api return self.object_type.invoke(self)
2015-08-10 13:34:14.833 8160 TRACE heat-api File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 207, in invoke
2015-08-10 13:34:14.833 8160 TRACE heat-api app = filter(app)
2015-08-10 13:34:14.833 8160 TRACE heat-api File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 1122, in auth_filter
2015-08-10 13:34:14.833 8160 TRACE heat-api return AuthProtocol(app, conf)
2015-08-10 13:34:14.833 8160 TRACE heat-api File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 521, in __init__
2015-08-10 13:34:14.833 8160 TRACE heat-api directory_name=self._conf_get('signing_dir'), log=self._LOG)
2015-08-10 13:34:14.833 8160 TRACE heat-api File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_signing_dir.py", line 38, in __init__
2015-08-10 13:34:14.833 8160 TRACE heat-api self._verify_signing_dir()
2015-08-10 13:34:14.833 8160 TRACE heat-api File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_signing_dir.py", line 83, in _verify_signing_dir
2015-08-10 13:34:14.833 8160 TRACE heat-api os.makedirs(self._directory_name, stat.S_IRWXU)
2015-08-10 13:34:14.833 8160 TRACE heat-api File "/usr/lib64/python2.7/os.py", line 157, in makedirs
2015-08-10 13:34:14.833 8160 TRACE heat-api mkdir(name, mode)
2015-08-10 13:34:14.833 8160 TRACE heat-api OSError: [Errno 17] File exists: '/tmp/keystone-signing-heat'
2015-08-10 13:34:14.833 8160 TRACE heat-api
/var/log/heat/heat-api.log (END)

Lance Bragstad (lbragstad) wrote :

Hi Steve,

You think you're hitting the issue because there are a large number of workers starting within a short period of time?

Steve Baker (steve-stevebaker) wrote :

Only 4 workers seems to be enough to trigger this fairly often for heat-api on a TripleO undercloud or overcloud.

Steve Martinelli (stevemar) wrote :

Thanks for triaging while reporting the bug!

tags: added: low-hanging-fruit
Changed in keystonemiddleware:
status: New → Triaged
importance: Undecided → Medium

Fix proposed to branch: master
Review: https://review.openstack.org/613651

Changed in keystonemiddleware:
assignee: nobody → Morgan Fainberg (mdrnstm)
status: Triaged → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/613675

Reviewed: https://review.openstack.org/613651
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=7e1b53625990bb08425645cb92f36e16bd67db7f
Submitter: Zuul
Branch: master

commit 7e1b53625990bb08425645cb92f36e16bd67db7f
Author: Morgan Fainberg <email address hidden>
Date: Fri Oct 26 10:32:28 2018 -0700

    Stop supporting revocation list

    With keystone's move to eliminating pki, pkiz, and uuid tokens the
    revocation list is no longer generated. Keystonemiddleware no longer
    needs to attempt to retrieve it and reference it.

    Change-Id: Ief3bf1941e62f9136dbed11877bca81c4102041b
    closes-bug: #1361743
    partial-bug: #1649735
    partial-bug: #1736985

Changed in keystonemiddleware:
assignee: Morgan Fainberg (mdrnstm) → Gage Hugo (gagehugo)
Gage Hugo (gagehugo) on 2019-05-15
Changed in keystonemiddleware:
assignee: Gage Hugo (gagehugo) → Morgan Fainberg (mdrnstm)
Changed in keystonemiddleware:
assignee: Morgan Fainberg (mdrnstm) → Gage Hugo (gagehugo)
Gage Hugo (gagehugo) on 2019-05-21
Changed in keystonemiddleware:
assignee: Gage Hugo (gagehugo) → Morgan Fainberg (mdrnstm)
Changed in keystonemiddleware:
assignee: Morgan Fainberg (mdrnstm) → Gage Hugo (gagehugo)
Gage Hugo (gagehugo) on 2019-05-29
Changed in keystonemiddleware:
assignee: Gage Hugo (gagehugo) → Morgan Fainberg (mdrnstm)
Changed in keystonemiddleware:
assignee: Morgan Fainberg (mdrnstm) → Colleen Murphy (krinkle)

Reviewed: https://review.opendev.org/613675
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=b3e84aafc0302b1a87754f438696794076ba844f
Submitter: Zuul
Branch: master

commit b3e84aafc0302b1a87754f438696794076ba844f
Author: Morgan Fainberg <email address hidden>
Date: Fri Oct 26 12:33:28 2018 -0700

    Remove PKI/PKIZ support

    Keystone server no longer supports PKI/PKIZ. This change removes
    keystonemiddleware's support of PKI/PKIZ and associated code.

    Change-Id: I9a6639a2aa3774be61972d57f38220f66fd5c0e8
    closes-bug: #1649735
    partial-bug: #1736985

Changed in keystonemiddleware:
status: In Progress → Fix Released

This issue was fixed in the openstack/keystonemiddleware 7.0.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers