Keystone errors 'ArgsAlreadyParsedError' and 'osrandom engine already registered'

[Jesse Pretorius]

Keystone errors are showing up in Kilo installs, resulting in fairly obscure inconsistent failures for any actions against Keystone.

Stack trace 1:

mod_wsgi (pid=21383): Exception occurred processing WSGI script '/var/www/cgi-bin/keystone/main'.
Traceback (most recent call last):
  File "/var/www/cgi-bin/keystone/main", line 25, in <module>
    application = wsgi_server.initialize_application(name)
  File "/usr/local/lib/python2.7/dist-packages/keystone/server/wsgi.py", line 38, in initialize_application
    common.configure()
  File "/usr/local/lib/python2.7/dist-packages/keystone/server/common.py", line 28, in configure
    config.configure()
  File "/usr/local/lib/python2.7/dist-packages/keystone/common/config.py", line 1112, in configure
    help='Do not monkey-patch threading system modules.'))
  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1794, in __inner
    result = f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1967, in register_cli_opt
    raise ArgsAlreadyParsedError("cannot register CLI option")
ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option

Stack trace 2:

Traceback (most recent call last):
  File "/var/www/cgi-bin/keystone/admin", line 25, in <module>
    application = wsgi_server.initialize_application(name)
  File "/usr/local/lib/python2.7/dist-packages/keystone/server/wsgi.py", line 51, in initialize_application
    startup_application_fn=loadapp)
  File "/usr/local/lib/python2.7/dist-packages/keystone/server/common.py", line 43, in setup_backends
    res = startup_application_fn()
  File "/usr/local/lib/python2.7/dist-packages/keystone/server/wsgi.py", line 48, in loadapp
    'config:%s' % config.find_paste_config(), name)
  File "/usr/local/lib/python2.7/dist-packages/keystone/service.py", line 45, in loadapp
    controllers.latest_app = deploy.loadapp(conf, name=name)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
    return loadobj(APP, uri, name=name, **kw)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
    return context.create()
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
    return self.object_type.invoke(self)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
    **context.local_conf)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
    val = callable(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 31, in urlmap_factory
    app = loader.get_app(app_name, global_conf=global_conf)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
    name=name, global_conf=global_conf).create()
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 362, in app_context
    APP, name=name, global_conf=global_conf)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 450, in get_context
    global_additions=global_additions)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 562, in _pipeline_app_context
    for name in pipeline[:-1]]
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 458, in get_context
    section)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 517, in _context_from_explici
t
    value = import_string(found_expr)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 22, in import_string
    return pkg_resources.EntryPoint.parse("x=" + s).load(False)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2355, in load
    return self.resolve()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2361, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python2.7/dist-packages/keystone/contrib/federation/routers.py", line 17, in <module>
    from keystone.contrib.federation import controllers
  File "/usr/local/lib/python2.7/dist-packages/keystone/contrib/federation/controllers.py", line 29, in <mod
ule>
    from keystone.contrib.federation import idp as keystone_idp
  File "/usr/local/lib/python2.7/dist-packages/keystone/contrib/federation/idp.py", line 23, in <module>
    from saml2 import client_base
  File "/usr/local/lib/python2.7/dist-packages/saml2/client_base.py", line 13, in <module>
    from saml2.entity import Entity
  File "/usr/local/lib/python2.7/dist-packages/saml2/entity.py", line 9, in <module>
    from saml2.metadata import ENDPOINTS
  File "/usr/local/lib/python2.7/dist-packages/saml2/metadata.py", line 3, in <module>
    from saml2.sigver import security_context
  File "/usr/local/lib/python2.7/dist-packages/saml2/sigver.py", line 8, in <module>
    from OpenSSL import crypto
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/rand.py", line 11, in <module>
    from OpenSSL._util import (
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/_util.py", line 7, in <module>
    binding = Binding()
  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 47, in
 __init__
    self._ensure_ffi_initialized()
  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 78, in
 _ensure_ffi_initialized
    cls._register_osrandom_engine()
  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 54, in
 _register_osrandom_engine
    raise RuntimeError("osrandom engine already registered")

The upstream recommendation has been to reduce the number of threads to 1 instead of the current dynamic value.

https://github.com/dolph/keystone-deploy/blob/master/playbooks/roles/keystone/templates/apache/keystone.vhost#L1-L3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/225145

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: kilo
Review: https://review.openstack.org/229275

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jesse Pretorius (<email address hidden>) on branch: kilo
Review: https://review.openstack.org/229275
Reason: This was a test - will revisit this when https://review.openstack.org/225145 merges.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/225145
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=0513c950fba3071a36661325969c1b2831a85d47
Submitter: Jenkins
Branch: master

commit 0513c950fba3071a36661325969c1b2831a85d47
Author: Jesse Pretorius <email address hidden>
Date: Fri Sep 18 14:29:46 2015 +0100

    Adjust default Keystone httpd processes and threads

    This patch sets the Apache threads and processes for Keystone to match
    the recommendations based on performance testing done by the Keystone
    core team.

    This resolves an issue where the cryptography library will not allow
    multiple threads to register the osrandom engine.

    To compensate for the reduction in threads, the number of processes has
    been increased. Both the values remain tunable to suit a deployers needs.

    Change-Id: I1e94a91642b904b865ddf56983239838ed9a15e6
    Closes-Bug: #1497283

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229275
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=1efd2e4f587aa899b84605ed7727466926132fdb
Submitter: Jenkins
Branch: kilo

commit 1efd2e4f587aa899b84605ed7727466926132fdb
Author: Jesse Pretorius <email address hidden>
Date: Fri Sep 18 14:29:46 2015 +0100

    Adjust default Keystone httpd processes and threads

    This patch sets the Apache threads and processes for Keystone to match
    the recommendations based on performance testing done by the Keystone
    core team.

    This resolves an issue where the cryptography library will not allow
    multiple threads to register the osrandom engine.

    To compensate for the reduction in threads, the number of processes has
    been increased. Both the values remain tunable to suit a deployers needs.

    Change-Id: I1e94a91642b904b865ddf56983239838ed9a15e6
    Closes-Bug: #1497283
    (cherry picked from commit 0513c950fba3071a36661325969c1b2831a85d47)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.11 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 11.2.12 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.14 release.