Keystone import errors and upstream config mismatch

[Jesse Pretorius]

With the latest updates in liberty, Keystone's Apache configuration currently does not work with the following logs showing in the keystone error log:

[Tue Aug 04 11:58:39.548474 2015] [:info] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] mod_wsgi (pid=9192, process='keystone', application='aio1_keystone_container-c849f6f4:5000|'): Loading WSGI script '/var/www/cgi-bin/keystone/main'.
[Tue Aug 04 11:58:39.549852 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] mod_wsgi (pid=9192): Target WSGI script '/var/www/cgi-bin/keystone/main' cannot be loaded as Python module.
[Tue Aug 04 11:58:39.549871 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] mod_wsgi (pid=9192): Exception occurred processing WSGI script '/var/www/cgi-bin/keystone/main'.
[Tue Aug 04 11:58:39.549900 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] Traceback (most recent call last):
[Tue Aug 04 11:58:39.549931 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] File "/var/www/cgi-bin/keystone/main", line 17, in <module>
[Tue Aug 04 11:58:39.549971 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] from keystone.server import wsgi as wsgi_server
[Tue Aug 04 11:58:39.549993 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] File "/usr/local/lib/python2.7/dist-packages/keystone/server/wsgi.py", line 29, in <module>
[Tue Aug 04 11:58:39.550016 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] from keystone import config
[Tue Aug 04 11:58:39.550029 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] File "/usr/local/lib/python2.7/dist-packages/keystone/config.py", line 22, in <module>
[Tue Aug 04 11:58:39.550047 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] from keystone.common import config
[Tue Aug 04 11:58:39.550059 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] File "/usr/local/lib/python2.7/dist-packages/keystone/common/config.py", line 16, in <module>
[Tue Aug 04 11:58:39.550077 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] import oslo_messaging
[Tue Aug 04 11:58:39.550105 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/__init__.py", line 18, in <module>
[Tue Aug 04 11:58:39.550125 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] from .notify import *
[Tue Aug 04 11:58:39.550136 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/notify/__init__.py", line 26, in <module>
[Tue Aug 04 11:58:39.550154 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] from .listener import *
[Tue Aug 04 11:58:39.550165 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/notify/listener.py", line 106, in <module>
[Tue Aug 04 11:58:39.550182 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] from oslo_messaging.notify import dispatcher as notify_dispatcher
[Tue Aug 04 11:58:39.550210 2015] [:error] [pid 9192:tid 139910207579904] [remote 172.29.236.100:55913] ImportError: cannot import name dispatcher
[Tue Aug 04 11:58:50.248257 2015] [:info] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] mod_wsgi (pid=9197, process='keystone', application='aio1_keystone_container-c849f6f4:35357|'): Loading WSGI script '/var/www/cgi-bin/keystone/admin'.
[Tue Aug 04 11:58:50.249714 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] mod_wsgi (pid=9197): Target WSGI script '/var/www/cgi-bin/keystone/admin' cannot be loaded as Python module.
[Tue Aug 04 11:58:50.249733 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] mod_wsgi (pid=9197): Exception occurred processing WSGI script '/var/www/cgi-bin/keystone/admin'.
[Tue Aug 04 11:58:50.249762 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] Traceback (most recent call last):
[Tue Aug 04 11:58:50.249793 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] File "/var/www/cgi-bin/keystone/admin", line 17, in <module>
[Tue Aug 04 11:58:50.249834 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] from keystone.server import wsgi as wsgi_server
[Tue Aug 04 11:58:50.249848 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] File "/usr/local/lib/python2.7/dist-packages/keystone/server/wsgi.py", line 29, in <module>
[Tue Aug 04 11:58:50.249868 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] from keystone import config
[Tue Aug 04 11:58:50.249881 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] File "/usr/local/lib/python2.7/dist-packages/keystone/config.py", line 22, in <module>
[Tue Aug 04 11:58:50.249900 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] from keystone.common import config
[Tue Aug 04 11:58:50.249917 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] File "/usr/local/lib/python2.7/dist-packages/keystone/common/config.py", line 16, in <module>
[Tue Aug 04 11:58:50.249938 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] import oslo_messaging
[Tue Aug 04 11:58:50.249949 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/__init__.py", line 18, in <module>
[Tue Aug 04 11:58:50.249967 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] from .notify import *
[Tue Aug 04 11:58:50.249978 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/notify/__init__.py", line 26, in <module>
[Tue Aug 04 11:58:50.249996 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] from .listener import *
[Tue Aug 04 11:58:50.250007 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/notify/listener.py", line 106, in <module>
[Tue Aug 04 11:58:50.250042 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] from oslo_messaging.notify import dispatcher as notify_dispatcher
[Tue Aug 04 11:58:50.250071 2015] [:error] [pid 9197:tid 139910232758016] [remote 172.29.236.100:61289] ImportError: cannot import name dispatcher

Also, https://bugs.launchpad.net/keystone/+bug/1456441 outlines another issue which may affect os-ansible-deployment as we also do not set a group name.

Finally, the upstream sample config has been updated with a desirable configuration which, when implemented, resolves the error seen with the latest code updates.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/209047

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/209047
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=98c9768497145e03d349f6ddee8dc89b4bda13f9
Submitter: Jenkins
Branch: master

commit 98c9768497145e03d349f6ddee8dc89b4bda13f9
Author: Jesse Pretorius <email address hidden>
Date: Mon Aug 3 22:10:43 2015 -0700

    Improve Keystone Apache configuration

    This patch does the following:

    1. Introduces two new Keystone variables which are useful for
       debugging the Keystone service. The values are defaulted
       to the same values as before the patch.
       - keystone_wsgi_processes: number of wsgi processes to run
       - keystone_wsgi_threads: number of wsgi threads to run

    4. Moves the keystone service and admin processes into their
       own wsgi groups for better isolation.

    5. Sets each wsgi process to run under the keystone group.

    6. Bring the configuration file in line with the upstream
       recommended configuration as at 4 Aug 2015 in order to
       overcome import race conditions.

    Change-Id: I861d1ef233dd6121452dc0e9e590d2d9f9b7973e
    Closes-Bug: #1481339

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: kilo
Review: https://review.openstack.org/212021

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/212021
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=bf20a18d19a29e64b1a8d8f1fa9292e8a38c4333
Submitter: Jenkins
Branch: kilo

commit bf20a18d19a29e64b1a8d8f1fa9292e8a38c4333
Author: Jesse Pretorius <email address hidden>
Date: Mon Aug 3 22:10:43 2015 -0700

    Improve Keystone Apache configuration

    This patch does the following:

    1. Introduces two new Keystone variables which are useful for
       debugging the Keystone service. The values are defaulted
       to the same values as before the patch.
       - keystone_wsgi_processes: number of wsgi processes to run
       - keystone_wsgi_threads: number of wsgi threads to run

    4. Moves the keystone service and admin processes into their
       own wsgi groups for better isolation.

    5. Sets each wsgi process to run under the keystone group.

    6. Bring the configuration file in line with the upstream
       recommended configuration as at 4 Aug 2015 in order to
       overcome import race conditions.

    Change-Id: I861d1ef233dd6121452dc0e9e590d2d9f9b7973e
    Closes-Bug: #1481339
    (cherry picked from commit 98c9768497145e03d349f6ddee8dc89b4bda13f9)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.11 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 11.2.12 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.14 release.