local configuration is not allowed in "keystone-paste.ini"

[Dave Chen]

According to funcation docs in keystone/common/wsgi.py, local configuration is allowed in the paste.deploy config files, such as "redis_host".

https://github.com/openstack/keystone/blob/73ad4036d62b3aa7cf50e11ddf7bee8278bbe4d0/keystone/common/wsgi.py#L380-L402

And in the following implementation, local configuration is indeed read from paste.deploy config files.

https://github.com/openstack/keystone/blob/73ad4036d62b3aa7cf50e11ddf7bee8278bbe4d0/keystone/common/wsgi.py#L406

But, the local_config is not allowed in the constructor where only "app" is passed into the constructor.

https://github.com/openstack/keystone/blob/73ad4036d62b3aa7cf50e11ddf7bee8278bbe4d0/keystone/common/wsgi.py#L409-L411

So, if we configure paste.deploy config files like what the method docs says, it will always fails as:
[Sun Sep 14 22:40:37.316517 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] mod_wsgi (pid=22196): Exception occurred processing WSGI script '/var/www/keystone/admin'.
[Sun Sep 14 22:40:37.316554 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] Traceback (most recent call last):
[Sun Sep 14 22:40:37.316564 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] File "/var/www/keystone/admin", line 59, in <module>
[Sun Sep 14 22:40:37.316626 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] name=name)
[Sun Sep 14 22:40:37.316644 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
[Sun Sep 14 22:40:37.316795 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] return loadobj(APP, uri, name=name, **kw)
[Sun Sep 14 22:40:37.316803 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
[Sun Sep 14 22:40:37.316822 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] return context.create()
[Sun Sep 14 22:40:37.316827 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
[Sun Sep 14 22:40:37.316851 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] return self.object_type.invoke(self)
[Sun Sep 14 22:40:37.316856 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
[Sun Sep 14 22:40:37.316863 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] **context.local_conf)
[Sun Sep 14 22:40:37.316868 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 58, in fix_call
[Sun Sep 14 22:40:37.316919 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] reraise(*exc_info)
[Sun Sep 14 22:40:37.316927 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] File "/usr/lib/python2.7/dist-packages/paste/deploy/compat.py", line 23, in reraise
[Sun Sep 14 22:40:37.316979 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] exec('raise t, e, tb', dict(t=t, e=e, tb=tb))
[Sun Sep 14 22:40:37.316986 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
[Sun Sep 14 22:40:37.316995 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] val = callable(*args, **kw)
[Sun Sep 14 22:40:37.316999 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 28, in urlmap_factory
[Sun Sep 14 22:40:37.317074 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] app = loader.get_app(app_name, global_conf=global_conf)
[Sun Sep 14 22:40:37.317080 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
[Sun Sep 14 22:40:37.317100 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] name=name, global_conf=global_conf).create()
[Sun Sep 14 22:40:37.317104 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
[Sun Sep 14 22:40:37.317121 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] return self.object_type.invoke(self)
[Sun Sep 14 22:40:37.317125 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 207, in invoke
[Sun Sep 14 22:40:37.317132 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] app = filter(app)
[Sun Sep 14 22:40:37.317136 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] File "/opt/stack/keystone/keystone/common/wsgi.py", line 406, in _factory
[Sun Sep 14 22:40:37.317278 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] return cls(app, **local_config)
[Sun Sep 14 22:40:37.317301 2014] [:error] [pid 22196:tid 139709922383616] [remote 10.239.4.160:27590] TypeError: __init__() got an unexpected keyword argument 'redis_host'.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/134124

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/134125

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dave Chen (<email address hidden>) on branch: master
Review: https://review.openstack.org/121439
Reason: Abandon this patch, and fix this issue in the following two patches,
https://review.openstack.org/#/c/134124/
https://review.openstack.org/#/c/134125/

pls drop by when you are okay, thanks all.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/134125
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=fd92c3848bf4ae3dc823690759a54fd7675afccc
Submitter: Jenkins
Branch: master

commit fd92c3848bf4ae3dc823690759a54fd7675afccc
Author: Dave Chen <email address hidden>
Date: Thu Nov 13 23:20:23 2014 +0800

    Remove local conf information from paste-ini

    Since keystone's PasteDeploy configuration file has been separated
    from the main keystone configuration file, "keystone.conf", all
    local configuration or driver-specific configuration parameters
    must go in the main keystone configuration file instead of PasteDeploy
    conf file, i.e. configuration in "keystone-paste.ini" is not supported.

    This patch is doc specific changes in order to make usage more
    clear, code specific change is submitted in another
    patch(https://review.openstack.org/#/c/134124/).

    DocImpact
    Partial-Bug: #1369388

    Change-Id: Ie81eaa621b4517da1a00a723503ea4b8cbe84b8e

[David Stanek (dstanek)]

Is there an example config that breaks? I was able to add arbitrary config values to the application section without a problem. Since we have been moving away from this practice the local_conf is just thrown away, but I don't get an error.

I think the best way forward here would be to remove the part of the docstrings that talk about the Paste-style configuration and to raise a deprecation warning if used. Then after the proper amount of time has passed we remove the support altogether.

[David Stanek (dstanek)]

I'm also curious about what we want to support. It's possible for deployers to user other middleware and configure it using paste. Do we want to support this? Or should we be saying we'll only support a subset of paste syntax?

[Dave Chen (wei-d-chen)]

David, the issue is still there, since there is some code changing in the past half years, you may not hit the exception when you put some local conf under some filter.

You can simply reproduce the issue in this way:
1. config the paste ini.
[filter:url_normalize]
redis_host = "127.0.0.1"
paste.filter_factory = keystone.middleware:NormalizingFilter.factory

2. restart keystone service or some workaround to make it effective.
sudo touch /var/www/keystone/admin
sudo touch /var/www/keystone/main

3. just type follow command or any command you like
keystone tenant-list

4. the exception will be found the exception message in keystone's log
2015-04-20 16:58:29.329485 Traceback (most recent call last):
2015-04-20 16:58:29.329498 File "/var/www/keystone/main", line 25, in <module>
2015-04-20 16:58:29.329525 application = wsgi_server.initialize_application(name)
2015-04-20 16:58:29.329531 File "/opt/stack/keystone/keystone/server/wsgi.py", line 51, in initialize_application
2015-04-20 16:58:29.329540 startup_application_fn=loadapp)
2015-04-20 16:58:29.329550 File "/opt/stack/keystone/keystone/server/common.py", line 43, in setup_backends
2015-04-20 16:58:29.329558 res = startup_application_fn()
2015-04-20 16:58:29.329563 File "/opt/stack/keystone/keystone/server/wsgi.py", line 48, in loadapp
2015-04-20 16:58:29.329570 'config:%s' % config.find_paste_config(), name)
2015-04-20 16:58:29.329577 File "/opt/stack/keystone/keystone/service.py", line 45, in loadapp
2015-04-20 16:58:29.329673 controllers.latest_app = deploy.loadapp(conf, name=name)
2015-04-20 16:58:29.329683 File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2015-04-20 16:58:29.329840 return loadobj(APP, uri, name=name, **kw)
2015-04-20 16:58:29.329848 File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2015-04-20 16:58:29.329858 return context.create()
2015-04-20 16:58:29.329862 File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2015-04-20 16:58:29.329870 return self.object_type.invoke(self)
2015-04-20 16:58:29.329875 File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2015-04-20 16:58:29.329891 **context.local_conf)
2015-04-20 16:58:29.329895 File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 58, in fix_call
2015-04-20 16:58:29.329935 reraise(*exc_info)
2015-04-20 16:58:29.329942 File "/usr/lib/python2.7/dist-packages/paste/deploy/compat.py", line 23, in reraise
2015-04-20 16:58:29.329984 exec('raise t, e, tb', dict(t=t, e=e, tb=tb))
2015-04-20 16:58:29.329991 File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2015-04-20 16:58:29.329999 val = callable(*args, **kw)
2015-04-20 16:58:29.330003 File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 28, in urlmap_factory
2015-04-20 16:58:29.330091 app = loader.get_app(app_name, global_conf=global_conf)
2015-04-20 16:58:29.330098 File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2015-04-20 16:58:29.330116 name=name, global_conf=gl...

[David Stanek (dstanek)]

The issue have are having is specific our our implementation of keystone.common.wsgi.Middleware not accepting **local_config in its __init__. You can add arbitrary config values to our Applications without an issue. You can also add middleware from outside of Keystone and configure it that way.

Since we advertise using Paste I would expect this to work so I definitely think this need to be deprecated. The actual bug as you said above is that our Middelware impl is broken. We have two options to fix this:
 1. accept **local_config in the __init__ and do nothing with it, but making sure we add a deprecation message
 2. accept **local_config in the __init__ and raise an exception (maybe NotImplemented?)

[Dave Chen (wei-d-chen)]

David, thanks for the advice, accept **local_config and do nothing seems weird, someone may argue that why we pass a parameter but do nothing at all, I like the second and some warning message is good in my opinion.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/225177

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dave Chen (<email address hidden>) on branch: master
Review: https://review.openstack.org/225177
Reason: Thanks for the comments, this has been consolidate into it's parent.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/134124
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=33994ed2c4202dc879ee69fe47e2e1321c74ba0c
Submitter: Jenkins
Branch: master

commit 33994ed2c4202dc879ee69fe47e2e1321c74ba0c
Author: Dave Chen <email address hidden>
Date: Thu Nov 13 23:04:51 2014 +0800

    Fix the issues found with local conf

    Since keystone's paste.deploy configuration file has been separated
    from the main keystone configuration file (keystone.conf), all
    local configuration or driver-specific configuration parameters
    must be in the main keystone configuration file instead of PasteDeploy
    conf file.

    This patch changes to log a warning if there's any local_conf defined in
    PasteDeploy INI and remove `local_config` argument from wsgi middleware
    since it doesn't work.

    DocImpact
    Closes-Bug: #1369388

    Change-Id: Iaf6ad869b61e4330a4ea48e606fd9eda69b9cd12

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/keystone 9.0.0.0b1 development milestone.