Restarting the web server causes users to get kicked out

[Erik Olof Gunnar Andersson]

Starting with Django 1.9 users are kicked out to the login screen after the web server is restarted. This is especially severe when running Horizon with a high number of processes.

However, if Horizon is running with Django 1.8.19 or older, Horizon can be restarted with little to no impact.

Reproduced in Devstack stable/queens using the following additional steps.

1) Configured Apache with 30 processes.
> WSGIDaemonProcess horizon user=stack group=stack processes=30 threads=1 home=/opt/stack/horizon display-name=%{GROUP}

2) Configure Horizon to use Memcached.
SESSION_ENGINE = 'django.contrib.sessions.backends.cache'
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
    },
}

3) Log in to Horizon.

4) Restarted Apache.

5) Hit F5 and you will be kicked out to the login screen. Keep hitting F5 or clicking on pages and you will randomly be kicked out back to the login screen.

It will keep kicking you out until all processes has been used at least once.

[Ivan Kolodyazhny (e0ne)]

I agree that it's not good UX here. I'm setting this issue as 'confirmed' for now until we'll figure out if we can fix it from the horizon side. It could be a Django issue too

[Mathieu Gagné (mgagne)]

I managed to identify the issue.

The "request" attribute is not available [1] when the first request happens after a server restart. This means the whole authentication mechanism falls back to "AnonymousUser" and you are getting redirected to the login page due to horizon.exceptions.NotAuthenticated being raised. [2]

The "request" attribute is monkey patched by openstack_auth.utils. [3] But there is nowhere in Horizon where this module is imported at startup. It's only introspected by openstack_dashboarb.urls due to AUTHENTICATION_URLS setting. [4]

This means the monkey patching won't happen until a user requests a page under auth/ which will have the side-effect of monkey patching django.contrib.auth.middleware as expected.

This also means that once a request is done, the session will be restored and you will be properly authenticated if you directly request a page other than auth/.

If you have multiple processes/workers, you will need to hit all of them once to fix the session issue and trigger the monkey patching for all of them.

[1] https://github.com/openstack/horizon/blob/a53f012fa1c0724ee4d532e782e6b2fe88ef8fa8/openstack_auth/backend.py#L66-L87
[2] https://github.com/openstack/horizon/blob/a53f012fa1c0724ee4d532e782e6b2fe88ef8fa8/horizon/decorators.py#L40-L54
[3] https://github.com/openstack/horizon/blob/a53f012fa1c0724ee4d532e782e6b2fe88ef8fa8/openstack_auth/utils.py#L63
[4] https://github.com/openstack/horizon/blob/a53f012fa1c0724ee4d532e782e6b2fe88ef8fa8/openstack_dashboard/urls.py#L52

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/614673
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=0d163613265e036818fe567793a4fc88fe140d4a
Submitter: Zuul
Branch: master

commit 0d163613265e036818fe567793a4fc88fe140d4a
Author: Mathieu Gagné <email address hidden>
Date: Wed Oct 31 22:24:31 2018 -0400

    Fix django.contrib.auth.middleware monkey patching

    The "request" attribute is not available in
    openstack_auth.backend.KeystoneBackend.get_user when session data is restored
    and it's the first request to happen after a server restart.

    As stated by the function document, the "request" attribute needs to be
    monkey-patched by openstack_auth.utils.patch_middleware_get_user
    for this function to work properly.

    This should happen in openstack_auth.urls at import time. But there is nowhere
    in Horizon where this module is imported at startup. It's only introspected
    by openstack_dashboard.urls due to AUTHENTICATION_URLS setting.

    Without this monkey-patching, the whole authentication mechanism falls back
    to "AnonymousUser" and you will get redirected to the login page due
    to horizon.exceptions.NotAuthenticated being raised by
    horizon.decorators.require_auth as request.user.is_authenticated will be False.

    But if a user requests a page under auth/, it will have the side-effect of
    monkey-patching django.contrib.auth.middleware as expected. This means that
    once this request is completed, all following requests to pages other than
    the ones under auth/ will have there sessions properly restored and
    you will be properly authenticated.

    Therefore this change introduces a dummy middleware which sole purpose is
    to perform this monkey-patching as early as possible.

    There is also some cleanup to get rid of the previous attempts at
    monkeypatching.

    Closes-bug: #1764622
    Change-Id: Ib9912090a87b716e7f5710f6f360b0df168ec2e3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/618997

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/619001

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/618997
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=8851866aad5b3826f0a3d569405707bffa593c80
Submitter: Zuul
Branch: stable/rocky

commit 8851866aad5b3826f0a3d569405707bffa593c80
Author: Mathieu Gagné <email address hidden>
Date: Wed Oct 31 22:24:31 2018 -0400

    Fix django.contrib.auth.middleware monkey patching

    The "request" attribute is not available in
    openstack_auth.backend.KeystoneBackend.get_user when session data is restored
    and it's the first request to happen after a server restart.

    As stated by the function document, the "request" attribute needs to be
    monkey-patched by openstack_auth.utils.patch_middleware_get_user
    for this function to work properly.

    This should happen in openstack_auth.urls at import time. But there is nowhere
    in Horizon where this module is imported at startup. It's only introspected
    by openstack_dashboard.urls due to AUTHENTICATION_URLS setting.

    Without this monkey-patching, the whole authentication mechanism falls back
    to "AnonymousUser" and you will get redirected to the login page due
    to horizon.exceptions.NotAuthenticated being raised by
    horizon.decorators.require_auth as request.user.is_authenticated will be False.

    But if a user requests a page under auth/, it will have the side-effect of
    monkey-patching django.contrib.auth.middleware as expected. This means that
    once this request is completed, all following requests to pages other than
    the ones under auth/ will have there sessions properly restored and
    you will be properly authenticated.

    Therefore this change introduces a dummy middleware which sole purpose is
    to perform this monkey-patching as early as possible.

    There is also some cleanup to get rid of the previous attempts at
    monkeypatching.

    Closes-bug: #1764622

    Conflicts:
     openstack_dashboard/settings.py
     openstack_dashboard/test/helpers.py

    Change-Id: Ib9912090a87b716e7f5710f6f360b0df168ec2e3
    (cherry picked from commit 0d163613265e036818fe567793a4fc88fe140d4a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/619001
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=6ae1f5e21d87ba3e11bd124eb1268823a1641205
Submitter: Zuul
Branch: stable/queens

commit 6ae1f5e21d87ba3e11bd124eb1268823a1641205
Author: Mathieu Gagné <email address hidden>
Date: Wed Oct 31 22:24:31 2018 -0400

    Fix django.contrib.auth.middleware monkey patching

    The "request" attribute is not available in
    openstack_auth.backend.KeystoneBackend.get_user when session data is restored
    and it's the first request to happen after a server restart.

    As stated by the function document, the "request" attribute needs to be
    monkey-patched by openstack_auth.utils.patch_middleware_get_user
    for this function to work properly.

    This should happen in openstack_auth.urls at import time. But there is nowhere
    in Horizon where this module is imported at startup. It's only introspected
    by openstack_dashboard.urls due to AUTHENTICATION_URLS setting.

    Without this monkey-patching, the whole authentication mechanism falls back
    to "AnonymousUser" and you will get redirected to the login page due
    to horizon.exceptions.NotAuthenticated being raised by
    horizon.decorators.require_auth as request.user.is_authenticated will be False.

    But if a user requests a page under auth/, it will have the side-effect of
    monkey-patching django.contrib.auth.middleware as expected. This means that
    once this request is completed, all following requests to pages other than
    the ones under auth/ will have there sessions properly restored and
    you will be properly authenticated.

    Therefore this change introduces a dummy middleware which sole purpose is
    to perform this monkey-patching as early as possible.

    There is also some cleanup to get rid of the previous attempts at
    monkeypatching.

    Closes-bug: #1764622

    [backport specific notice]
    Queens horizon supports Django 1.8 and 1.11.
    The key point is Django 2.0 is not supported.
    Django 2.0 dropped the legacy Django middleware interface and
    the new interface is not supported in Django 1.8.
    Thus, this backport changes OpenstackAuthMonkeyPatchMiddleware in
    openstack_auth/middlware.py to use the legacy middleware interface.

    Conflicts:
     openstack_dashboard/settings.py
     openstack_dashboard/test/helpers.py

    Change-Id: Ib9912090a87b716e7f5710f6f360b0df168ec2e3
    (cherry picked from commit 0d163613265e036818fe567793a4fc88fe140d4a)
    (cherry picked from commit 8851866aad5b3826f0a3d569405707bffa593c80)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 14.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 15.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 13.0.2 release.