Error when logging back in after timeout

[Rob Cresswell]

Internal Server Error: /auth/login/
Traceback (most recent call last):
  File "/Users/robcresswell/horizon/.venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/robcresswell/horizon/.venv/lib/python2.7/site-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
    return view(request, *args, **kwargs)
  File "/Users/robcresswell/horizon/.venv/lib/python2.7/site-packages/django/utils/decorators.py", line 110, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/Users/robcresswell/horizon/.venv/lib/python2.7/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/Users/robcresswell/horizon/.venv/lib/python2.7/site-packages/openstack_auth/views.py", line 112, in login
    **kwargs)
  File "/Users/robcresswell/horizon/.venv/lib/python2.7/site-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
    return view(request, *args, **kwargs)
  File "/Users/robcresswell/horizon/.venv/lib/python2.7/site-packages/django/utils/decorators.py", line 110, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/Users/robcresswell/horizon/.venv/lib/python2.7/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/Users/robcresswell/horizon/.venv/lib/python2.7/site-packages/django/contrib/auth/views.py", line 51, in login
    auth_login(request, form.get_user())
  File "/Users/robcresswell/horizon/.venv/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 102, in login
    if _get_user_session_key(request) != user.pk or (
  File "/Users/robcresswell/horizon/.venv/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 59, in _get_user_session_key
    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
  File "/Users/robcresswell/horizon/.venv/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 969, in to_python
    params={'value': value},
ValidationError: [u"'63bb5e0a7a3c42049236df7438246e57' value must be an integer."]

Django: 1.8.4
D-O-A: 1.4.0

My initial thoughts are: the move to Django 1.8 has introduced some different behaviour OR https://review.openstack.org/#/c/142481/ has caused shenanigans.

[Lin Hua Cheng (lin-hua-cheng)]

environment:
Django (1.8.4)
django-openstack-auth==1.4.0

[Lin Hua Cheng (lin-hua-cheng)]

I can't reproduce the issue.

Used the same env :

Django==1.8.4
django-appconf==1.0.1
django-babel==0.4.0
django-compressor==1.5
django-nose==1.4.1
django-openstack-auth==1.4.0
django-pyscss==2.0.2

Maybe Matthias have more luck.

[Richard Jones (r1chardj0n3s)]

I have been able to reproduce this, though intermittently. Am looking into it further.

[Neill Cox (neillc)]

I'm also unable to reproduce this.

Using the same environment as Lin Hua Cheng

[Richard Jones (r1chardj0n3s)]

To reliably reproduce, first log in as "admin" and then use the login form to log in as "demo". Django attempts to determine if the existing session belongs to the same user as the just-authenticated user, and in doing so tries to force the user object's PK to int().

I believe the issue is that Django assumes that the user model PK is an integer, and django-openstack-auth is setting it to a hash string. Unfortunately, as far as I can tell, you can't change the type of the PK from the integer, so it would seem that we need to set the user's PK value to an integer. Exactly what that integer should be could be tricky, though I'm not sure it actually matters at all, since it's not a reference into an actual database.

Setting the User id property to something other than the user_id hash will break all manner of openstack_dashboard code that assumes user.id is the same as user.token.user['id'] - a very rough grep-based estimate shows 105 uses of user.id (and a bunch of those are in tests which will probably need knock-on changes to mocks).

It's really tempting to just monkey-patch django.contrib.auth._get_user_session_key() ...

[Richard Jones (r1chardj0n3s)]

Oh, a third option (apart from monkey-patching django's _get_user_session_key() and replacing all instances of user.id with user.token.user['id']) is to replace the User class in django-openstack-auth with one that doesn't base itself off of AbstractBaseUser, and thus implements all of its own model attributes -- just to replace the id attribute.

A heck of a lot of work... the monkey-patching approach is looking really attractive right now...

[Richard Jones (r1chardj0n3s)]

Oh, so django-openstack-auth already monkey-patches django. Righto then.

[Richard Jones (r1chardj0n3s)]

Whoops, sorry folks, I stuffed up launchpadding there, and horizon got removed for a bit.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/220382

[Richard Jones (r1chardj0n3s)]

Clearer instructions to reproduce:

1) login into a devstack as "admin"
2) using the browser back button (DO NOT LOG OUT) go back to the login form
3) login using "demo"

[David Lyle (david-lyle)]

Ok, I can reproduce on Chrome, Firefox disables the submit button.

[Timur Sufiev (tsufiev-x)]

Interesting. At first, no matter how hard I tried, I was only able to get 'Forbidden (CSRF token missing or incorrect.): /auth/login/' error (both Chrome and Chromium on Ubuntu 14.04, fresh devstack, Django and django-openstack-auth as above, DEBUG = True). Submit was disabled in FF. But that was on local Horizon (run withing Django development server, WEBROOT = '/').

Then it tried Horizon on devstack (Apache, WEBROOT = '/dashboard/') and it was reproducible. Finally, just because of curiosity I tweaked Horizon & Apache on devstack to use WEBROOT = '/' and the bug was still reproducible. Weird...

[Timur Sufiev (tsufiev-x)]

One more update: setting DEBUG = True on devstack causes CSRF token error to appear. Setting DEBUG = False on local Horizon doesn't change anything (still CSRF error).

[Timur Sufiev (tsufiev-x)]

Finally, I've found a way for stable bug reproducing inside development environment.

1. Set token.expiry at /etc/keystone/keystone.conf at devstack to really small value (say, 60 seconds).
2. login as admin
3. wait for a minute
4. refresh the page
5. once logged out, try to login as demo user
6. once you're banned from accessing /admin page...
7. set all the needed breakpoints and login as admin

So far I was able to get into django.contrib.auth.__init__, code

def _get_user_session_key(request):
    # This value in the session is always serialized to a string, so we need
    # to convert it back to Python whenever we access it.
    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])

which explained me, why Richard's fix is working. Yet I'm still wondering whether it could be solved in a more elegant way...

[Richard Jones (r1chardj0n3s)]

Timur, in comments #5 and #6 I describe a couple of alternative methods to fix this but they would be significant amounts of work, even though they would be cleaner (I don't know about "elegant" though :)

[Timur Sufiev (tsufiev-x)]

I have to admit that trying option 1 (changing PK in the Django model) was a dead-end. Seems that monkey patching is best way to get job done. Though it's possible to redefine pk on a particular model via metaclasses (see attachment), somehow Django stores every app's model in a registry which I don't understand how to modify.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/222478

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/222480

[Richard Jones (r1chardj0n3s)]

Thanks to a great suggestion from Lin, who suggested looking at the user model problem differently and considering the PK attribute, I have created two patches: one for DOA and the other for Horizon. These fix the problem without needing a monkey-patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Richard Jones (<email address hidden>) on branch: master
Review: https://review.openstack.org/220382
Reason: A better solution has been found!

[Richard Jones (r1chardj0n3s)]

Proposed fix in DOA https://review.openstack.org/#/c/222478/

[Andrew Bruce (andybrucenet)]

Hello maintainers...I have about 8 minutes to spend on this particular bug and my fix I'm sure falls under "monkey patching" from above. Basically, the problem is that the session key is stored in the data schema as an "AutoField" with a default conversion to Integer in the to_python method. Which fails with session keys as they are Very Long Strings of Chars. I took a quick look at the proposed changes and my brain doesn't grok how setting the user_id in keystone auth changes things??

The solution I used...stupidly simple so I'm sure wrong:

def _get_user_session_key(request):
    # This value in the session is always serialized to a string, so we need
    # to convert it back to Python whenever we access it.
    #return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
    # ABr, 20150916: newsflash...this will never be an int. so the above call loses every time.
    if request.session[SESSION_KEY] is None:
        return None
    return str(request.session[SESSION_KEY])

Is there really any way that the session key could be anything but a string??

[Richard Jones (r1chardj0n3s)]

Hi Andrew,

Yep, that is pretty much the same as the approach used in my abandoned monkey-patch solution (https://review.openstack.org/220382). the current patch is a better solution, though it's being held up by gate issues.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/222478
Committed: https://git.openstack.org/cgit/openstack/django_openstack_auth/commit/?id=8c64de92f4148d85704b10ea1f7bc441db2ddfee
Submitter: Jenkins
Branch: master

commit 8c64de92f4148d85704b10ea1f7bc441db2ddfee
Author: Richard Jones <email address hidden>
Date: Fri Sep 11 16:10:06 2015 +1000

    Replace default User model PK

    The default Django User model PK is an int() AutoField
    and django-openstack-auth sets this to a hash string. Django
    then breaks trying to coerce that string to an int().

    This patch adds a new explicit PK to the d-o-a User
    model. It also adds the standard Django "models.py" so
    that the consumer application (Horizon) may use it.

    The consumer application must set:

       AUTH_USER_MODEL = 'openstack_auth.User'

    to use the new model in place of the default 'auth.User'.

    The approach in this patch was inspired by Lin Hua
    Cheng <email address hidden>.

    Partial-Bug: 1491117
    Change-Id: I549209eb0bb0ddf36d92ee9dc1a9bac799ce67e5

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/225188

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/222480
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=ee2771ab1a855342089abe5206fc6a5071a6d99e
Submitter: Jenkins
Branch: master

commit ee2771ab1a855342089abe5206fc6a5071a6d99e
Author: Richard Jones <email address hidden>
Date: Fri Sep 11 16:16:44 2015 +1000

    Use the User model from d-o-a

    This patch moves us to explicitly using the replacement User
    model from django-openstack-auth.

    Change-Id: I558b9e0af3dd4c2029f1376cb9da08ef0bcc142e
    Closes-Bug: 1491117
    Depends-On: I86030706cc5d188c1537094657e90565f2ad8c05

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matthias Runge (<email address hidden>) on branch: master
Review: https://review.openstack.org/225188