[ocata] horizon first login error on a fresh install and after apache restart

[Dmitrii Shcherbakov]

memcached and haproxy config:

http://paste.ubuntu.com/25374788/

balance source # <-- seems right

memcached instances are listening locally which is ok given that horizon is a django app which will use this cache via a localhost connection - it doesn't need to have a socket bound to a different address. It would have been a problem if 'balance source' was not used on haproxy.

The issue is trivially reproducible after apache2 restart.

/var/log/apache2/error.log: http://paste.ubuntu.com/25374943/

"[Wed Aug 23 06:50:13.790605 2017] [wsgi:error] [pid 510168:tid 140037591574272] File "/usr/share/openstack-dashboard/openstack_dashboard/policy.py", line 24, in check
[Wed Aug 23 06:50:13.790608 2017] [wsgi:error] [pid 510168:tid 140037591574272] return policy_check(actions, request, target)
[Wed Aug 23 06:50:13.790611 2017] [wsgi:error] [pid 510168:tid 140037591574272] File "/usr/lib/python2.7/dist-packages/openstack_auth/policy.py", line 148, in check
[Wed Aug 23 06:50:13.790614 2017] [wsgi:error] [pid 510168:tid 140037591574272] enforcer = _get_enforcer()
[Wed Aug 23 06:50:13.790617 2017] [wsgi:error] [pid 510168:tid 140037591574272] File "/usr/lib/python2.7/dist-packages/openstack_auth/policy.py", line 57, in _get_enforcer
[Wed Aug 23 06:50:13.790620 2017] [wsgi:error] [pid 510168:tid 140037591574272] if os.path.isfile(enforcer.policy_path):
[Wed Aug 23 06:50:13.790623 2017] [wsgi:error] [pid 510168:tid 140037591574272] File "/usr/lib/python2.7/genericpath.py", line 37, in isfile
[Wed Aug 23 06:50:13.790627 2017] [wsgi:error] [pid 510168:tid 140037591574272] st = os.stat(path)
[Wed Aug 23 06:50:13.790630 2017] [wsgi:error] [pid 510168:tid 140037591574272] TypeError: coercing to Unicode: need string or buffer, NoneType found"

Related info:

https://docs.openstack.org/horizon/latest/admin/sessions.html#memcached

https://github.com/openstack/charm-openstack-dashboard/blob/stable/17.02/hooks/charmhelpers/contrib/openstack/templates/memcached.conf#L38-L41

https://github.com/openstack/charm-openstack-dashboard/blob/stable/17.02/templates/newton/local_settings.py#L132-L141

Configuration in local_settings.py is for django:
https://docs.djangoproject.com/en/1.11/topics/cache/#memcached

---

I have not found what causes this behavior yet.

[Ante Karamatić (ivoks)]

Because of this (when using keystone v3 api):

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
    },
}
...
SESSION_ENGINE = 'django.contrib.sessions.backends.cache'
...

...and memcache is not replicated between three horizons. All three IPs of dashboard units should be configured here.

Or we should use cookies for sessions:

SESSION_ENGINE = 'django.contrib.sessions.backends.signed_cookies'

There's also an option to relate horizon to mysql and then it should store session data in the database, but this template looks weird:

{% if database_host -%}
SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db'
DATABASES = {
    'default': {
        # Database configuration here
        'ENGINE': 'django.db.backends.mysql',
        'NAME': '{{ database }}',
        'USER': '{{ database_user }}',
        'PASSWORD': '{{ database_password }}',
        'HOST': '{{ database_host }}',
        'default-character-set': 'utf8'
    }
}
{% else -%}
{% if api_version == "3" -%}
# Warning: Please add DB relation for Keystone v3 + HA deployments
SESSION_ENGINE = 'django.contrib.sessions.backends.cache'
{% endif -%}

It implies that that in any case, SESSION_ENGINE will be set to cache, regardless of mysql relation.

[Liam Young (gnuoy)]

For keystone v3 the DB relation is compulsory (https://github.com/openstack/charm-openstack-dashboard/commit/94df23fbc5e89cc6251053467722a17a5bd4a420)

[Liam Young (gnuoy)]

Fwiw it was release noted here: https://wiki.ubuntu.com/OpenStack/OpenStackCharms/ReleaseNotes1607#openstack-dashboard_.2B-_Keystone_v3

[Dmitrii Shcherbakov (dmitriis)]

ubuntu@maas:~/bundles⟫ juju status openstack-dashboard
...
Relation Provides Consumes Type
...
identity-service keystone openstack-dashboard regular
...
shared-db mysql openstack-dashboard regular
...

http://paste.ubuntu.com/25378419/ (don't mind the 2 stopped services - I've done this to get http requests only on a single unit)

rendered local_settings.py:

http://paste.ubuntu.com/25378414/

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
    },
}
SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db'
DATABASES = {
    'default': {
        # Database configuration here
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'horizon',
        'USER': 'horizon',
        'PASSWORD': '<world's-secrets>',
        'HOST': '10.232.1.206',
        'default-character-set': 'utf8'
    }
}

[Ante Karamatić (ivoks)]

That should teach me not to do code review at 8pm. I was wrong, I misread that 'else' for SESSION_ENGINE.

It would be interesting to see if the problem persist after all thee memcaches are used, instead of localhost.

[Dmitrii Shcherbakov (dmitriis)]

Ante,

Independent memcache synchronization is not a problem here, in the paste above I killed 2/3 apache services so that requests go only to a single node https://paste.ubuntu.com/25378419/ - it has it's own memcache and the problem is still there.

Given that we have haproxy's source balancing, requests from the same client will go to a single node while it's up anyway.

https://git.launchpad.net/~usd-import-team/ubuntu/+source/haproxy/tree/doc/configuration.txt?h=applied/ubuntu/xenial-updates#n1981

"The source IP address is hashed and divided by the total weight of the running servers to designate which server will receive the request. This ensures that the same client IP address will always reach the same server as long as no server goes down or up."

[Ante Karamatić (ivoks)]

So, here's the thing...

Template for local_settings.py defines policy jsons:

{% if api_version == "3" -%}
POLICY_FILES = {
    'identity': 'keystonev3_policy.json',
    'compute': 'nova_policy.json',
    'volume': 'cinder_policy.json',
    'image': 'glance_policy.json',
    'orchestration': 'heat_policy.json',
    'network': 'neutron_policy.json',
    'telemetry': 'ceilometer_policy.json',
}
{% endif -%}

but ceilometer_policy.json is nowhere to be found. Once this is removed, issue is gone. I've put some prints to find the root cause, and here it is:

[Thu Aug 24 08:34:35.851241 2017] [wsgi:error] [pid 3732:tid 139814644647680] /usr/share/openstack-dashboard/openstack_dashboard/conf/nova_policy.json
[Thu Aug 24 08:34:35.872971 2017] [wsgi:error] [pid 3732:tid 139814644647680] /usr/share/openstack-dashboard/openstack_dashboard/conf/neutron_policy.json
[Thu Aug 24 08:34:35.882375 2017] [wsgi:error] [pid 3732:tid 139814644647680] /usr/share/openstack-dashboard/openstack_dashboard/conf/glance_policy.json
[Thu Aug 24 08:34:35.882664 2017] [wsgi:error] [pid 3732:tid 139814644647680] None
[Thu Aug 24 08:34:35.919878 2017] [wsgi:error] [pid 3732:tid 139814644647680] Internal Server Error: /identity/
[Thu Aug 24 08:34:35.919895 2017] [wsgi:error] [pid 3732:tid 139814644647680] Traceback (most recent call last):
[Thu Aug 24 08:34:35.919899 2017] [wsgi:error] [pid 3732:tid 139814644647680] File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 132, in get_response
[Thu Aug 24 08:34:35.919904 2017] [wsgi:error] [pid 3732:tid 139814644647680] response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Thu Aug 24 08:34:35.919907 2017] [wsgi:error] [pid 3732:tid 139814644647680] File "/usr/share/openstack-dashboard/horizon/decorators.py", line 36, in dec
[Thu Aug 24 08:34:35.919911 2017] [wsgi:error] [pid 3732:tid 139814644647680] return view_func(request, *args, **kwargs)
[Thu Aug 24 08:34:35.919914 2017] [wsgi:error] [pid 3732:tid 139814644647680] File "/usr/share/openstack-dashboard/horizon/decorators.py", line 52, in dec
[Thu Aug 24 08:34:35.919918 2017] [wsgi:error] [pid 3732:tid 139814644647680] return view_func(request, *args, **kwargs)
[Thu Aug 24 08:34:35.919921 2017] [wsgi:error] [pid 3732:tid 139814644647680] File "/usr/share/openstack-dashboard/horizon/decorators.py", line 36, in dec
[Thu Aug 24 08:34:35.919925 2017] [wsgi:error] [pid 3732:tid 139814644647680] return view_func(request, *args, **kwargs)
[Thu Aug 24 08:34:35.919928 2017] [wsgi:error] [pid 3732:tid 139814644647680] File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 71, in view
[Thu Aug 24 08:34:35.919931 2017] [wsgi:error] [pid 3732:tid 139814644647680] return self.dispatch(request, *args, **kwargs)
[Thu Aug 24 08:34:35.919935 2017] [wsgi:error] [pid 3732:tid 139814644647680] File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 89, in dispatch
[Thu Aug 24 08:34:35.919938 2017] [wsgi:error] [pid 3732:tid 139814644647680] return handler(request, *args, **kwargs)
[Thu Aug 24 08:34:35.919942 2017] [wsgi:error] [pid 3732:tid 139814644647680] File "/usr/share/openstack-...

[Dmitrii Shcherbakov (dmitriis)]

ceilometer_policy.json removal is a part of the major telemetry stack change.

Per Ocata ceilometer API is deprecated, event API is removed so the dashboard was removed before the Ocata release alongside with the policy file.

https://git.openstack.org/cgit/openstack/horizon/commit/?id=20ea82b9efe78516286c4b35c5dc644b296b4313

[Ante Karamatić (ivoks)]

https://review.openstack.org/#/c/497163/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/497163
Committed: https://git.openstack.org/cgit/openstack/charm-openstack-dashboard/commit/?id=d5072e4030d9f597cdb696a379f5ace9dadb8705
Submitter: Jenkins
Branch: master

commit d5072e4030d9f597cdb696a379f5ace9dadb8705
Author: Dmitrii Shcherbakov <email address hidden>
Date: Thu Aug 24 12:06:08 2017 +0300

    add local_settings template for Ocata

    ceilometer_policy.json file was removed as a part of
    20ea82b9efe78516286c4b35c5dc644b296b4313 horizon commit made before the
    Ocata release due to ceilometer API deprecation.

    Having this in a template for Ocata results in an error on first login
    via a given http server.

    Therefore, this change adds another template for Ocata+ which does not
    contain a reference to that file in POLICY_FILES.

    Change-Id: I3320622684c3185db45094563f277d4c7ce0694d
    Closes-Bug: 1712617

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.02
Review: https://review.openstack.org/497463

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/497463
Committed: https://git.openstack.org/cgit/openstack/charm-openstack-dashboard/commit/?id=37e2e8ad54b8304d91381c78e41ecad426e67c5e
Submitter: Jenkins
Branch: stable/17.02

commit 37e2e8ad54b8304d91381c78e41ecad426e67c5e
Author: Dmitrii Shcherbakov <email address hidden>
Date: Thu Aug 24 12:06:08 2017 +0300

    add local_settings template for Ocata

    ceilometer_policy.json file was removed as a part of
    20ea82b9efe78516286c4b35c5dc644b296b4313 horizon commit made before the
    Ocata release due to ceilometer API deprecation.

    Having this in a template for Ocata results in an error on first login
    via a given http server.

    Therefore, this change adds another template for Ocata+ which does not
    contain a reference to that file in POLICY_FILES.

    Change-Id: I3320622684c3185db45094563f277d4c7ce0694d
    Closes-Bug: 1712617
    (cherry picked from commit d5072e4030d9f597cdb696a379f5ace9dadb8705)