Openstack dashboard doesn't work after primary controller shutdown

[Artem Panchenko]

api: '1.0'
astute_sha: 702af3db6f5bca92525bc8322d7d5d7675ec857e
auth_required: true
build_id: 2014-11-12_21-56-33
build_number: '3'
feature_groups:
- mirantis
fuellib_sha: 7d711bb3d2572d55463cd38b9df3ad7a20a34262
fuelmain_sha: e5e534ade6f3765a87feee3d44d39df68ae28f80
nailgun_sha: bbc9dfe78a0c33040dcd16de9a40a3491788719c
ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
production: docker
release: 5.1.1

After killing of primary controller node Horizon became broken, it showed the following message after log in:

"Something went wrong! An unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator."

I enabled debug mode in dashboard settings and got the error traceback:

http://paste.openstack.org/show/132787/

Full page with debug info is attached.

[Vladimir Kuklin (vkuklin)]

The problem may be related to the fact that we django does not tolerate memcached node death. The best solution is to fix this in django itself introducing connection pooling for django memcached backend. But I am not sure we can do this in 5.1.1.

[Timur Sufiev (tsufiev-x)]

I had a strong suspicion that this bug is caused by django-openstack-auth being obsolete for 5.1.1. Maxim, could you please check it for both versions?

[Dmitry Borodaenko (angdraug)]

Unless there's a workaround for this bug, I think its priority should be Critical.

[Max Yatsenko (myatsenko)]

The bug was not reproduced with ISO that contained openstack-auth-1.1.7 packet:

To test if this bug was observed with openstack-auth packet version: 1.1.7
I took "fuel-5.1.1-20-2014-11-18_11-07-36.iso" and deployed the following
configurations:

Configuration1:
OpenStack Release: Icehouse on CentOS 6.5
Deployment Mode: Multi-node with HA
Compute: QEMU
Network: Nova-Network
Storage Backends: Cinder (Default), Glance (Default

3 Controllers
3 Compute

# rpm -qa | grep openstack-auth
python-django-openstack-auth-1.1.7-1.mira1.noarch

---------------------------------------------------

Configuration2:
OpenStack Release: Icehouse on Ubuntu 12.04.4
Deployment Mode: Multi-node with HA
Compute: QEMU
Network: Nova-Network
Storage Backends: Cinder (Default), Glance (Default

3 Controllers
3 Compute

# dpkg -l | grep openstack-auth
ii python-openstack-auth 1.1.7-ubuntu1 Django authentication backend for Openstack

---------------------------------------------------

In both configurations when I shutted down (killed) primary controller
haproxy switched to another one and I was able to log into dashboard,
the message "Something went wrong! An unexpected error has occurred.
Try refreshing the page. If that doesn't help, contact your local administrator."
was not observed.

[Andrey Sledzinskiy (asledzinskiy)]

Issues wasn't reproduced on
{

    "build_id": "2014-11-18_21-51-41",
    "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
    "build_number": "22",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "2fcab95dc43a248ba867065e96ab764ee73882d1",
    "production": "docker",
    "fuelmain_sha": "ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9",
    "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1.1",
    "release_versions": {
        "2014.1.3-5.1.1": {
            "VERSION": {
                "build_id": "2014-11-18_21-51-41",
                "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
                "build_number": "22",
                "api": "1.0",
                "nailgun_sha": "2fcab95dc43a248ba867065e96ab764ee73882d1",
                "production": "docker",
                "fuelmain_sha": "ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9",
                "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "5.1.1",
                "fuellib_sha": "b928ba300daedfc56b906b7c208641cb27698072"
            }
        }
    },
    "fuellib_sha": "b928ba300daedfc56b906b7c208641cb27698072"

}

Also tried it on real hardware - wasn't reproduced too on
{

    "build_id": "2014-11-18_11-07-36",
    "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
    "build_number": "20",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "2fcab95dc43a248ba867065e96ab764ee73882d1",
    "production": "docker",
    "fuelmain_sha": "ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9",
    "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1.1",
    "release_versions": {
        "2014.1.3-5.1.1": {
            "VERSION": {
                "build_id": "2014-11-18_11-07-36",
                "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
                "build_number": "20",
                "api": "1.0",
                "nailgun_sha": "2fcab95dc43a248ba867065e96ab764ee73882d1",
                "production": "docker",
                "fuelmain_sha": "ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9",
                "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "5.1.1",
                "fuellib_sha": "16a38252b467ca8c013e3955509b41f66a102f4b"
            }
        }
    },
    "fuellib_sha": "16a38252b467ca8c013e3955509b41f66a102f4b"

}

[Max Yatsenko (myatsenko)]

Bug was not reproduced for fuel-6.0 ( ISO: fuel-6.0-113-2014-11-18_11-56-14.iso):

ISO contained "openstack-auth-1.1.7" package and the same scheme (see above: Configuration1 and Configuration2) was deployed for CentOS and Ubuntu releases, after "killing" primary controller I was able to log into dashboard, any error/warning messages were not observed .