Centos HA: Internal Server Error in Horizon because of wrong caches LOCATION parameter in dashboard local_settings

[Aleksandr Didenko]

fuel-4.0-124 iso
 {"release": "4.0", "nailgun_sha": "8d80f823c38c2af6dc98173bcbe348d022960a3d", "ostf_sha": "cf48dac2a6e7ad284fc93c529f3d1e4668504028", "astute_sha": "ae026938f272f69afbe89c9900bf1c3df483557c", "fuellib_sha": "687a554eb9b6ae4dcc114f34e9690e601b40610c"}

Bare-metal deployment
KVM on CentOS, ha, 3 controllers, 1 compute

Horizon on http://172.16.1.2/dashboard/ gives "Internal Server Error"

Apache error log:
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] mod_wsgi (pid=5288): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'.
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] Traceback (most recent call last):
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] File "/usr/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 243, in __call__
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] signals.request_finished.send(sender=self.__class__)
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] File "/usr/lib/python2.6/site-packages/django/dispatch/dispatcher.py", line 172, in send
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] response = receiver(signal=self, sender=sender, **named)
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] File "/usr/lib/python2.6/site-packages/django/core/cache/backends/memcached.py", line 82, in close
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] self._cache.disconnect_all()
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] File "/usr/lib/python2.6/site-packages/django/core/cache/backends/memcached.py", line 31, in _cache
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] self._client = self._lib.Client(self._servers)
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] File "/usr/lib/python2.6/site-packages/memcache.py", line 153, in __init__
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] self.set_servers(servers)
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] File "/usr/lib/python2.6/site-packages/memcache.py", line 174, in set_servers
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] self.servers = [_Host(s, self.debuglog) for s in servers]
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] File "/usr/lib/python2.6/site-packages/memcache.py", line 813, in __init__
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] raise ValueError('Unable to parse connection string: "%s"' % host)
[Wed Dec 11 15:12:41 2013] [error] [client 192.168.0.4] ValueError: Unable to parse connection string: "192.168.0.3:11211192.168.0.4:11211192.168.0.5:11211"

It looks like LOCATION is incorrect in CACHES section of /etc/openstack-dashboard/local_settings config:

CACHES = {
    'default': {
        'BACKEND' : 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION' : "192.168.0.3:11211192.168.0.4:11211192.168.0.5:11211"
    },
}

Snapshot attached (problem environment nodes: node-11, node-12, node-13, node-14)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/61485

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/61485
Committed: http://github.com/stackforge/fuel-library/commit/8c5c9ea4322da4de3909c8d98022c3d54c6777fd
Submitter: Jenkins
Branch: master

commit 8c5c9ea4322da4de3909c8d98022c3d54c6777fd
Author: Vladimir Kuklin <email address hidden>
Date: Wed Dec 11 20:50:45 2013 +0400

    Fix dashboard CACHE location string

    Change-Id: I4b74f55f25f61af99b076fc6700e18e3c3bc4185
    Closes-bug: 1259969