Comment 4 for bug 1525877

I've found the most likely cause of 500 error on node-3 /var/log/apache/horizon_error.log (node-2 and node-1 corresponding files were empty indeed).

Mon Dec 14 11:55:28.290869 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] mod_wsgi (pid=24944): Exception occurred processing WSGI script '/usr/share/openstack-das
hboard/openstack_dashboard/wsgi/django.wsgi'.
[Mon Dec 14 11:55:28.290912 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] Traceback (most recent call last):
[Mon Dec 14 11:55:28.290931 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/django/core/handlers/wsgi.py", line 168, in __ca
ll__
[Mon Dec 14 11:55:28.291025 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] self.load_middleware()
[Mon Dec 14 11:55:28.291039 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 46, in load_
middleware
[Mon Dec 14 11:55:28.291108 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] mw_instance = mw_class()
[Mon Dec 14 11:55:28.291118 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/django/middleware/locale.py", line 23, in __init
__
[Mon Dec 14 11:55:28.291154 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] for url_pattern in get_resolver(None).url_patterns:
[Mon Dec 14 11:55:28.291164 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 372, in url_patterns
[Mon Dec 14 11:55:28.291308 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
[Mon Dec 14 11:55:28.291322 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 366, in urlconf_module
[Mon Dec 14 11:55:28.291336 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] self._urlconf_module = import_module(self.urlconf_name)
[Mon Dec 14 11:55:28.291342 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
[Mon Dec 14 11:55:28.291375 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] __import__(name)
[Mon Dec 14 11:55:28.291382 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/urls.py", line 35, in <module>
[Mon Dec 14 11:55:28.291414 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] url(r'^api/', include('openstack_dashboard.api.rest.urls')),
[Mon Dec 14 11:55:28.291422 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/django/conf/urls/__init__.py", line 28, in include
[Mon Dec 14 11:55:28.291453 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] urlconf_module = import_module(urlconf_module)
[Mon Dec 14 11:55:28.291460 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
[Mon Dec 14 11:55:28.291470 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] __import__(name)
[Mon Dec 14 11:55:28.291475 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/__init__.py", line 36, in <module>
[Mon Dec 14 11:55:28.291516 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] from openstack_dashboard.api import cinder
[Mon Dec 14 11:55:28.291523 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/cinder.py", line 30, in <module>
[Mon Dec 14 11:55:28.291653 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] from cinderclient.v2.contrib import list_extensions as cinder_list_extensions
[Mon Dec 14 11:55:28.291665 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/cinderclient/v2/__init__.py", line 17, in <module>
[Mon Dec 14 11:55:28.291696 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] from cinderclient.v2.client import Client # noqa
[Mon Dec 14 11:55:28.291704 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/cinderclient/v2/client.py", line 16, in <module>
[Mon Dec 14 11:55:28.291757 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] from cinderclient import client
[Mon Dec 14 11:55:28.291773 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 44, in <module>
[Mon Dec 14 11:55:28.291978 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] from eventlet import sleep
[Mon Dec 14 11:55:28.291994 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/eventlet/__init__.py", line 10, in <module>
[Mon Dec 14 11:55:28.292037 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] from eventlet import convenience
[Mon Dec 14 11:55:28.292052 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 3, in <module>
[Mon Dec 14 11:55:28.292101 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] from eventlet import greenio
[Mon Dec 14 11:55:28.292110 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/eventlet/greenio/__init__.py", line 3, in <module>
[Mon Dec 14 11:55:28.292133 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] from eventlet.greenio.base import * # noqa
[Mon Dec 14 11:55:28.292140 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 441, in <module>
[Mon Dec 14 11:55:28.292257 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] from OpenSSL import SSL
[Mon Dec 14 11:55:28.292273 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/OpenSSL/__init__.py", line 8, in <module>
[Mon Dec 14 11:55:28.292320 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] from OpenSSL import rand, crypto, SSL
[Mon Dec 14 11:55:28.292333 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/OpenSSL/rand.py", line 11, in <module>
[Mon Dec 14 11:55:28.292403 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] from OpenSSL._util import (
[Mon Dec 14 11:55:28.292412 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/OpenSSL/_util.py", line 7, in <module>
[Mon Dec 14 11:55:28.292454 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] binding = Binding()
[Mon Dec 14 11:55:28.292462 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 63, in __init__
[Mon Dec 14 11:55:28.292521 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] self._ensure_ffi_initialized()
[Mon Dec 14 11:55:28.292529 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 95, in _ensure_ffi_initialized
[Mon Dec 14 11:55:28.292546 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] cls._register_osrandom_engine()
[Mon Dec 14 11:55:28.292557 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 70, in _register_osrandom_engine
[Mon Dec 14 11:55:28.292571 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] raise RuntimeError("osrandom engine already registered")
[Mon Dec 14 11:55:28.292591 2015] [:error] [pid 24944:tid 140366006642432] [remote 192.168.0.7:3209] RuntimeError: osrandom engine already registered