horizon "unable to retrieve" errors in HA

[Chris Ricker]

Note: this is only an issue in HA scenarios, not AIO

Note: this is on boxes that have https://review.openstack.org/94956 and https://review.openstack.org/94959 as horizon can't even be logged into without those two

after login almost every tab under the system panel displays errors of the form

Error: unable to retrieve $foo

$foo is "usage information", "limit information", "volume limit information", "hypervisor information", "hypervisor statistics", "host aggregates list", "availability zone list", "instance list", etc

logs show client errors, such as:

2014-05-23 15:01:58,879 1221 DEBUG openstack_dashboard.api.cinder cinderclient connection created using token "9e276223a8a04eb897c3d2dd2dc4c76e" and url "http://172.29.75.155:8776/v1/11c98883975449378c4b6808faac1f72"
2014-05-23 15:01:58,879 1221 DEBUG cinderclient.client
REQ: curl -i http://172.29.75.155:8776/v1/11c98883975449378c4b6808faac1f72/limits -X GET -H "X-Auth-Project-Id: 11c98883975449378c4b6808faac1f72" -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: 9e276223a8a04eb897c3d2dd2dc4c76e"

2014-05-23 15:01:59,052 1221 DEBUG cinderclient.client RESP: [500] CaseInsensitiveDict({'date': 'Fri, 23 May 2014 15:01:08 GMT', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-803e26ef-027a-428f-a19b-268e56b1bf19', 'x-openstack-request-id': 'req-803e26ef-027a-428f-a19b-268e56b1bf19'})
RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

2014-05-23 15:01:59,053 1221 WARNING horizon.exceptions Recoverable error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-803e26ef-027a-428f-a19b-268e56b1bf19)