@Bogdan I am still getting an error loading IP/horizon/admin/metering/ Is this because of the hang in close-wait? ==> /var/log/docker-logs/remote/node-1.lax3.ubiquity.io/ceilometer-alarm-evaluator.log <== 2014-10-08T18:20:34.262950+01:00 warning: 2014-10-08 17:20:34.254 5260 WARNING ceilometerclient.common.http [-] Request returned failure status. 2014-10-08T18:20:34.264554+01:00 err: 2014-10-08 17:20:34.255 5260 ERROR ceilometer.alarm.service [-] alarm evaluation cycle failed 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service Traceback (most recent call last): 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 91, in _evaluate_assigned_alarms 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service alarms = self._assigned_alarms() 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 134, in _assigned_alarms 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service 'value': True}]) 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/v2/alarms.py", line 71, in list 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service return self._list(options.build_url(self._path(), q)) 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/common/base.py", line 58, in _list 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service resp, body = self.api.json_request('GET', url) 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/common/http.py", line 191, in json_request 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service resp, body_iter = self._http_request(url, method, **kwargs) 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/common/http.py", line 174, in _http_request 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service raise exc.from_response(resp, ''.join(body_iter)) 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service HTTPBadGateway: HTTPBadGateway (HTTP 502) 2014-10-08 17:20:34.255 5260 TRACE ceilometer.alarm.service ==> /var/log/docker-logs/remote/node-1.lax3.ubiquity.io/ceilometer-agent-central.log <== 2014-10-08T18:21:05.748833+01:00 warning: 2014-10-08 17:21:05.739 1940 AUDIT ceilometer.pipeline [-] Pipeline meter_sink: Publishing samples 2014-10-08T18:21:05.751635+01:00 warning: 2014-10-08 17:21:05.743 1940 AUDIT ceilometer.publisher.rpc [-] Publishing 16 samples on metering 2014-10-08T18:21:05.754505+01:00 warning: 2014-10-08 17:21:05.746 1940 AUDIT ceilometer.pipeline [-] Pipeline meter_sink: Published samples 2014-10-08T18:21:05.755592+01:00 info: 2014-10-08 17:21:05.746 1940 INFO ceilometer.agent [-] Polling pollster switch.port.receive.packets 2014-10-08T18:21:05.756700+01:00 info: 2014-10-08 17:21:05.746 1940 INFO ceilometer.agent [-] Polling pollster switch.flow.packets 2014-10-08T18:21:05.756821+01:00 info: 2014-10-08 17:21:05.747 1940 INFO ceilometer.agent [-] Polling pollster switch.port.transmit.bytes 2014-10-08T18:21:05.756821+01:00 info: 2014-10-08 17:21:05.747 1940 INFO ceilometer.agent [-] Polling pollster storage.objects 2014-10-08T18:21:05.759835+01:00 warning: 2014-10-08 17:21:05.751 1940 WARNING ceilometer.agent [-] Continue after error from storage.objects: Account HEAD failed: http://23.109.32.2:8080/v1/AUTH_6c5c801c896d4c41831fa2ec1faa002c 400 Bad Request 2014-10-08 17:21:05.751 1940 TRACE ceilometer.agent Traceback (most recent call last): 2014-10-08 17:21:05.751 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/agent.py", line 90, in poll_and_publish 2014-10-08 17:21:05.751 1940 TRACE ceilometer.agent resources=source_resources or agent_resources, 2014-10-08 17:21:05.751 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 92, in get_samples 2014-10-08 17:21:05.751 1940 TRACE ceilometer.agent for tenant, account in self._iter_accounts(manager.keystone, cache): 2014-10-08 17:21:05.751 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 61, in _iter_accounts 2014-10-08 17:21:05.751 1940 TRACE ceilometer.agent ksclient, cache)) 2014-10-08 17:21:05.751 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 77, in _get_account_info 2014-10-08 17:21:05.751 1940 TRACE ceilometer.agent ksclient.auth_token)) 2014-10-08 17:21:05.751 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 426, in head_account 2014-10-08 17:21:05.751 1940 TRACE ceilometer.agent http_response_content=body) 2014-10-08 17:21:05.751 1940 TRACE ceilometer.agent ClientException: Account HEAD failed: http://23.109.32.2:8080/v1/AUTH_6c5c801c896d4c41831fa2ec1faa002c 400 Bad Request 2014-10-08 17:21:05.751 1940 TRACE ceilometer.agent 2014-10-08T18:21:05.760875+01:00 info: 2014-10-08 17:21:05.752 1940 INFO ceilometer.agent [-] Polling pollster image.size 2014-10-08T18:21:05.764420+01:00 warning: 2014-10-08 17:21:05.756 1940 AUDIT ceilometer.pipeline [-] Pipeline meter_sink: Publishing samples 2014-10-08T18:21:05.766136+01:00 warning: 2014-10-08 17:21:05.757 1940 AUDIT ceilometer.publisher.rpc [-] Publishing 16 samples on metering 2014-10-08T18:21:05.768419+01:00 warning: 2014-10-08 17:21:05.760 1940 AUDIT ceilometer.pipeline [-] Pipeline meter_sink: Published samples 2014-10-08T18:21:05.769491+01:00 info: 2014-10-08 17:21:05.760 1940 INFO ceilometer.agent [-] Polling pollster switch.flow.duration.nanoseconds 2014-10-08T18:21:05.770536+01:00 info: 2014-10-08 17:21:05.760 1940 INFO ceilometer.agent [-] Polling pollster hardware.cpu.load.5min 2014-10-08T18:21:05.770624+01:00 info: 2014-10-08 17:21:05.761 1940 INFO ceilometer.agent [-] Polling pollster switch.port.receive.frame_error 2014-10-08T18:21:05.770624+01:00 info: 2014-10-08 17:21:05.761 1940 INFO ceilometer.agent [-] Polling pollster hardware.cpu.load.1min 2014-10-08T18:21:05.771650+01:00 info: 2014-10-08 17:21:05.761 1940 INFO ceilometer.agent [-] Polling pollster switch.table.active.entries 2014-10-08T18:21:05.771738+01:00 info: 2014-10-08 17:21:05.762 1940 INFO ceilometer.agent [-] Polling pollster storage.objects.size 2014-10-08T18:21:05.772748+01:00 warning: 2014-10-08 17:21:05.764 1940 WARNING ceilometer.agent [-] Continue after error from storage.objects.size: Account HEAD failed: http://23.109.32.2:8080/v1/AUTH_6c5c801c896d4c41831fa2ec1faa002c 400 Bad Request 2014-10-08 17:21:05.764 1940 TRACE ceilometer.agent Traceback (most recent call last): 2014-10-08 17:21:05.764 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/agent.py", line 90, in poll_and_publish 2014-10-08 17:21:05.764 1940 TRACE ceilometer.agent resources=source_resources or agent_resources, 2014-10-08 17:21:05.764 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 111, in get_samples 2014-10-08 17:21:05.764 1940 TRACE ceilometer.agent for tenant, account in self._iter_accounts(manager.keystone, cache): 2014-10-08 17:21:05.764 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 61, in _iter_accounts 2014-10-08 17:21:05.764 1940 TRACE ceilometer.agent ksclient, cache)) 2014-10-08 17:21:05.764 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 77, in _get_account_info 2014-10-08 17:21:05.764 1940 TRACE ceilometer.agent ksclient.auth_token)) 2014-10-08 17:21:05.764 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 426, in head_account 2014-10-08 17:21:05.764 1940 TRACE ceilometer.agent http_response_content=body) 2014-10-08 17:21:05.764 1940 TRACE ceilometer.agent ClientException: Account HEAD failed: http://23.109.32.2:8080/v1/AUTH_6c5c801c896d4c41831fa2ec1faa002c 400 Bad Request 2014-10-08 17:21:05.764 1940 TRACE ceilometer.agent 2014-10-08T18:21:05.773813+01:00 info: 2014-10-08 17:21:05.765 1940 INFO ceilometer.agent [-] Polling pollster hardware.cpu.load.15min 2014-10-08T18:21:05.773898+01:00 info: 2014-10-08 17:21:05.765 1940 INFO ceilometer.agent [-] Polling pollster switch.table 2014-10-08T18:21:05.774831+01:00 info: 2014-10-08 17:21:05.765 1940 INFO ceilometer.agent [-] Polling pollster switch.port.receive.bytes 2014-10-08T18:21:05.774884+01:00 info: 2014-10-08 17:21:05.766 1940 INFO ceilometer.agent [-] Polling pollster switch.port.transmit.drops 2014-10-08T18:21:05.774884+01:00 info: 2014-10-08 17:21:05.766 1940 INFO ceilometer.agent [-] Polling pollster switch.port.receive.drops 2014-10-08T18:21:05.775914+01:00 info: 2014-10-08 17:21:05.766 1940 INFO ceilometer.agent [-] Polling pollster switch.flow.duration.seconds 2014-10-08T18:21:05.775958+01:00 info: 2014-10-08 17:21:05.767 1940 INFO ceilometer.agent [-] Polling pollster storage.containers.objects 2014-10-08T18:21:05.778055+01:00 warning: 2014-10-08 17:21:05.769 1940 WARNING ceilometer.agent [-] Continue after error from storage.containers.objects: Account GET failed: http://23.109.32.2:8080/v1/AUTH_6c5c801c896d4c41831fa2ec1faa002c?format=json 400 Bad Request [first 60 chars of response] InvalidBu 2014-10-08 17:21:05.769 1940 TRACE ceilometer.agent Traceback (most recent call last): 2014-10-08 17:21:05.769 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/agent.py", line 90, in poll_and_publish 2014-10-08 17:21:05.769 1940 TRACE ceilometer.agent resources=source_resources or agent_resources, 2014-10-08 17:21:05.769 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 151, in get_samples 2014-10-08 17:21:05.769 1940 TRACE ceilometer.agent for project, account in self._iter_accounts(manager.keystone, cache): 2014-10-08 17:21:05.769 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 61, in _iter_accounts 2014-10-08 17:21:05.769 1940 TRACE ceilometer.agent ksclient, cache)) 2014-10-08 17:21:05.769 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 77, in _get_account_info 2014-10-08 17:21:05.769 1940 TRACE ceilometer.agent ksclient.auth_token)) 2014-10-08 17:21:05.769 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 392, in get_account 2014-10-08 17:21:05.769 1940 TRACE ceilometer.agent http_response_content=body) 2014-10-08 17:21:05.769 1940 TRACE ceilometer.agent ClientException: Account GET failed: http://23.109.32.2:8080/v1/AUTH_6c5c801c896d4c41831fa2ec1faa002c?format=json 400 Bad Request [first 60 chars of response] InvalidBu 2014-10-08 17:21:05.769 1940 TRACE ceilometer.agent 2014-10-08T18:21:05.779179+01:00 info: 2014-10-08 17:21:05.770 1940 INFO ceilometer.agent [-] Polling pollster switch.port.receive.errors 2014-10-08T18:21:05.779266+01:00 info: 2014-10-08 17:21:05.770 1940 INFO ceilometer.agent [-] Polling pollster hardware.disk.size.used 2014-10-08T18:21:05.780246+01:00 info: 2014-10-08 17:21:05.770 1940 INFO ceilometer.agent [-] Polling pollster switch.port 2014-10-08T18:21:05.780301+01:00 info: 2014-10-08 17:21:05.770 1940 INFO ceilometer.agent [-] Polling pollster storage.containers.objects.size 2014-10-08T18:21:05.782283+01:00 warning: 2014-10-08 17:21:05.773 1940 WARNING ceilometer.agent [-] Continue after error from storage.containers.objects.size: Account GET failed: http://23.109.32.2:8080/v1/AUTH_6c5c801c896d4c41831fa2ec1faa002c?format=json 400 Bad Request [first 60 chars of response] InvalidBu 2014-10-08 17:21:05.773 1940 TRACE ceilometer.agent Traceback (most recent call last): 2014-10-08 17:21:05.773 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/agent.py", line 90, in poll_and_publish 2014-10-08 17:21:05.773 1940 TRACE ceilometer.agent resources=source_resources or agent_resources, 2014-10-08 17:21:05.773 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 174, in get_samples 2014-10-08 17:21:05.773 1940 TRACE ceilometer.agent for project, account in self._iter_accounts(manager.keystone, cache): 2014-10-08 17:21:05.773 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 61, in _iter_accounts 2014-10-08 17:21:05.773 1940 TRACE ceilometer.agent ksclient, cache)) 2014-10-08 17:21:05.773 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift.py", line 77, in _get_account_info 2014-10-08 17:21:05.773 1940 TRACE ceilometer.agent ksclient.auth_token)) 2014-10-08 17:21:05.773 1940 TRACE ceilometer.agent File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 392, in get_account 2014-10-08 17:21:05.773 1940 TRACE ceilometer.agent http_response_content=body) 2014-10-08 17:21:05.773 1940 TRACE ceilometer.agent ClientException: Account GET failed: http://23.109.32.2:8080/v1/AUTH_6c5c801c896d4c41831fa2ec1faa002c?format=json 400 Bad Request [first 60 chars of response] InvalidBu 2014-10-08 17:21:05.773 1940 TRACE ceilometer.agent 2014-10-08T18:21:05.783391+01:00 info: 2014-10-08 17:21:05.774 1940 INFO ceilometer.agent [-] Polling pollster switch.table.lookup.packets 2014-10-08T18:21:05.783437+01:00 info: 2014-10-08 17:21:05.774 1940 INFO ceilometer.agent [-] Polling pollster hardware.network.outgoing.bytes 2014-10-08T18:21:05.784496+01:00 info: 2014-10-08 17:21:05.775 1940 INFO ceilometer.agent [-] Polling pollster switch.port.transmit.packets 2014-10-08T18:21:05.784585+01:00 info: 2014-10-08 17:21:05.775 1940 INFO ceilometer.agent [-] Polling pollster switch.port.collision.count 2014-10-08T18:21:05.784585+01:00 info: 2014-10-08 17:21:05.775 1940 INFO ceilometer.agent [-] Polling pollster power