compute agent gives up discovery if it receives a HTTP 503 from nova-api

[Frode Nordahl]

If the ceilometer-agent on the compute node is started before the Nova API is quite ready to receive requests and subsequently receives a HTTP 503 error, it gives up discovery and goes into a indefinite loop doing nothing.

The end user will have to identify this and manually restart the ceilometer-agent on the compute nodes to have meters updating.

ceilometer-agent should keep retrying discovery until it succeeds.

2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client [req-70ea1c70-356b-4518-97d8-edc1652264f5 admin - - - -] Unknown Error (HTTP 503) (Request-ID: req-298939b5-601c-4592-acf3-5b01e4876ec2)
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 52, in with_logging
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client return func(*args, **kwargs)
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 157, in instance_get_all_by_host
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client search_opts=search_opts))
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 749, in list
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client "servers")
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 242, in _list
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client resp, body = self.api.client.get(url)
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client return self.request(url, 'GET', **kwargs)
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 94, in request
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client raise exceptions.from_response(resp, body, url, method)
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client ClientException: Unknown Error (HTTP 503) (Request-ID: req-298939b5-601c-4592-acf3-5b01e4876ec2)
2016-11-23 08:01:44.659 1862 ERROR ceilometer.nova_client

[gordon chung (chungg)]

i don't know if we should retry until it succeeds but i understand the bug. maybe it's better to just let the service die after retrying for set time.

can you confirm what code branch you see this error on?

[gordon chung (chungg)]

don't use nova-api anymore for polling by default but people are welcome to fix it.