Detailed bug description:
If an instance was launched from the image with some custom field (f.e. "od_distro=" instead of "os_distro") ceilometer pooling can't get information about the instance and fully stopped on the compute host.
Expected results:
Ceilometer should skip this instance
Actual result:
Ceilometer skipped all resources
2016-10-20 12:37:05.331 32020 DEBUG novaclient-debug [req-80c09252-4204-4a7b-a0c3-41c7472cfad1 admin - - - -] REQ: curl -g -i -X GET http://172.16.33.3:8774/v2.1/flavors/1 -H "User-Agent: python-novaclient" -H
"Accept: application/json" -H "X-Auth-Token: {SHA1}c4fa1fdfaafd980dc6dc9d0ba17f9ae756848a2b" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
2016-10-20 12:37:05.443 32020 DEBUG novaclient-debug [req-80c09252-4204-4a7b-a0c3-41c7472cfad1 admin - - - -] RESP: [200] Content-Length: 356 X-Compute-Request-Id: req-4001eaf0-a756-4932-836c-ba63591a722c Vary:
X-OpenStack-Nova-API-Version Connection: close X-Openstack-Nova-Api-Version: 2.1 Date: Thu, 20 Oct 2016 12:37:05 GMT Content-Type: application/json
RESP BODY: {"flavor": {"name": "m1.tiny", "links": [{"href": "http://172.16.33.3:8774/v2.1/flavors/1", "rel": "self"}, {"href": "http://172.16.33.3:8774/flavors/1", "rel": "bookmark"}], "ram": 512, "OS-FLV-DISA
BLED:disabled": false, "vcpus": 1, "swap": "", "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 0, "disk": 1, "id": "1"}}
_http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-10-20 12:37:05.444 32020 DEBUG novaclient-debug [req-80c09252-4204-4a7b-a0c3-41c7472cfad1 admin - - - -] REQ: curl -g -i -X GET http://172.16.33.3:8774/v2.1/images/23ee6b59-4f35-474d-b07e-c4c46fe99853 -H "
User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}c4fa1fdfaafd980dc6dc9d0ba17f9ae756848a2b" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
2016-10-20 12:37:05.896 32020 DEBUG novaclient-debug [req-80c09252-4204-4a7b-a0c3-41c7472cfad1 admin - - - -] RESP: [500] Content-Length: 205 X-Compute-Request-Id: req-a524a7df-c3ad-4982-8c41-bc97fb2ea0cc Vary:
X-OpenStack-Nova-API-Version Connection: close X-Openstack-Nova-Api-Version: 2.1 Date: Thu, 20 Oct 2016 12:37:05 GMT Content-Type: application/json; charset=UTF-8
RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'glanceclient.exc.HTTPBadGateway'>", "code":
500}}
_http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client [req-80c09252-4204-4a7b-a0c3-41c7472cfad1 admin - - - -] Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nov
a API log if possible.
<class 'glanceclient.exc.HTTPBadGateway'> (HTTP 500) (Request-ID: req-a524a7df-c3ad-4982-8c41-bc97fb2ea0cc)
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client Traceback (most recent call last):
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 52, in with_logging
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client return func(*args, **kwargs)
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 157, in instance_get_all_by_host
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client search_opts=search_opts))
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 91, in _with_flavor_and_image
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client self._with_image(instance, image_cache)
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 129, in _with_image
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client image = self.nova_client.images.get(iid)
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/v2/images.py", line 55, in get
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client return self._get("/images/%s" % base.getid(image), "image")
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 339, in _get
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client resp, body = self.api.client.get(url)
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client return self.request(url, 'GET', **kwargs)
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 94, in request
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client raise exceptions.from_response(resp, body, url, method)
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client <class 'glanceclient.exc.HTTPBadGateway'> (HTTP 500) (Request-ID: req-a524a7df-c3ad-4982-8c41-bc97fb2ea0cc)
2016-10-20 12:37:05.897 32020 ERROR ceilometer.nova_client
2016-10-20 12:37:05.901 32020 INFO ceilometer.agent.manager [req-80c09252-4204-4a7b-a0c3-41c7472cfad1 admin - - - -] Skip pollster network.outgoing.bytes, no resources found this cycle
2016-10-20 12:37:05.901 32020 INFO ceilometer.agent.manager [req-80c09252-4204-4a7b-a0c3-41c7472cfad1 admin - - - -] Skip pollster disk.write.requests, no resources found this cycle
2016-10-20 12:37:05.902 32020 INFO ceilometer.agent.manager [req-80c09252-4204-4a7b-a0c3-41c7472cfad1 admin - - - -] Skip pollster disk.device.read.requests.rate, no resources found this cycle
Additional information:
The issue related to the following bug: https://bugs.launchpad.net/mos/+bug/1635241
Michael - please provide MOS version, steps to reproduce and diagnostic snapshot.