Hyper-V CPU pollster,could not get CPU time for instance: list index out of range

[Jun Xie]

environment: Windows Server 2012 Datacenter,with hyper-V

debugged, and found that in ceilometer\compute\virt\hyperv\utilsv2.py :
 element.associators(wmi_association_class=self._METRICS_ME) in _get_metrics() returns [ ].

http://paste.openstack.org/show/51563/

2013-11-11 05:45:10.569 1432 INFO ceilometer.compute.manager [-] Polling pollster cpu
2013-11-11 05:45:10.569 1432 INFO ceilometer.compute.pollsters.cpu [-] checking instance 144af1c4-1663-4070-88c8-db65ae13d96a
2013-11-11 05:45:10.944 1432 ERROR ceilometer.compute.pollsters.cpu [-] could not get CPU time for 144af1c4-1663-4070-88c8-db65ae13d96a: list index out of range
2013-11-11 05:45:10.944 1432 ERROR ceilometer.compute.pollsters.cpu [-] list index out of range
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu Traceback (most recent call last):
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu File "C:\Program Files (x86)\IBM\SmartCloud Entry\Hyper-V Agent\Python27\lib\site-packages\ceilometer\compute\pollsters\cpu.py", line 36, in get_samples
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu cpu_info = manager.inspector.inspect_cpus(instance_name)
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu File "C:\Program Files (x86)\IBM\SmartCloud Entry\Hyper-V Agent\Python27\lib\site-packages\ceilometer\compute\virt\hyperv\inspector.py", line 45, in inspect_cpus
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu cpu_count, uptime) = self._utils.get_cpu_metrics(instance_name)
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu File "C:\Program Files (x86)\IBM\SmartCloud Entry\Hyper-V Agent\Python27\lib\site-packages\ceilometer\compute\virt\hyperv\utilsv2.py", line 91, in get_cpu_metrics
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu cpu_metric_aggr = self._get_metrics(vm, cpu_metrics_def)[0]
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu IndexError: list index out of range
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu

[Jun Xie (junxiebj)]

looks like setting: enable_instance_metrics_collection to true makes the error msg away, looks like this is not a problem , although this error msg is not seens again, but the CPU time used always stays 0(Expected: should be Cumulative). so this issue can be linked to the following one :
https://bugs.launchpad.net/ceilometer/+bug/1250336

[Jun Xie (junxiebj)]

ceilometer sample-list -m cpu|grep a7d2b685-cf54-4726-b535-c42870cfc7be

| a7d2b685-cf54-4726-b535-c42870cfc7be | cpu | cumulative | 0.0 | ns | 2013-11-12T16:01:52 |
| a7d2b685-cf54-4726-b535-c42870cfc7be | cpu | cumulative | 0.0 | ns | 2013-11-12T16:11:51 |
| a7d2b685-cf54-4726-b535-c42870cfc7be | cpu | cumulative | 0.0 | ns | 2013-11-12T16:21:51 |

[Jun Xie (junxiebj)]

the value of cpu is always 0,: causing memory error

http://paste.openstack.org/show/51941/

2013-11-12 08:21:51.101 2548 INFO ceilometer.compute.manager [-] Polling pollster cpu
2013-11-12 08:21:51.101 2548 INFO ceilometer.compute.pollsters.cpu [-] checking instance a7d2b685-cf54-4726-b535-c42870cfc7be
2013-11-12 08:21:51.492 2548 INFO ceilometer.compute.pollsters.cpu [-] CPUTIME USAGE: {'OS-EXT-STS:task_state': None, 'addresses': {u'network1': [{u'OS-EXT-IPS-MAC:mac_addr': u'fa:16:3e:2e:4c:26', u'version': 4, u'addr': u'10.0.1.3', u'OS-EXT-IPS:type': u'fixed'}]}, 'links': [{u'href': u'http://10.1.1.151:8774/v2/0013b7b825174c869afa6a513493044b/servers/a7d2b685-cf54-4726-b535-c42870cfc7be', u'rel': u'self'}, {u'href': u'http://10.1.1.151:8774/0013b7b825174c869afa6a513493044b/servers/a7d2b685-cf54-4726-b535-c42870cfc7be', u'rel': u'bookmark'}], 'image': {u'id': u'fc79cd16-45f3-4f9c-9857-442385b7f955', u'links': [{u'href': u'http://10.1.1.151:8774/0013b7b825174c869afa6a513493044b/images/fc79cd16-45f3-4f9c-9857-442385b7f955', u'rel': u'bookmark'}], 'name': u'hyperv-image'}, 'ramdisk_id': None, 'manager': <novaclient.v1_1.servers.ServerManager object at 0x030DB1B0>, 'OS-EXT-STS:vm_state': u'active', 'OS-EXT-SRV-ATTR:instance_name': u'instance-00000004', 'OS-SRV-USG:launched_at': u'2013-11-12T15:56:26.000000', 'flavor': {'name': u'xj-flavor', u'links': [{u'href': u'http://10.1.1.151:8774/0013b7b825174c869afa6a513493044b/flavors/7', u'rel': u'bookmark'}], 'ram': 1024, 'ephemeral': 0, 'vcpus': 1, 'disk': 10, u'id': u'7'}, 'id': u'a7d2b685-cf54-4726-b535-c42870cfc7be', 'security_groups': [{u'name': u'default'}], 'user_id': u'49774643974642b5898facdea26c92c9', 'OS-DCF:diskConfig': u'MANUAL', 'accessIPv4': u'', 'accessIPv6': u'', 'progress': 0, 'OS-EXT-STS:power_state': 1, 'OS-EXT-AZ:availability_zone': u'nova', 'config_drive': u'', 'status': u'ACTIVE', 'updated': u'2013-11-12T08:56:03Z', 'hostId': u'bb9341a44b83789f578a5d5d87c260f5f7a593d50f0413c09a12a50c', 'OS-EXT-SRV-ATTR:host': u'WIN-RGVGHQAJUUB', 'OS-SRV-USG:terminated_at': None, 'kernel_id': None, 'key_name': None, 'OS-EXT-SRV-ATTR:hypervisor_hostname': u'WIN-RGVGHQAJUUB', 'name': u'hyperv8888', 'created': u'2013-11-12T08:55:53Z', 'tenant_id': u'0013b7b825174c869afa6a513493044b', 'os-extended-volumes:volumes_attached': [], '_info': {u'OS-EXT-STS:task_state': None, u'addresses': {u'network1': [{u'OS-EXT-IPS-MAC:mac_addr': u'fa:16:3e:2e:4c:26', u'version': 4, u'addr': u'10.0.1.3', u'OS-EXT-IPS:type': u'fixed'}]}, u'links': [{u'href': u'http://10.1.1.151:8774/v2/0013b7b825174c869afa6a513493044b/servers/a7d2b685-cf54-4726-b535-c42870cfc7be', u'rel': u'self'}, {u'href': u'http://10.1.1.151:8774/0013b7b825174c869afa6a513493044b/servers/a7d2b685-cf54-4726-b535-c42870cfc7be', u'rel': u'bookmark'}], u'image': {u'id': u'fc79cd16-45f3-4f9c-9857-442385b7f955', u'links': [{u'href': u'http://10.1.1.151:8774/0013b7b825174c869afa6a513493044b/images/fc79cd16-45f3-4f9c-9857-442385b7f955', u'rel': u'bookmark'}], 'name': u'hyperv-image'}, u'OS-EXT-STS:vm_state': u'active', u'OS-EXT-SRV-ATTR:instance_name': u'instance-00000004', u'OS-SRV-USG:launched_at': u'2013-11-12T15:56:26.000000', u'flavor': {'name': u'xj-...

[Alessandro Pilotti (alexpilotti)]

Correct, enable_instance_metrics_collection must be enabled in Nova. It must anyway not raise an exception if not set.

We'll check right away into the "0" cumulative value that you are experiencing.

[Jun Xie (junxiebj)]

 thanks,Alessandro. BTW, is there any doc available for configuring ceilometer + hyper-v? I am now wondering whether I have missed some other configurations(in order to let ceilometer correctly collect the meters of the hyper-v node), if any.

I now know about this one: enable_instance_metrics_collection = true in nova.conf , anything else which are specific for hyper environment?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/57017

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/57017
Committed: http://github.com/openstack/ceilometer/commit/f15e2cc9248c99e9b5c6b371ecce4f0ed49450cc
Submitter: Jenkins
Branch: master

commit f15e2cc9248c99e9b5c6b371ecce4f0ed49450cc
Author: Claudiu Belu <email address hidden>
Date: Mon Nov 18 05:26:17 2013 -0800

    Fixes Hyper-V Inspector cpu metrics bug

    Closes-Bug: 1249959

    Change-Id: I39513648ab25d46b2639b24a94e7344954ce3aeb

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/83024

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/83024
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=6637acdd8d9da4a2cc9e977a0837cc25483372d6
Submitter: Jenkins
Branch: stable/havana

commit 6637acdd8d9da4a2cc9e977a0837cc25483372d6
Author: Claudiu Belu <email address hidden>
Date: Mon Nov 18 05:26:17 2013 -0800

    Checks for presence of Hyper-V cpu metrics before accessing

    Adds a default value if there are no cpu metrics to be retrieved.

    Closes-Bug: 1249959
    (cherry picked from commit f15e2cc9248c99e9b5c6b371ecce4f0ed49450cc)

    Change-Id: I39513648ab25d46b2639b24a94e7344954ce3aeb