Ceilometer 5.0 can't pollster hardware snmp meter (cpu_util, memory.usage,....)

[Ke Vo Danh]

Hi all,

I have a problem with ceilometer 5.0.

I'm running ceilometer 5.0 with openstack liberty. Services and logs looking good (not error), but when i run command "ceilometer meter-list". I can't find some hardware smnp meter: cpu_util, memory.usage,.......

# Info
OS : centos 7.0
snmp 5.7.2-24

# Services running:
ceilometer-api
ceilometer-collector
ceilometer-polling --polling-namespaces central

# Test ceilometer

$ ceilometer meter-list

# I can find some meters

disk.ephemeral.size
disk.root.size
disk.size.total
disk.size.used
 image
 image.size
 instance
 instance:test
 instance:m1.tiny
 memory
 vcpus

# my logs
2015-12-28 09:43:48.889 29878 DEBUG ceilometer.hardware.pollsters.generic [-] Hardware snmp meter definition file: /home/openstack/deployment/iaas-liberty/ceilometer-5.0.0/ceilometer/hardware/pollsters/data/snmp.yaml setup_meters_config /home/openstack/deployment/iaas-liberty/ceilometer-5.0.0/ceilometer/hardware/pollsters/generic.py:232
2015-12-28 09:43:49.062 29878 INFO ceilometer.hardware.pollsters.generic [-] Meter Definitions: {'metric': [{'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.10.1.3.1', 'type': 'lambda x: float(str(x))', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.cpu.load.1min', 'unit': 'process'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.10.1.3.2', 'type': 'lambda x: float(str(x))', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.cpu.load.5min', 'unit': 'process'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.10.1.3.3', 'type': 'lambda x: float(str(x))', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.cpu.load.15min', 'unit': 'process'}, {'snmp_inspector': {'post_op': '_post_op_disk', 'oid': '1.3.6.1.4.1.2021.9.1.6', 'type': 'int', 'matching_type': 'type_prefix', 'metadata': {'device': {'oid': '1.3.6.1.4.1.2021.9.1.3', 'type': 'str'}, 'path': {'oid': '1.3.6.1.4.1.2021.9.1.2', 'type': 'str'}}}, 'type': 'gauge', 'name': 'hardware.disk.size.total', 'unit': 'KB'}, {'snmp_inspector': {'post_op': '_post_op_disk', 'oid': '1.3.6.1.4.1.2021.9.1.8', 'type': 'int', 'matching_type': 'type_prefix', 'metadata': {'device': {'oid': '1.3.6.1.4.1.2021.9.1.3', 'type': 'str'}, 'path': {'oid': '1.3.6.1.4.1.2021.9.1.2', 'type': 'str'}}}, 'type': 'gauge', 'name': 'hardware.disk.size.used', 'unit': 'KB'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.4.5.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.memory.total', 'unit': 'KB'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.4.6.0', 'type': 'int', 'matching_type': 'type_exact', 'post_op': '_post_op_memory_avail_to_used'}, 'type': 'gauge', 'name': 'hardware.memory.used', 'unit': 'KB'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.4.3.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.memory.swap.total', 'unit': 'KB'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.4.4.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.memory.swap.avail', 'unit': 'KB'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.4.14.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.memory.buffer', 'unit': 'KB'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.4.15.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.memory.cached', 'unit': 'KB'}, {'snmp_inspector': {'post_op': '_post_op_net', 'oid': '1.3.6.1.2.1.2.2.1.10', 'type': 'int', 'matching_type': 'type_prefix', 'metadata': {'mac': {'oid': '1.3.6.1.2.1.2.2.1.6', 'type': "lambda x: x.prettyPrint().replace('0x', '')"}, 'speed': {'oid': '1.3.6.1.2.1.2.2.1.5', 'type': 'lambda x: int(x) / 8'}, 'name': {'oid': '1.3.6.1.2.1.2.2.1.2', 'type': 'str'}}}, 'type': 'cumulative', 'name': 'hardware.network.incoming.bytes', 'unit': 'B'}, {'snmp_inspector': {'post_op': '_post_op_net', 'oid': '1.3.6.1.2.1.2.2.1.16', 'type': 'int', 'matching_type': 'type_prefix', 'metadata': {'mac': {'oid': '1.3.6.1.2.1.2.2.1.6', 'type': "lambda x: x.prettyPrint().replace('0x', '')"}, 'speed': {'oid': '1.3.6.1.2.1.2.2.1.5', 'type': 'lambda x: int(x) / 8'}, 'name': {'oid': '1.3.6.1.2.1.2.2.1.2', 'type': 'str'}}}, 'type': 'cumulative', 'name': 'hardware.network.outgoing.bytes', 'unit': 'B'}, {'snmp_inspector': {'post_op': '_post_op_net', 'oid': '1.3.6.1.2.1.2.2.1.20', 'type': 'int', 'matching_type': 'type_prefix', 'metadata': {'mac': {'oid': '1.3.6.1.2.1.2.2.1.6', 'type': "lambda x: x.prettyPrint().replace('0x', '')"}, 'speed': {'oid': '1.3.6.1.2.1.2.2.1.5', 'type': 'lambda x: int(x) / 8'}, 'name': {'oid': '1.3.6.1.2.1.2.2.1.2', 'type': 'str'}}}, 'type': 'cumulative', 'name': 'hardware.network.outgoing.errors', 'unit': 'packet'}, {'snmp_inspector': {'oid': '1.3.6.1.2.1.4.10.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'cumulative', 'name': 'hardware.network.ip.outgoing.datagrams', 'unit': 'datagrams'}, {'snmp_inspector': {'oid': '1.3.6.1.2.1.4.3.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'cumulative', 'name': 'hardware.network.ip.incoming.datagrams', 'unit': 'datagrams'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.11.11.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.system_stats.cpu.idle', 'unit': '%'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.11.57.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'cumulative', 'name': 'hardware.system_stats.io.outgoing.blocks', 'unit': 'blocks'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.11.58.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'cumulative', 'name': 'hardware.system_stats.io.incoming.blocks', 'unit': 'blocks'}]}

2015-12-28 09:43:49.966 29878 INFO ceilometer.pipeline [-] Pipeline config: {'sources': [{'interval': 600, 'meters': ['*'], 'name': 'meter_source', 'sinks': ['meter_sink']}, {'interval': 600, 'meters': ['cpu'], 'name': 'cpu_source', 'sinks': ['cpu_sink', 'cpu_delta_sink']}, {'interval': 600, 'meters': ['disk.read.bytes', 'disk.read.requests', 'disk.write.bytes', 'disk.write.requests', 'disk.device.read.bytes', 'disk.device.read.requests', 'disk.device.write.bytes', 'disk.device.write.requests'], 'name': 'disk_source', 'sinks': ['disk_sink']}, {'interval': 600, 'meters': ['network.incoming.bytes', 'network.incoming.packets', 'network.outgoing.bytes', 'network.outgoing.packets'], 'name': 'network_source', 'sinks': ['network_sink']}], 'sinks': [{'publishers': ['notifier://'], 'transformers': None, 'name': 'meter_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}}}], 'name': 'cpu_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'delta', 'parameters': {'target': {'name': 'cpu.delta'}, 'growth_only': True}}], 'name': 'cpu_delta_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': '(disk\\.device|disk)\\.(read|write)\\.(bytes|requests)', 'unit': '(B|request)'}}, 'target': {'type': 'gauge', 'map_to': {'name': '\\1.\\2.\\3.rate', 'unit': '\\1/s'}}}}], 'name': 'disk_sink'}, {'publishers': ['notifier://'], 'transformers': [{'name': 'rate_of_change', 'parameters': {'source': {'map_from': {'name': 'network\\.(incoming|outgoing)\\.(bytes|packets)', 'unit': '(B|packet)'}}, 'target': {'type': 'gauge', 'map_to': {'name': 'network.\\1.\\2.rate', 'unit': '\\1/s'}}}}], 'name': 'network_sink'}]}

2015-12-28 09:47:39.764 610 INFO ceilometer.agent.manager [-] Skip pollster network.services.vpn, no resources found this cycle
2015-12-28 09:47:39.764 610 INFO ceilometer.agent.manager [-] Skip pollster switch.port.receive.overrun_error, no resources found this cycle
2015-12-28 09:47:39.764 610 INFO ceilometer.agent.manager [-] Skip pollster switch.port.transmit.drops, no resources found this cycle
2015-12-28 09:47:40.112 610 INFO ceilometer.agent.manager [-] Skip pollster network.services.vpn.connections, no resources found this cycle
2015-12-28 09:47:40.112 610 INFO ceilometer.agent.manager [-] Skip pollster switch.port, no resources found this cycle
2015-12-28 09:47:40.415 610 INFO ceilometer.agent.manager [-] Polling pollster network.services.lb.vip in the context of meter_source

.......

 I Hope you guys can resolve my issue.

Thanks.

[Lianhao Lu (lianhao-lu)]

From the log, I saw you didn't specify any resources in the pipeline file for snmp metrics. The default hardware discovery plugin only works if your target machine is provisioned by tripleO/ironic. You have to modify the pipeline file to add resources if your target machine is not managed by tripleO. for example:

---
sources:
    - name: snmp_source
      interval: 60
      meters:
          - "hardware.*"
      resources:
          - snmp://192.168.4.101
          - snmp://192.168.4.102
      discovery:
      sinks:
          - meter_sink
sinks:
    - name: meter_sink
      transformers:
      publishers:
          - notifier://

[gordon chung (chungg)]

thanks llu, please let us know if this persists

[Ke Vo Danh (dacbao91)]

Thanks all, I resolved my problem. My problem is not from pipeline config.

That's from here:

ceilometer-5.0.0/ceilometer/hardware/inspector/snmp.py

I saw method find_matching_oids always return empty array.

def find_matching_oids(oid_cache, oid, match_type, find_one=True):
        matched = []
        if match_type == PREFIX:
            for key in oid_cache.keys():
                # Because "oid" param is string like 1.3.6.1.4.1.2021.4.5.0
                # key is instance like Mibxxxxxx (fomat of snmp :)))
                if key.startswith(oid):
                    matched.append(key)
                    if find_one:
                       break
        else:
            if oid in oid_cache:
                matched.append(oid)
        return matched

--> I try to edit the method :-- > _query_oids(self, host, oids, cache, is_bulk)

# save result into cache
        if is_bulk:
            for var_bind_table_row in data:
                for name, val in var_bind_table_row:
                    # using raw data
                    oid_cache[str(name)] = val
                    # oid_cache[name.prettyPrint()] = val
        else:
            for name, val in data:
                # oid_cache[name.prettyPrint()] = val
                oid_cache[str(name)] = val

and that worked.

Thanks and best regards.

[mkrish004c (murali-krishnaswamy)]

Thanks You Danh, it worked for me as well

[mkrish004c (murali-krishnaswamy)]

Fix is available in Mitaka version, need to back port to Liberty.
https://github.com/openstack/ceilometer/blob/master/ceilometer/hardware/inspector/snmp.py

[satheeshprabhakaran.v (vsp661992)]

Thanks Danh it worked for me too.