1minstats job crashes on invalid ENTITY-MIB references from CISCO-PROCESS-MIB

[Peter Gervai]

Found this in the log today:

2015-05-26 12:50:36,344 [ERROR jobs.jobhandler] [1minstats ***] Plugin nav.ipdevpoll.plugins.statsystem.StatSystem(u'***') reported an unhandled failure
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 397, in errback
    self._startRunCallbacks(fail)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 464, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 551, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1101, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1043, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nav/ipdevpoll/plugins/statsystem.py", line 69, in handle
    cpu = yield self._collect_cpu()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1043, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nav/ipdevpoll/plugins/statsystem.py", line 118, in _collect_cpu
    load = yield self._get_cpu_loadavg(mib)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1045, in _inlineCallbacks
    result = g.send(result)
  File "/usr/lib/python2.7/dist-packages/nav/ipdevpoll/plugins/statsystem.py", line 139, in _get_cpu_loadavg
    interval)
  File "/usr/lib/python2.7/dist-packages/nav/metrics/templates.py", line 46, in metric_path_for_cpu_load
    cpu_name=escape_metric_name(cpu_name),
  File "/usr/lib/python2.7/dist-packages/nav/metrics/names.py", line 33, in escape_metric_name
    string = string.replace(char, "_")
exceptions.AttributeError: 'NoneType' object has no attribute 'replace'

2015-05-26 12:50:36,345 [ERROR jobs.jobhandler] [1minstats ***] Job '1minstats' for *** aborted: Job aborted due to plugin failure (cause=AttributeError("'NoneType' object has no attribute 'replace'",))

[Morten Brekkevold (mbrekkevold)]

Is this NAV 4.2? What type of device is this being collected from? It appears that ipdevpoll collects a nameless CPU, which means it cannot generate a meaningful Graphite metric name to store its load data in.

[Peter Gervai (grin)]

nav v4.2.6
the device is a cisco ASR-9000 series router
I am completely able to imagine a nameless CPU from cisco... :-( But if you give me a specific oid I could try to retrieve it and see what it results as I tend to remember CPUs all have names there.

[Morten Brekkevold (mbrekkevold)]

> nav v4.2.6
> the device is a cisco ASR-9000 series router
> I am completely able to imagine a nameless CPU from cisco... :-( But if you
> give me a specific oid I could try to retrieve it and see what it results as I
> tend to remember CPUs all have names there.

Well, there are two different Cisco proprietary MIBs that can be sourced
for CPU information. I'm not sure which one is supported by ASR-9000,
but hopefully it's the CISCO-PROCESS-MIB.

There is the CISCO-PROCESS-MIB::cpmCPUTotalTable, and the
OLD-CISCO-CPU-MIB::avgBusy5 and OLD-CISCO-CPU-MIB::avgBusy1 values.

You can fetch the full cpmCPUTotalTable using the snmptable command if
you have the MIB file available. Otherwise, do this to get
CISCO-PROCESS-MIB::cpmCPUTotalPhysicalIndex :

 snmpwalk -v2c -c <community> <ip> .1.3.6.1.4.1.9.9.109.1.1.1.1.2

Each line should represent a CPU in the device. Every value should be a
reference to a row in ENTITY-MIB::entPhysicalTable.

You should be able to get the name of each CPU by appending its
cpmCPUTotalPhysicalIndex value to ENTITY-MIB::entPhysicalName, like so:

 snmpget -v2c -c <community> <ip> ENTITY-MIB::entPhysicalName.<ID>

Or if you don't have the MIB installed:

 snmpget -v2c -c <community> <ip> .1.3.6.1.2.1.47.1.1.1.1.7.<ID>

(if you do have the ENTITY-MIB installed, you can use the snmptable
command to get a complete listing of entities and their attributes from
ENTITY-MIB::entPhysicalTable)

[Morten Brekkevold (mbrekkevold)]

I now see this intermittently in log reports from one of our own customers, which gave me a better opportunity to debug.

The device in question in this case is a Nexus 5548 with FEX modules. It normally reports a single CPU in CISCO-PROCESS-MIB, without an entPhysicalIndex reference (a value of 0). Intermittently, though, it does report an entPhysicalIndex which appears to be invalid.

The invalid reference is in turn transformed into a null string, which is passed around. It should instead behave just as if there was no entPhysicalIndex reference.

[Morten Brekkevold (mbrekkevold)]

fix here: https://nav.uninett.no/hg/stable/rev/36ceb3dc8065