R3.0: build 32: contrail-alarm-gen toggling from active and initializing

[Pulkit Tandon]

BUG Template : Ubuntu

OS version: 3.13.0-40-generic #69-Ubuntu
Contrail Version: 3.0.2.0-32

Setup details:
Multi node setup.
Issue observed on nodeg31

Configurations and Description:
Ran a test script which just create VMs.
Cannot comment on exact step where problem occurred.

When i check contrail-status, I observed that contrail-alarm-gen status is toggling continuously.
Is says following:
contrail-alarm-gen initializing (Redis-UVE:77.77.2.11:6379[None] connection down)

Restarted the supervisor-analytics but the problem still exist.

Log file dump:
05/09/2016 03:01:08 PM [contrail-alarm-gen]: redis-uve failed 'NoneType' object has no attribute 'items' for : (u'77.77.2.11', 6379, 15937) tb Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/opserver/uveserver.py", line 380, in get_uve
    rsp = pa.aggregate(key, flat, base_url)
  File "/usr/lib/python2.7/dist-packages/opserver/uveserver.py", line 763, in aggregate
    default_res[0][0])
  File "/usr/lib/python2.7/dist-packages/opserver/opserver_util.py", line 100, in uve_attr_flatten
    for k, v in inp[sname].items():
AttributeError: 'NoneType' object has no attribute 'items'

[Anish Mehta (amehta00)]

IS this readily reproducible?
I'd like to look at a system in this state.
I'll also commit some changes to get more information in logs.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20168
Submitter: Anish Mehta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20169
Submitter: Anish Mehta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20168
Submitter: Anish Mehta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20169
Submitter: Anish Mehta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20169
Committed: http://github.org/Juniper/contrail-controller/commit/adb73d5c8c0a4808eb00905b6687a9c5673311dc
Submitter: Zuul
Branch: R3.0

commit adb73d5c8c0a4808eb00905b6687a9c5673311dc
Author: Anish Mehta <email address hidden>
Date: Thu May 12 11:33:17 2016 -0700

We need more descriptive error messages when get_uve has an Exception.
Also optimizing lua script that clears generators - the per-UVE-key "ZREM" call is removed.
Partial-Bug:1579684

Change-Id: Ibc36c0901291799b87d2c387c9496b8447394d24

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20168
Committed: http://github.org/Juniper/contrail-controller/commit/9c9fe81f6478f17f4a384c2b9ddc63fc781bafaa
Submitter: Zuul
Branch: master

commit 9c9fe81f6478f17f4a384c2b9ddc63fc781bafaa
Author: Anish Mehta <email address hidden>
Date: Thu May 12 11:33:17 2016 -0700

We need more descriptive error messages when get_uve has an Exception.
Also optimizing lua script that clears generators - the per-UVE-key "ZREM" call is removed.
Partial-Bug:1579684

Change-Id: Ibc36c0901291799b87d2c387c9496b8447394d24

[Anish Mehta (amehta00)]

We have added instrumentation to get more info next time this happens.
Please reopen if this is seen again.

[Anish Mehta (amehta00)]

We have added instrumentation to get more info next time this happens.
Please reopen if this is seen again.

[Pulkit Tandon (pulkitt)]

Did not got reproduced after the occurrence when it was reported.