[4.1.0.0-3] node-status alarm not getting generated when supervisor-vrouter is inactive

[Ankit Jain]

Issue: The script stops supervisor-vrouter and then expects node-status alarm to get generated.

root@nodei1:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: inactive
unix:///var/run/supervisord_vrouter.sockno

The node-status alarm was not generated for the same and the script failed.

http://nodec7:8080/proxy?proxyURL=http://192.168.192.6:8081
{ }

Pasting the script logs below:

root@d16dd2729e40:/contrail-test# python -m testtools.run serial_scripts.analytics.test_analytics_serial.AnalyticsTestSanity.test_vrouter_process_status_alarms
Tests running...
2017-11-22 17:30:40,730 - INFO - Project ctest-AnalyticsTestSanity-49271850 not found, creating it
2017-11-22 17:30:41,233 - INFO - Created Project:ctest-AnalyticsTestSanity-49271850, ID : 5b09dce7-4535-4c9b-8178-ae4643c656b9
2017-11-22 17:30:43,374 - INFO - Verify alarms generated after stopping the service supervisor-vrouter:
2017-11-22 17:30:43,409 - INFO - stop contrail-vrouter-agent.service on nodei1
2017-11-22 17:30:44,810 - INFO - stop contrail-vrouter-nodemgr.service on nodei1
2017-11-22 17:30:46,130 - INFO - Process supervisor-vrouter stopped
2017-11-22 17:31:23,874 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:23,874 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:24,916 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:24,916 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:25,966 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:25,966 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:26,997 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:26,997 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:28,037 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:28,037 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:29,071 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:29,072 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:30,111 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:30,111 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:31,152 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:31,152 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:32,202 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:32,203 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:33,244 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:33,244 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:34,291 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:34,291 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:35,342 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:35,342 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:36,385 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:36,385 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:37,430 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:37,431 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:38,472 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:38,472 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:39,515 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:39,515 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:40,554 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:40,554 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:41,587 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:41,587 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:42,619 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:42,620 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:43,648 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:43,648 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:44,689 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:44,689 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:45,717 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:45,717 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:46,758 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:46,758 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:47,801 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:47,801 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:48,848 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:31:48,848 - WARNING - Alarm type default-global-system-config:system-defined-node-status alarms not generated yet ..wait .checking again
2017-11-22 17:31:49,849 - ERROR - Alarm type node-status not generated for role vrouter
{u'vrouter': [{u'name': u'nodei3',
               u'value': {u'UVEAlarms': {u'alarms': [{u'ack': False,
                                                      u'alarm_rules': {u'or_list': [{u'and_list': [{u'condition': {u'operand1': u'VrouterStatsAgent.in_bps_ewm.*.sigma',
                                                                                                                   u'operand2': {u'json_value': u'2'},
                                                                                                                   u'operation': u'>=',
                                                                                                                   u'variables': [u'VrouterStatsAgent.in_bps_ewm.__key']},
                                                                                                    u'match': [{u'json_operand1_value': u'2.33906',
                                                                                                                u'json_variables': {u'VrouterStatsAgent.in_bps_ewm.__key': u'"vlan128"'}}]}]}]},
                                                      u'description': u'Physical Bandwidth usage anomaly.',
                                                      u'severity': 2,
                                                      u'timestamp': 1511352090854899,
                                                      u'token': u'eyJ0aW1lc3RhbXAiOiAxNTExMzUyMDkwODU0ODk5LCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiMTkyLjE2OC4xOTIuNSJ9',
                                                      u'type': u'default-global-system-config:system-defined-phyif-bandwidth'}]}}},
              {u'name': u'nodei2',
               u'value': {u'UVEAlarms': {u'alarms': [{u'ack': False,
                                                      u'alarm_rules': {u'or_list': [{u'and_list': [{u'condition': {u'operand1': u'VrouterStatsAgent.in_bps_ewm.*.sigma',
                                                                                                                   u'operand2': {u'json_value': u'2'},
                                                                                                                   u'operation': u'>=',
                                                                                                                   u'variables': [u'VrouterStatsAgent.in_bps_ewm.__key']},
                                                                                                    u'match': [{u'json_operand1_value': u'2.57262',
                                                                                                                u'json_variables': {u'VrouterStatsAgent.in_bps_ewm.__key': u'"vlan128"'}}]}]}]},
                                                      u'description': u'Physical Bandwidth usage anomaly.',
                                                      u'severity': 2,
                                                      u'timestamp': 1511352082279912,
                                                      u'token': u'eyJ0aW1lc3RhbXAiOiAxNTExMzUyMDgyMjc5OTEyLCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiMTkyLjE2OC4xOTIuNyJ9',
                                                      u'type': u'default-global-system-config:system-defined-phyif-bandwidth'}]}}}]}
2017-11-22 17:31:49,854 - INFO - Alarms generated in the system are
 : None

2017-11-22 17:31:49,854 - INFO - start contrail-vrouter-agent.service on nodei1
2017-11-22 17:31:55,276 - INFO - start contrail-vrouter-nodemgr.service on nodei1
2017-11-22 17:32:07,502 - INFO - Alarm type node-status cleared for role vrouter
2017-11-22 17:32:07,502 - INFO - Verify alarms generated after stopping the service contrail-vrouter-agent:
2017-11-22 17:32:07,502 - INFO - stop contrail-vrouter-agent.service on nodei1
2017-11-22 17:32:08,872 - INFO - Process contrail-vrouter-agent stopped
2017-11-22 17:32:08,872 - INFO - Soaking enabled..waiting 15 secs for soak timer to expire
2017-11-22 17:32:23,926 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:23,926 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:24,965 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:24,965 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:25,999 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:25,999 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:27,037 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:27,037 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:28,089 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:28,089 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:29,119 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:29,120 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:30,160 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:30,160 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:31,205 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:31,205 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:32,248 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:32,249 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:33,295 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:33,295 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:34,340 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:34,340 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:35,381 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:35,381 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:36,422 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:36,422 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:37,463 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:37,463 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:38,503 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:38,504 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:39,544 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:39,544 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:40,586 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:40,587 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:41,624 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:41,624 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:42,662 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:42,663 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:43,691 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:43,692 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:44,733 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:44,733 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:45,770 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:45,771 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:46,815 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:46,815 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:47,858 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:47,858 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again

2017-11-22 17:32:48,893 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:48,893 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:49,936 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:49,936 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:50,980 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:50,980 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:52,021 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:52,022 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:53,068 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:53,068 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:54,111 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:54,111 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:55,159 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:55,159 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:56,198 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:56,199 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:57,241 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:57,241 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:58,284 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:58,284 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:32:59,327 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:32:59,327 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:00,369 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:00,370 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:01,411 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:01,411 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:02,453 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:02,453 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:03,500 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:03,501 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:04,544 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:04,545 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:05,587 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:05,587 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:06,626 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:06,627 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:07,667 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:07,667 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:08,712 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:08,712 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:09,753 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:09,753 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:10,800 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:10,800 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:11,841 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:11,842 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:12,871 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:12,871 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:13,913 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:13,913 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:14,951 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:14,951 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:15,988 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:15,989 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:17,027 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:17,027 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:18,068 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:18,068 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:19,109 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:19,109 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:20,147 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:20,147 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:21,174 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:21,175 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:22,221 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:22,221 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:23,264 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:23,264 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:24,300 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:24,300 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:25,338 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:25,338 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:26,375 - INFO - vrouter alarms generated for nodei1
2017-11-22 17:33:26,375 - WARNING - Alarm type default-global-system-config:system-defined-process-status alarms not generated yet ..wait .checking again
2017-11-22 17:33:27,376 - ERROR - Alarm type process-status not generated for role vrouter
{u'vrouter': [{u'name': u'nodei3',
               u'value': {u'UVEAlarms': {u'alarms': [{u'ack': False,
                                                      u'alarm_rules': {u'or_list': [{u'and_list': [{u'condition': {u'operand1': u'VrouterStatsAgent.in_bps_ewm.*.sigma',
                                                                                                                   u'operand2': {u'json_value': u'2'},
                                                                                                                   u'operation': u'>=',
                                                                                                                   u'variables': [u'VrouterStatsAgent.in_bps_ewm.__key']},
                                                                                                    u'match': [{u'json_operand1_value': u'2.33906',
                                                                                                                u'json_variables': {u'VrouterStatsAgent.in_bps_ewm.__key': u'"vlan128"'}}]}]}]},
                                                      u'description': u'Physical Bandwidth usage anomaly.',
                                                      u'severity': 2,
                                                      u'timestamp': 1511352090854899,
                                                      u'token': u'eyJ0aW1lc3RhbXAiOiAxNTExMzUyMDkwODU0ODk5LCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiMTkyLjE2OC4xOTIuNSJ9',
                                                      u'type': u'default-global-system-config:system-defined-phyif-bandwidth'}]}}}]}
2017-11-22 17:33:27,379 - INFO - Alarms generated in the system are
 : None

2017-11-22 17:33:27,379 - INFO - start contrail-vrouter-agent.service on nodei1
2017-11-22 17:33:42,802 - INFO - Soaking enabled..waiting 15 secs for soak timer to expire
2017-11-22 17:33:57,851 - INFO - All alarms cleared
2017-11-22 17:33:58,498 - INFO - Deleted project: ctest-AnalyticsTestSanity-49271850, ID : 5b09dce7-4535-4c9b-8178-ae4643c656b9
======================================================================
FAIL: serial_scripts.analytics.test_analytics_serial.AnalyticsTestSanity.test_vrouter_process_status_alarms[cb_sanity,sanity,vcenter,vcenter_compute]
----------------------------------------------------------------------
Empty attachments:
  pythonlogging:''

Traceback (most recent call last):
  File "serial_scripts/analytics/test_analytics_serial.py", line 237, in test_vrouter_process_status_alarms
    assert self.analytics_obj.verify_vrouter_alarms()
AssertionError

[Santosh Gupta (sangupta)]

Stopping supervisor-vrouter on nodei2 and nodei3 causes system-defined-node-status alarm.
But same doesn’t show up for nodei1. This was reproducible on the setup.
This is not a caching issue since the alarm doesn’t show up even after multiple stop/start of supervisor-vrouter on nodei1.
I restarted alarmgen on this node to set higher logging level but issue is not seen now.

If seen again, please check the following and leave setup in that state.
1. Alarmgen alarms config is present for the alarm system-defined-node-status.
http://nodec7.englab.juniper.net:5995/Snh_AlarmConfigRequest?name=

2. alarmgen Kafka partition status
nodei1:Compute:contrail-vrouter-agent:0 >>> ObjectVRouter:nodei1 >>> NodeStatus
http://nodec7.englab.juniper.net:5995/Snh_PartitionStatusReq?partition=-1

Once alarmgen is in this state, the alarm would not be raised. If this doesn’t happens rarely please take off the sanity blocker tag.

[Sudheendra Rao (sudheendra-k)]

removing sanityblocker as the problem was not seen on build4 of R4.1, will provide the setup in the failed state if it's reproducible again.