Build 2689: Alarms : 'any_of' list empty when AddressMismatch alarms are generated

Bug #1529776 reported by Ankit Jain
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
Trunk
Fix Committed
Critical
Anish Mehta

Bug Description

Alarms of type AddressMismatch are always being shown up for control node and vrouter in the alarms UVEs

'any_of' list in Addressmismatch alarms does not show any info ( rule, operand1, operand2, json values as expected) about the alarm

In case of vrouter, self ip is matching with virtual_router_ip_address as shown below but Addressmismatch are still being gererated

"self_ip_list": [

    "40.41.1.5"

],

"virtual_router_ip_address": "\"40.41.1.5\"",

Alarms:

"control-node": [

{

    "name": "nodeh2",
    "value":

{

    "UVEAlarms":

{

    "alarms":

[

{

    "any_of":

[

{

    "all_of":

[

{

    "json_operand2_value": "2",
    "json_operand1_value": "1",
    "rule":

{

    "oper": "!=",
    "operand1":

{

    "keys":

    [
        "BgpRouterState",
        "num_up_bgp_peer"
    ]

},
"operand2":
{

    "keys":

                            [
                                "BgpRouterState",
                                "num_bgp_peer"
                            ]
                        }
                    }
                }
            ]
        }
    ],
    "severity": ​4,
    "ack": false,
    "timestamp": ​1451303988206767,
    "token": "eyJ0aW1lc3RhbXAiOiAxNDUxMzAzOTg4MjA2NzY3LCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiNDAuNDMuMS43In0=",
    "type": "BgpConnectivity"

},
{

    "any_of":

[

                        {
                            "all_of": [ ]
                        }
                    ],
                    "severity": ​3,
                    "ack": false,
                    "timestamp": ​1451303988206880,
                    "token": "eyJ0aW1lc3RhbXAiOiAxNDUxMzAzOTg4MjA2ODgwLCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiNDAuNDMuMS43In0=",
                    "type": "AddressMismatchControl"
                }
            ]
        }
    }

},
{

    "name": "nodeh1",
    "value":

{

    "UVEAlarms":

{

    "alarms":

[

{

    "any_of":

[

{

    "all_of":

[

{

    "json_operand2_value": "2",
    "json_operand1_value": "1",
    "rule":

{

    "oper": "!=",
    "operand1":

{

    "keys":

    [
        "BgpRouterState",
        "num_up_bgp_peer"
    ]

},
"operand2":
{

    "keys":

                            [
                                "BgpRouterState",
                                "num_bgp_peer"
                            ]
                        }
                    }
                }
            ]
        }
    ],
    "severity": ​4,
    "ack": false,
    "timestamp": ​1451303987027264,
    "token": "eyJ0aW1lc3RhbXAiOiAxNDUxMzAzOTg3MDI3MjY0LCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiNDAuNDMuMS43In0=",
    "type": "BgpConnectivity"

},
{

    "any_of":

[

                            {
                                "all_of": [ ]
                            }
                        ],
                        "severity": ​3,
                        "ack": false,
                        "timestamp": ​1451303987027381,
                        "token": "eyJ0aW1lc3RhbXAiOiAxNDUxMzAzOTg3MDI3MzgxLCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiNDAuNDMuMS43In0=",
                        "type": "AddressMismatchControl"
                    }
                ]
            }
        }
    }

],

vrouter": [

{

    "name": "nodei5",
    "value":

{

    "UVEAlarms":

{

    "alarms":

[

{

    "any_of":

[

                        {
                            "all_of": [ ]
                        }
                    ],
                    "severity": ​3,
                    "ack": false,
                    "timestamp": ​1451303987538340,
                    "token": "eyJ0aW1lc3RhbXAiOiAxNDUxMzAzOTg3NTM4MzQwLCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiNDAuNDMuMS43In0=",
                    "type": "AddressMismatchCompute"
                }
            ]
        }
    }

},
{

    "name": "nodeh6",
    "value":

{

    "UVEAlarms":

{

    "alarms":

[

{

    "any_of":

[

                        {
                            "all_of": [ ]
                        }
                    ],
                    "severity": ​3,
                    "ack": false,
                    "timestamp": ​1451303988261344,
                    "token": "eyJ0aW1lc3RhbXAiOiAxNDUxMzAzOTg4MjYxMzQ0LCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiNDAuNDMuMS43In0=",
                    "type": "AddressMismatchCompute"
                }
            ]
        }
    }

},
{

    "name": "nodei4",
    "value":

{

    "UVEAlarms":

{

    "alarms":

[

{

    "any_of":

[

{

    "all_of":

[

{

    "json_operand1_value": "\"PROCESS_STATE_FATAL\"",
    "rule":

{

    "oper": "!=",
    "operand1":

{

    "keys":

    [
        "NodeStatus",
        "process_info",
        "process_state"
    ]

},
"operand2":

    {
        "json_value": "\"PROCESS_STATE_RUNNING\""
    }

},
"json_vars":

                    {
                        "NodeStatus.process_info.process_name": "openstack-nova-compute"
                    }
                }
            ]
        }
    ],
    "severity": ​3,
    "ack": false,
    "timestamp": ​1451303988092445,
    "token": "eyJ0aW1lc3RhbXAiOiAxNDUxMzAzOTg4MDkyNDQ1LCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiNDAuNDMuMS43In0=",
    "type": "ProcessStatus"

},
{

    "any_of":

[

                        {
                            "all_of": [ ]
                        }
                    ],
                    "severity": ​3,
                    "ack": false,
                    "timestamp": ​1451303988092558,
                    "token": "eyJ0aW1lc3RhbXAiOiAxNDUxMzAzOTg4MDkyNTU4LCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiNDAuNDMuMS43In0=",
                    "type": "AddressMismatchCompute"
                }
            ]
        }
    }

},
{

    "name": "nodeh7",
    "value":

{

    "UVEAlarms":

{

    "alarms":

[

{

    "any_of":

[

                                {
                                    "all_of": [ ]
                                }
                            ],
                            "severity": ​3,
                            "ack": false,
                            "timestamp": ​1451303988227474,
                            "token": "eyJ0aW1lc3RhbXAiOiAxNDUxMzAzOTg4MjI3NDc0LCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiNDAuNDMuMS43In0=",
                            "type": "AddressMismatchCompute"
                        }
                    ]
                }
            }
        }
    ]

}

Vrouter UVEs:

{

    "ComputeCpuState":

{

    "cpu_info":

[

        {
            "mem_res": ​97436,
            "cpu_share": ​0.0541667,
            "used_sys_mem": ​2759388,
            "mem_virt": ​598628,
            "one_min_cpuload": ​0.0325
        }
    ]

},
"NodeStatus":
{

    "deleted": false,
    "disk_usage_info":

[

{

    "partition_space_available_1k": ​388973968,
    "partition_space_used_1k": ​4231588,
    "partition_name": "/dev/mapper/nodeh6--vg-root",
    "partition_type": "ext4"

},

    {
        "partition_space_available_1k": ​160917,
        "partition_space_used_1k": ​67614,
        "partition_name": "/dev/sda1",
        "partition_type": "ext2"
    }

],
"process_info":
[

{

    "process_name": "contrail-vrouter-agent",
    "start_count": ​1,
    "process_state": "PROCESS_STATE_RUNNING",
    "last_stop_time": null,
    "core_file_list": [ ],
    "last_start_time": "1451288226525243",
    "stop_count": ​0,
    "last_exit_time": null,
    "exit_count": ​0

},
{

    "process_name": "contrail-vrouter-nodemgr",
    "start_count": ​1,
    "process_state": "PROCESS_STATE_RUNNING",
    "last_stop_time": null,
    "core_file_list": [ ],
    "last_start_time": "1451288222189497",
    "stop_count": ​0,
    "last_exit_time": null,
    "exit_count": ​0

},

    {
        "process_name": "openstack-nova-compute",
        "start_count": ​1,
        "process_state": "PROCESS_STATE_RUNNING",
        "last_stop_time": null,
        "core_file_list": [ ],
        "last_start_time": "1451288221102082",
        "stop_count": ​0,
        "last_exit_time": null,
        "exit_count": ​0
    }

],
"process_status":
[

{

    "instance_id": "0",
    "module_id": "contrail-vrouter-nodemgr",
    "state": "Functional",
    "description": null

},
{

    "instance_id": "0",
    "module_id": "contrail-vrouter-agent",
    "state": "Functional",
    "description": null,
    "connection_infos":

[

{

    "server_addrs":

    [
        "40.43.1.6:5269"
    ],
    "status": "Up",
    "type": "XMPP",
    "name": "control-node:40.43.1.6",
    "description": "OpenSent"

},
{

    "server_addrs":

    [
        "40.43.1.7:5269"
    ],
    "status": "Up",
    "type": "XMPP",
    "name": "control-node:40.43.1.7",
    "description": "OpenSent"

},
{

    "server_addrs":

    [
        "40.43.1.6:53"
    ],
    "status": "Up",
    "type": "XMPP",
    "name": "dns-server:40.43.1.6",
    "description": "OpenSent"

},
{

    "server_addrs":

    [
        "40.43.1.7:53"
    ],
    "status": "Up",
    "type": "XMPP",
    "name": "dns-server:40.43.1.7",
    "description": "OpenSent"

},
{

    "server_addrs":

    [
        "40.43.1.7:8086"
    ],
    "status": "Up",
    "type": "Collector",
    "name": null,
    "description": "Established"

},
{

    "server_addrs":

    [
        "40.43.1.5:5998"
    ],
    "status": "Up",
    "type": "Discovery",
    "name": "Collector",
    "description": "SubscribeResponse"

},
{

    "server_addrs":

    [
        "40.43.1.5:5998"
    ],
    "status": "Up",
    "type": "Discovery",
    "name": "dns-server",
    "description": "SubscribeResponse"

},
{

    "server_addrs":

                    [
                        "40.43.1.5:5998"
                    ],
                    "status": "Up",
                    "type": "Discovery",
                    "name": "xmpp-server",
                    "description": "SubscribeResponse"
                }
            ]
        }
    ]

},
"ContrailConfig":
{

    "elements":

[

[

    {
        "fq_name": "[\"default-global-system-config\", \"nodeh6\"]",
        "uuid": "\"60c434d3-7c3b-4a78-aea1-e6d548c202bf\"",
        "parent_href": "\"http://0.0.0.0:9100/global-system-config/fac4d0bb-cce3-465f-9130-e6e564984d23\"",
        "parent_type": "\"global-system-config\"",
        "perms2": "{\"owner\": \"8a01a7492f2a47e99de61b77b49b7603\", \"owner_access\": 7, \"global_access\": 0, \"share\": []}",
        "virtual_router_type": "[]",
        "display_name": "\"nodeh6\"",
        "virtual_router_ip_address": "\"40.41.1.5\"",
        "id_perms": "{\"enable\": true, \"uuid\": {\"uuid_mslong\": 6972756206003374712, \"uuid_lslong\": 12583592637572973247}, \"created\": \"2015-12-28T06:58:44.497200\", \"description\": null, \"creator\": null, \"user_visible\": true, \"last_modified\": \"2015-12-28T06:58:44.497200\", \"permissions\": {\"owner\": \"admin\", \"owner_access\": 7, \"other_access\": 7, \"group\": \"admin\", \"group_access\": 7}}",
        "parent_uuid": "\"fac4d0bb-cce3-465f-9130-e6e564984d23\""
    },
    "nodeg32:Config:contrail-api:0",
    "nodeh1:Config:contrail-api:0"

],
[

            {
                "fq_name": "[\"default-global-system-config\", \"nodeh6\"]",
                "uuid": "\"60c434d3-7c3b-4a78-aea1-e6d548c202bf\"",
                "parent_href": "\"http://0.0.0.0:9100/global-system-config/fac4d0bb-cce3-465f-9130-e6e564984d23\"",
                "parent_type": "\"global-system-config\"",
                "perms2": "{\"owner\": \"8a01a7492f2a47e99de61b77b49b7603\", \"owner_access\": 7, \"global_access\": 0, \"share\": []}",
                "virtual_router_type": "[]",
                "display_name": "\"nodeh6\"",
                "virtual_router_ip_address": "\"40.41.1.5\"",
                "id_perms": "{\"enable\": true, \"description\": null, \"creator\": null, \"created\": \"2015-12-28T06:58:44.497200\", \"user_visible\": true, \"last_modified\": \"2015-12-28T07:31:20.954696\", \"permissions\": {\"owner\": \"admin\", \"owner_access\": 7, \"other_access\": 7, \"group\": \"admin\", \"group_access\": 7}, \"uuid\": {\"uuid_mslong\": 6972756206003374712, \"uuid_lslong\": 12583592637572973247}}",
                "parent_uuid": "\"fac4d0bb-cce3-465f-9130-e6e564984d23\""
            },
            "nodeh2:Config:contrail-api:0"
        ]
    ]

},
"UVEAlarms":
{

    "alarms":

[

{

    "any_of":

[

                {
                    "all_of": [ ]
                }
            ],
            "severity": ​3,
            "ack": false,
            "timestamp": ​1451303988261344,
            "token": "eyJ0aW1lc3RhbXAiOiAxNDUxMzAzOTg4MjYxMzQ0LCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiNDAuNDMuMS43In0=",
            "type": "AddressMismatchCompute"
        }
    ]

},
"VrouterStatsAgent":
{

    "exception_packets": ​44028,
    "in_bytes": ​18469820,
    "udp_sport_bitmap":

[

    "1",
    "0",
    "0",
    "0",
    "1222120640",
    "1814315008",
    "545788928",
    "9868"

],
"total_in_bandwidth_utilization": ​0,
"cpu_info":
{

    "sys_mem_info":

{

    "total": ​65683312,
    "used": ​2759388,
    "free": ​62923924,
    "buffers": ​167556

},
"num_cpu": ​4,
"cpu_share": ​0.0541667,
"meminfo":
{

    "virt": ​598628,
    "peakvirt": ​659772,
    "res": ​97436

},
"cpuload":

    {
        "fifteen_min_avg": ​0.015,
        "five_min_avg": ​0.0225,
        "one_min_avg": ​0.0325
    }

},
"phy_if_10min_usage":
[

    {
        "in_bandwidth_usage": ​0,
        "out_bandwidth_usage": ​0,
        "name": "vlan4001"
    }

],
"uptime": ​1451288222523915,
"total_flows": ​220,
"phy_if_5min_usage":
[

    {
        "in_bandwidth_usage": ​0,
        "out_bandwidth_usage": ​0,
        "name": "vlan4001"
    }

],
"drop_stats":
{

    "ds_mcast_df_bit": ​0,
    "ds_flow_no_memory": ​0,
    "ds_composite_invalid_interface": ​0,
    "ds_push": ​0,
    "ds_invalid_if": ​0,
    "ds_pull": ​0,
    "ds_no_fmd": ​0,
    "ds_invalid_arp": ​0,
    "ds_trap_no_if": ​0,
    "ds_arp_reply_no_route": ​0,
    "ds_invalid_source": ​0,
    "ds_flow_action_invalid": ​0,
    "ds_invalid_packet": ​0,
    "ds_flow_invalid_protocol": ​0,
    "ds_discard": ​0,
    "ds_invalid_vnid": ​0,
    "ds_flow_table_full": ​0,
    "ds_invalid_label": ​0,
    "ds_garp_from_vm": ​0,
    "ds_frag_err": ​0,
    "ds_duplicated": ​0,
    "ds_invalid_nh": ​4,
    "ds_arp_no_route": ​0,
    "ds_misc": ​2,
    "ds_flood": ​0,
    "ds_interface_rx_discard": ​0,
    "ds_head_alloc_fail": ​0,
    "ds_flow_unusable": ​0,
    "ds_mcast_clone_fail": ​0,
    "ds_invalid_protocol": ​0,
    "ds_head_space_reserve_fail": ​0,
    "ds_interface_tx_discard": ​0,
    "ds_nowhere_to_go": ​0,
    "ds_cloned_original": ​942289,
    "ds_arp_no_where_to_go": ​0,
    "ds_l2_no_route": ​0,
    "ds_cksum_err": ​0,
    "ds_rewrite_fail": ​0,
    "ds_flow_queue_limit_exceeded": ​0,
    "ds_ttl_exceeded": ​0,
    "ds_flow_nat_no_rflow": ​0,
    "ds_clone_fail": ​0,
    "ds_invalid_mcast_source": ​0,
    "ds_interface_drop": ​0,
    "ds_pcow_fail": ​0,
    "ds_flow_action_drop": ​13

},
"tcp_sport_bitmap":
[

    "1",
    "0",
    "0",
    "0",
    "262144",
    "1073741824",
    "4096",
    "1024"

],
"out_tpkts": ​357622,
"phy_if_stats_list":
[

    {
        "out_bytes": ​480714739,
        "name": "vlan4001",
        "duplexity": ​1,
        "out_pkts": ​704712,
        "in_bytes": ​451868078,
        "in_pkts": ​796830,
        "speed": ​2000
    }

],
"exception_packets_dropped": ​8,
"xmpp_stats_list":
[

{

    "ip": "40.43.1.6",
    "reconnects": ​1,
    "in_msgs": ​79,
    "out_msgs": ​58

},

    {
        "ip": "40.43.1.7",
        "reconnects": ​5,
        "in_msgs": ​143,
        "out_msgs": ​80
    }

],
"flow_rate":
{

    "max_flow_deletes_per_second": ​0,
    "added_flows": ​0,
    "deleted_flows": ​0,
    "min_flow_adds_per_second": ​0,
    "min_flow_deletes_per_second": ​0,
    "max_flow_adds_per_second": ​0

},
"out_bytes": ​34577790,
"active_flows": ​2,
"udp_dport_bitmap":
[

    "1",
    "0",
    "0",
    "0",
    "1222120640",
    "1814315008",
    "545788928",
    "9868"

],
"in_tpkts": ​179865,
"vhost_stats":
{

    "out_bytes": ​398143914,
    "name": "vhost0",
    "duplexity": ​-1,
    "out_pkts": ​467849,
    "in_bytes": ​458029505,
    "in_pkts": ​553293,
    "speed": ​-1

},
"total_out_bandwidth_utilization": ​0,
"tcp_dport_bitmap":
[

    "1",
    "0",
    "0",
    "0",
    "262144",
    "1073741824",
    "4096",
    "0"

],
"phy_if_band":
[

        {
            "in_bandwidth_usage": ​0,
            "out_bandwidth_usage": ​0,
            "name": "vlan4001"
        }
    ],
    "aged_flows": ​218,
    "exception_packets_allowed": ​44020

},
"VrouterAgent":
{

    "dns_servers":

[

    "40.43.1.6",
    "40.43.1.7"

],
"log_category": "*",
"control_ip": "40.41.1.5",
"control_node_list_cfg":
[

    "0.0.0.0",
    "0.0.0.0"

],
"hostname_cfg": "nodeh6",
"ds_xs_instances": ​2,
"log_level": "SYS_NOTICE",
"tunnel_type": "MPLSoGRE",
"phy_if":
[

    {
        "name": "vlan4001",
        "mac_address": "00:25:90:c8:e8:1f"
    }

],
"log_local": true,
"vhost_if":
{

    "name": "vhost0",
    "mac_address": "00:25:90:c8:e8:1f"

},
"platform": "HOST",
"sandesh_http_port": ​8085,
"eth_name": "vlan4001",
"log_flow": false,
"headless_mode_cfg": false,
"down_interface_count": ​0,
"self_ip_list":
[

    "40.41.1.5"

],
"vn_count": ​2,
"config_file": "/etc/contrail/contrail-vrouter-agent.conf",
"vhost_cfg":
{

    "ip": "40.41.1.5",
    "ip_prefix_len": ​24,
    "name": "vhost0",
    "gateway": "40.41.1.1"

},
"ll_max_vm_flows_cfg": ​2048,
"ll_max_system_flows_cfg": ​2048,
"flow_cache_timeout_cfg": ​0,
"max_vm_flows_cfg": ​100,
"total_interface_count": ​2,
"dns_server_list_cfg":
[

    "0.0.0.0",
    "0.0.0.0"

],
"build_info": "{\"build-info\":[{\"build-time\":\"2015-12-28 01:48:10.128535\",\"build-hostname\":\"contrail-ec-build16\",\"build-git-ver\":\"9e91f9e\",\"build-user\":\"contrail-builder\",\"build-version\":\"3.0\",\"build-id\":\"3.0-2689\",\"build-number\":\"2689\"}]}",
"hypervisor": "kvm",
"ds_addr": "40.43.1.5",
"mode": "VROUTER",
"virtual_machine_list":
[

    "11f7a181-321d-4c21-89b6-70f2ef905f97",
    "f392c01d-8d60-4cba-aabc-a1188c8dd842"

],
"xmpp_peer_list":
[

{

    "status": true,
    "ip": "40.43.1.6",
    "primary": true,
    "setup_time": ​1451288223697569

},

    {
        "status": true,
        "ip": "40.43.1.7",
        "primary": false,
        "setup_time": ​1451303678168515
    }

],
"interface_list":
[

    "default-domain:admin:2a5e94f3-e94c-4146-9961-8dcd9a897992",
    "default-domain:admin:4af31613-9a42-41b6-a155-0375dde083a7"

],
"log_file": "/var/log/contrail/contrail-vrouter-agent.log",
"connected_networks":

        [
            "default-domain:admin:VN2",
            "default-domain:admin:VN1"
        ]
    }

}

information type: Proprietary → Public
Revision history for this message
Anish Mehta (amehta00) wrote :

Cannot reproduce this problem.
If you can put a system into this state, I'd like to look at it.

There is another problem withalarm_config_incorrect and alarm_address_mismatch (missing import)
I can patch your test system with the fix, until it merges.

Revision history for this message
Ankit Jain (ankitja) wrote :

Hi Anish,

You can check here :

Setup : nodeg13,nodeg20,nodea21

http://nodeg13:8081/analytics/alarms

{

    "name": "nodea21",
    "value":

{

    "UVEAlarms":

{

    "alarms":

[

{

    "any_of":

[

                        {
                            "all_of": [ ]
                        }
                    ],
                    "severity": ​3,
                    "ack": false,
                    "timestamp": ​1451974167127638,
                    "token": "eyJ0aW1lc3RhbXAiOiAxNDUxOTc0MTY3MTI3NjM4LCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiMTAuMjA0LjIxNy41MyJ9",
                    "type": "AddressMismatchCompute"
                }
            ]
        }
    }

},

Revision history for this message
Anish Mehta (amehta00) wrote :

Should be fixed by this review:
https://review.opencontrail.org/#/c/16108/

Revision history for this message
Anish Mehta (amehta00) wrote :

Handling of multiple config nodes during alarm evaluation

When multiple config nodes are present, the ContrailConfig UVE
struct can be a list-of-list instead of a dict.

This is because we are currently unable to detect if all structs
are equal. We would have to use OrderedDict and do a bunch of
intermediate sorting to do this.

To get around this, the alarm evaluation code will use only one
of the values of ContrailConfig. This means we will not
be able to detect whether different ifmap's have different
configs for an object. Keeping bug open to address that later.

Change-Id: Ic25bc12b417b746d611d4f2064ae1c646f4d1571
Partial-Bug:1530299

Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote :

Anish,

Can you take a look at this ? https://10.204.217.197:8143/#p=mon_infra_control&q[type]=controlNode&q[view]=details&q[focusedElement][node]=testbed-1-vm1&q[focusedElement][tab]=details

This is latest image built on Jan 27

Raj Reddy (rajreddy)
tags: added: blocker
tags: added: amex
Revision history for this message
Anish Mehta (amehta00) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.