SM R 3.1 build 17: HA: compute nodes are not listed in discovery

[sundarkh]

SM R 3.1 build 17: HA: compute nodes are not listed in discovery

SM : nodej8

   "server": [
        {
            "cluster_id": "cluster5sanity",
            "id": "nodeg21",
            "ip_address": "10.204.217.61",
            "roles": [
                "control",
                "webui",
                "openstack",
                "database",
                "collector",
                "config"
            ]
        },
        {
            "cluster_id": "cluster5sanity",
            "id": "nodec38",
            "ip_address": "10.204.217.23",
            "roles": [
                "compute"
            ]
        },
        {
            "cluster_id": "cluster5sanity",
            "id": "nodeg17",
            "ip_address": "10.204.217.57",
            "roles": [
                "compute"
            ]
        },
        {
            "cluster_id": "cluster5sanity",
            "id": "nodec58",
            "ip_address": "10.204.217.98",
            "roles": [
                "control",
                "webui",
                "openstack",
                "database",
                "collector",
                "config"
            ]
        },
        {
            "cluster_id": "cluster5sanity",
            "id": "nodeg33",
            "ip_address": "10.204.217.73",
            "roles": [
                "control",
                "webui",
                "openstack",
                "database",
                "collector",
                "config"
            ]
        }
    ]

The two compute nodes nodeg17, nodec38 are not listed in discovery of nodeg21 (cfgm node)
2016-08-07 14:32:25,322 - ERROR - All the required BGP controller are not found in agent introspect for nodec38

2016-08-07 14:32:25,325 - ERROR - All the required BGP controller are not found in agent introspect for nodeg17

Traces on the compute nodes

contrail-vrouter-agent.log" in compute node nodeg17

016-08-07 Sun 12:48:50:269.292 PDT nodeg17 [Thread 140658161133504, Pid 23056]: Configuration error. vhost interface name not specified
2016-08-07 Sun 12:48:51:288.710 PDT nodeg17 [Thread 139834353948608, Pid 23457]: Configuration error. vhost interface name not specified
2016-08-07 Sun 12:48:53:306.849 PDT nodeg17 [Thread 140346598787008, Pid 23891]: Configuration error. vhost interface name not specified
2016-08-07 Sun 12:48:56:385.176 PDT nodeg17 [Thread 140349693282240, Pid 24160]: Configuration error. vhost interface name not specified
2016-08-07 Sun 12:50:29:715.969 PDT nodeg17 [Thread 140575288293120, Pid 1756]: SANDESH: No Client: 1470599429715896 SandeshModuleClientTrace: data= [ name = nodeg17:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1756 http_port = 8085 start_time = 1470599429715378 collector_name = primary = 0.0.0.0:0 secondary = 0.0.0.0:0 rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_diff= [ [ _iter99->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 ], ] ] tx_msg_diff= [ [ _iter103->first = dropped_no_client _iter103->second = 1, ] ] ]
2016-0

nodec38 compute vroute-agentlog

Sun 12:52:15:634.827 PDT nodec38 [Thread 140343580407744, Pid 3754]: Configuration error. vhost interface name not specified
2016-08-07 Sun 12:52:17:651.080 PDT nodec38 [Thread 139986845235136, Pid 3839]: Configuration error. vhost interface name not specified
2016-08-07 Sun 12:52:20:668.215 PDT nodec38 [Thread 140584477059008, Pid 3953]: Configuration error. vhost interface name not specified
2016-08-07 Sun 12:58:07:197.027 PDT nodec38 [Thread 140269395683072, Pid 1807]: SANDESH: No Client: 1470599887196923 SandeshModuleClientTrace: data= [ name = nodec38:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 1807 http_port = 8085 start_time = 1470599887061373 collector_name = primary = 0.0.0.0:0 secondary = 0.0.0.0:0 rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_diff= [ [ _iter99->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 ], ] ] tx_msg_diff= [ [ _iter103->first = dropped_no_client _iter103->second = 1, ] ] ]
2016-08-07 Sun 12:58:07:245.558 PDT nodec38 [Thread 140269387286272, Pid 1807]: SANDESH: Send FAILED: 1470599887245385 NodeStatusUVE: data= [ name = nodec38 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] description = Collector connection down Number of connections:1, Expected: 4 ], ] ] ]
2016-08-07 Sun 12:58:07:245.599 PDT nodec38 [Thread 140269387286272, Pid 1807]: SANDESH: Send FAILED: 1470599887245407 NodeStatusUVE: data= [ name = nodec38 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], [ type = Discovery name = Collector server_addrs= [ [ (*_iter6) = 10.204.217.176:5998, ] ] status = Initializing description = Subscribe ], ] ] description = Collector, Discovery:Collector connection down Number of connec

[sundarkh (sundar-kh)]

one of the test case getting impacted
test_verify_generator_collector_connections

ERROR - All the required BGP controller are not found in agent introspect for nodec38

This is making the http://10.204.217.176:8081/analytics/uves/generator/nodeg21:Control:contrail-control:0?flat fail with authorisation failed

[Abhay Joshi (abhayj)]

We checked the configuration and both vrouter (nodeg17 and nodec38) agents are correctly connected to 2 of the 3 control nodes. Please see below:

http://nodeg21.englab.juniper.net:5998/clients

There seems to be a problem in the test case. Can someone look into this from test team? The analysis of failure may not be correct.

Logs are showing :

2016-08-08 17:40:15,501 - DEBUG - Requesting: http://10.204.217.176:8081/analytics/uves/generator/nodec38:Compute:contrail-vrouter-agent:0?flat
2016-08-08 17:40:15,524 - DEBUG - Requesting: http://10.204.217.176:8081/analytics/uves/vrouter/nodec38?flat
2016-08-08 17:40:15,548 - ERROR - vrouter nodec38 not connected to 2 xmpp peers, should connect to at least one more from set(['10.204.217.61'])
2016-08-08 17:40:18,551 - DEBUG - Requesting: http://10.204.217.176:8081/analytics/uves/generator/nodec38:Compute:contrail-vrouter-agent:0?flat
2016-08-08 17:40:18,567 - DEBUG - Requesting: http://10.204.217.176:8081/analytics/uves/vrouter/nodec38?flat
2016-08-08 17:40:18,592 - ERROR - vrouter nodec38 not connected to 2 xmpp peers, should connect to at least one more from set(['10.204.217.61'])

Test seems to look for connection from agent to node 10.204.217.61, but it is already connected to 10.204.217.73 and 10.204.217.98).

[Abhay Joshi (abhayj)]

Output of curl -H "X-Auth-Token: 41a9de9971d5411d95aa20ab0ec61d93" http://10.204.217.176:8081/analytics/uves/vrouter/nodeg17 and nodec38 attached.

[Sudheendra Rao (sudheendra-k)]

the problem was due to wrong testbed.py configuration.