[Mainline/Mitaka] xmpp connection getting reset while running testcases

[Suresh]

   In the latest mainline build 44, xmpp connection getting reset while running testcases. Because of this issue, many testcases hitting intermittent issues such as ping failure, VN verification failure etc.

   Able to see the below failure messages in contrail-contril.log and contrail-vrouter-agent.log

contrail-contril.log:
=====================
2017-09-11 Mon 12:36:46:966.125 IST nodem7 [Thread 47126734178048, Pid 21478]: BGP [SYS_CRIT]: XmppPeerInstanceLog: XMPP Peer nodem7:10.10.10.10 RECV in instance default-domain:ctest-TestIntfMirror-32613908:ctest-vn3-01828050:ctest-vn3-01828050 : Received route without subscribe controller/src/bgp/bgp_xmpp_channel.cc 747
2017-09-11 Mon 12:36:46:966.300 IST nodem7 [Thread 47126734178048, Pid 21478]: BGP [SYS_CRIT]: XmppPeerInstanceLog: XMPP Peer nodem7:10.10.10.10 RECV in instance default-domain:ctest-TestIntfMirror-32613908:ctest-vn3-01828050:ctest-vn3-01828050 : Received route without subscribe controller/src/bgp/bgp_xmpp_channel.cc 747
2017-09-11 Mon 12:36:47:017.698 IST nodem7 [Thread 47125970855680, Pid 21478]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem7:10.10.10.10 RECV Mode Server: PassiveOpen in state: Idle peer ip: 10.10.10.10 ( ) controller/src/xmpp/xmpp_state_machine.cc 1327
2017-09-11 Mon 12:37:05:085.025 IST nodem7 [Thread 47126801286912, Pid 21478]: BGP [SYS_CRIT]: XmppPeerInstanceLog: XMPP Peer nodem7:10.10.10.8 RECV in instance default-domain:ctest-TestIntfMirror-32613908:ctest-src_parent_port_vn-04114043:ctest-src_parent_port_vn-04114043 : Received route without subscribe controller/src/bgp/bgp_xmpp_channel.cc 747
2017-09-11 Mon 12:37:05:134.930 IST nodem7 [Thread 47125958260480, Pid 21478]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem7:10.10.10.8 RECV Mode Server: PassiveOpen in state: Idle peer ip: 10.10.10.8 ( ) controller/src/xmpp/xmpp_state_machine.cc 1327
2017-09-11 Mon 12:37:33:710.380 IST nodem7 [Thread 47125949863680, Pid 21478]: IFMap [SYS_WARN]: IFMapIdentifierNotFound: Cant find identifier target:64510:8000016 controller/src/ifmap/ifmap_server_table.cc 255

contrail-vrouter-agent.log:
===========================
root@nodem8:/var/log/contrail# tail -f contrail-vrouter-agent.log
2017-09-11 Mon 10:37:29:747.289 IST nodem8 [Thread 139782277740288, Pid 34275]: IFMapAgent [SYS_ERR]: InterfaceConfiguration: Virtual-network UUID mismatch for interface: 40ed75e8-7513-4fdf-ac77-3df132a8aade configuration VN uuid 00000000-0000-0000-0000-000000000000 compute VN uuid c74b365b-3c43-4a57-bf56-1dbf675c1f24 controller/src/vnsw/agent/oper/vm_interface_config.cc 814
2017-09-11 Mon 12:02:04:410.651 IST nodem8 [Thread 139782436730816, Pid 34275]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.10.10.8:39298::10.10.10.14:5269 < Read failed due to error 104 : Connection reset by peer controller/src/io/tcp_session.cc 480
2017-09-11 Mon 12:02:04:410.808 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.14 RECV Mode Client: Event: Tcp Connection Closed peer ip: 10.10.10.14 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1312
2017-09-11 Mon 12:02:04:410.964 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.7 RECV Mode Client: Event: Tcp Connection Closed peer ip: 10.10.10.7 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1312
2017-09-11 Mon 12:02:04:468.495 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.14 RECV Mode Client: Event: Tcp Connected peer ip: 10.10.10.14 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1297
2017-09-11 Mon 12:02:04:468.671 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.7 RECV Mode Client: Event: Tcp Connected peer ip: 10.10.10.7 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1297
2017-09-11 Mon 12:07:34:488.403 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.14 RECV Mode Client: Event: Tcp Connection Closed peer ip: 10.10.10.14 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1312
2017-09-11 Mon 12:07:34:488.580 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.7 RECV Mode Client: Event: Tcp Connection Closed peer ip: 10.10.10.7 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1312
2017-09-11 Mon 12:07:34:540.885 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.7 RECV Mode Client: Event: Tcp Connected peer ip: 10.10.10.7 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1297
2017-09-11 Mon 12:07:34:541.072 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.14 RECV Mode Client: Event: Tcp Connected peer ip: 10.10.10.14 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1297
2017-09-11 Mon 12:33:03:690.704 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.7 RECV Mode Client: Event: Tcp Connection Closed peer ip: 10.10.10.7 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1312
2017-09-11 Mon 12:33:03:709.333 IST nodem8 [Thread 139782273541888, Pid 34275]: VRouter operation failed. Error <19:Object not present>. Object <NextHop Index: 35 Type: Tunnel to 10.10.10.8>. Operation <Addition>. Message number :10889
2017-09-11 Mon 12:33:03:741.519 IST nodem8 [Thread 139782265145088, Pid 34275]: VRouter operation failed. Error <22:Invalid object parameters>. Object <NextHop Index: 35 Type: Tunnel to 10.10.10.8>. Operation <Deletion>. Message number :10961
2017-09-11 Mon 12:33:03:744.254 IST nodem8 [Thread 139782436730816, Pid 34275]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodem8:10.10.10.7 RECV Mode Client: Event: Tcp Connected peer ip: 10.10.10.7 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1297

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35507
Submitter: Naveen N (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35512
Submitter: Naveen N (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35512
Committed: http://github.com/Juniper/contrail-controller/commit/8e388d259ddb856564dcd8bf137d3e265807a41b
Submitter: Zuul (<email address hidden>)
Branch: master

commit 8e388d259ddb856564dcd8bf137d3e265807a41b
Author: Naveen N <email address hidden>
Date: Tue Sep 12 09:59:45 2017 +0530

* Dont export multicast route upon VRF delete

Ideally upon VRF delete control-node implictly deletes all the
paths added by agent in particular VRF, in problematic scenario
after VRF unsubscribe was done agent used to send a route delete
for multicast route, resulting in control-node closing the channel.

Change-Id: I440fa9bc561bf50229a7134122093d487b93bfb7
Closes-bug: #1716356

[Sergey Kreys (skreys)]

Are there plans to backport this fix to other branches?
We see similar issue on R3.1.1.x