VM object still present in Agent's config DB. Agent has not sent Unsubscribe for VM bd4466bb-184d-4a9a-981d-6a6302faf2f7. This can happen when there is Xmpp Channel flap because of presence of issue https://bugs.launchpad.net/juniperopenstack/+bug/1593716 in R2.21.2-36
(gdb) p Agent::singleton_->cfg_
$3 = (AgentConfig *) 0x7f9830000d00
(gdb) p $3->vm_cfg_table_
There is no member or method named vm_cfg_table_.
(gdb) p $3->cfg_vm_table_
$4 = (autogen::DBTable_Agent_VirtualMachine *) 0x7f983000c830
(gdb) dump_ifmap_entries 0x7f983000c830
--------------------------------------------
0x7f98281e2f70 name=00256b4b-752e-4c10-a8f6-2f7821e07f81
0x7f98044892c0 name=03e0b056-7d87-43c8-83d8-7f96179a8800
0x7f9814111e60 name=05a4ec9e-544f-4843-a729-b3d4427f30c8
0x7f98343f5bb0 name=0daf5cdc-52ab-4317-a7aa-f1318a373926
0x7f9834398d40 name=18c494c9-61d7-4008-bf13-1dff51d96acb
0x7f98142bee80 name=1fc8ae71-6331-495f-a5ee-e4298ece5725
0x7f9810940b70 name=3d95a1e3-f6d0-4804-8c3c-d7d62b78cfb2
0x7f97fc062b90 name=41d4c09b-84bc-48f2-8e2b-0ea575c98ca0
0x7f98042258e0 name=41f56cb8-3117-4822-8708-c478ebc7b08c
0x7f97f0a59390 name=4c4b151b-9cfe-40d5-8eb2-db7e0b4d1e02
0x7f980032d2a0 name=5ac03053-8049-4508-820d-511c7f365bf2
0x7f980401c5d0 name=842a9d4d-e9f4-4c04-a0b2-2875beeedab1
0x7f98346b0900 name=87d3cc5c-50cc-4c92-bbe1-6cd1140601c1
0x7f981c381c90 name=88e80b6c-c6b7-4068-b501-d118999bb6f8
0x7f981c2f04b0 name=aab86767-ddb6-4728-b7b0-65ee09c2cca2
0x7f982c313d00 name=b8f9616d-035b-4c06-8493-2b4d8d57831c
0x7f983411fa80 name=b9b7a201-83dd-4423-90e7-e29c59c6cfa7
0x7f97fc937340 name=bd4466bb-184d-4a9a-981d-6a6302faf2f7
0x7f97fca69fb0 name=d0b1d2ea-d5fb-4c45-8425-68e805e6a6ea
0x7f983c459e80 name=e5be3c1f-935c-4451-b0a9-80f50049c52d
0x7f9830338130 name=eb23f5d4-0706-429f-8e92-36bf6e7d3983
0x7f9844299800 name=eec8f83f-6ee5-47a5-804c-33b5399c7255
0x7f9811083940 name=f7d53973-0bc4-4c47-884f-7ae0d851bf56
Xmpp Channel flaps confirmed by trace logs of controller taken from core file
controller_trace.log:3220:2017-01-25 09:32:55.881445 AgentXmppSession: peer = "10.3.135.73" event = "NOT_READY" tree_builder = "NULL" message = "BGP peer decommissioned for xmpp channel." file = "controller/src/vnsw/agent/controller/controller_peer.cc" line = 1465
controller_trace.log:3249:2017-01-25 09:32:55.881880 AgentXmppSession: peer = "10.3.135.72" event = "NOT_READY" tree_builder = "NULL" message = "BGP peer selected asconfig peer on decommission of old config peer." file = "controller/src/vnsw/agent/controller/controller_peer.cc" line = 1491
controller_trace.log:3252:2017-01-25 09:32:55.881891 AgentXmppSession: peer = "10.3.135.73" event = "NOT_READY" tree_builder = "10.3.135.72" message = "Peer elected Multicast Tree Builder" file = "controller/src/vnsw/agent/controller/controller_peer.cc" line = 1543
controller_trace.log:3311:2017-01-26 11:26:12.898147 AgentXmppSession: peer = "10.3.135.74" event = "NOT_READY" tree_builder = "NULL" message = "BGP peer decommissioned for xmpp channel." file = "controller/src/vnsw/agent/controller/controller_peer.cc" line = 1465
controller_trace.log:3667:2017-02-28 23:54:58.624388 AgentXmppSession: peer = "10.3.135.72" event = "NOT_READY" tree_builder = "NULL" message = "BGP peer decommissioned for xmpp channel." file = "controller/src/vnsw/agent/controller/controller_peer.cc" line = 1465
controller_trace.log:3698:2017-02-28 23:54:58.624927 AgentXmppSession: peer = "10.3.135.73" event = "NOT_READY" tree_builder = "NULL" message = "BGP peer selected asconfig peer on decommission of old config peer." file = "controller/src/vnsw/agent/controller/controller_peer.cc" line = 1491
controller_trace.log:3701:2017-02-28 23:54:58.624939 AgentXmppSession: peer = "10.3.135.72" event = "NOT_READY" tree_builder = "10.3.135.73" message = "Peer elected Multicast Tree Builder" file = "controller/src/vnsw/agent/controller/controller_peer.cc" line = 1543
We are hitting the issue /bugs.launchpad .net/juniperope nstack/ +bug/1593716
https:/
VM object still present in Agent's config DB. Agent has not sent Unsubscribe for VM bd4466bb- 184d-4a9a- 981d-6a6302faf2 f7. This can happen when there is Xmpp Channel flap because of presence of issue https:/ /bugs.launchpad .net/juniperope nstack/ +bug/1593716 in R2.21.2-36
(gdb) p Agent:: singleton_ ->cfg_ :DBTable_ Agent_VirtualMa chine *) 0x7f983000c830 ------- ------- ------- ------- ------- -- 752e-4c10- a8f6-2f7821e07f 81 7d87-43c8- 83d8-7f96179a88 00 544f-4843- a729-b3d4427f30 c8 52ab-4317- a7aa-f1318a3739 26 61d7-4008- bf13-1dff51d96a cb 6331-495f- a5ee-e4298ece57 25 f6d0-4804- 8c3c-d7d62b78cf b2 84bc-48f2- 8e2b-0ea575c98c a0 3117-4822- 8708-c478ebc7b0 8c 9cfe-40d5- 8eb2-db7e0b4d1e 02 8049-4508- 820d-511c7f365b f2 e9f4-4c04- a0b2-2875beeeda b1 50cc-4c92- bbe1-6cd1140601 c1 c6b7-4068- b501-d118999bb6 f8 ddb6-4728- b7b0-65ee09c2cc a2 035b-4c06- 8493-2b4d8d5783 1c 83dd-4423- 90e7-e29c59c6cf a7 184d-4a9a- 981d-6a6302faf2 f7 d5fb-4c45- 8425-68e805e6a6 ea 935c-4451- b0a9-80f50049c5 2d 0706-429f- 8e92-36bf6e7d39 83 6ee5-47a5- 804c-33b5399c72 55 0bc4-4c47- 884f-7ae0d851bf 56
$3 = (AgentConfig *) 0x7f9830000d00
(gdb) p $3->vm_cfg_table_
There is no member or method named vm_cfg_table_.
(gdb) p $3->cfg_vm_table_
$4 = (autogen:
(gdb) dump_ifmap_entries 0x7f983000c830
-------
0x7f98281e2f70 name=00256b4b-
0x7f98044892c0 name=03e0b056-
0x7f9814111e60 name=05a4ec9e-
0x7f98343f5bb0 name=0daf5cdc-
0x7f9834398d40 name=18c494c9-
0x7f98142bee80 name=1fc8ae71-
0x7f9810940b70 name=3d95a1e3-
0x7f97fc062b90 name=41d4c09b-
0x7f98042258e0 name=41f56cb8-
0x7f97f0a59390 name=4c4b151b-
0x7f980032d2a0 name=5ac03053-
0x7f980401c5d0 name=842a9d4d-
0x7f98346b0900 name=87d3cc5c-
0x7f981c381c90 name=88e80b6c-
0x7f981c2f04b0 name=aab86767-
0x7f982c313d00 name=b8f9616d-
0x7f983411fa80 name=b9b7a201-
0x7f97fc937340 name=bd4466bb-
0x7f97fca69fb0 name=d0b1d2ea-
0x7f983c459e80 name=e5be3c1f-
0x7f9830338130 name=eb23f5d4-
0x7f9844299800 name=eec8f83f-
0x7f9811083940 name=f7d53973-
Xmpp Channel flaps confirmed by trace logs of controller taken from core file
controller_ trace.log: 3220:2017- 01-25 09:32:55.881445 AgentXmppSession: peer = "10.3.135.73" event = "NOT_READY" tree_builder = "NULL" message = "BGP peer decommissioned for xmpp channel." file = "controller/ src/vnsw/ agent/controlle r/controller_ peer.cc" line = 1465 trace.log: 3249:2017- 01-25 09:32:55.881880 AgentXmppSession: peer = "10.3.135.72" event = "NOT_READY" tree_builder = "NULL" message = "BGP peer selected asconfig peer on decommission of old config peer." file = "controller/ src/vnsw/ agent/controlle r/controller_ peer.cc" line = 1491 trace.log: 3252:2017- 01-25 09:32:55.881891 AgentXmppSession: peer = "10.3.135.73" event = "NOT_READY" tree_builder = "10.3.135.72" message = "Peer elected Multicast Tree Builder" file = "controller/ src/vnsw/ agent/controlle r/controller_ peer.cc" line = 1543 trace.log: 3311:2017- 01-26 11:26:12.898147 AgentXmppSession: peer = "10.3.135.74" event = "NOT_READY" tree_builder = "NULL" message = "BGP peer decommissioned for xmpp channel." file = "controller/ src/vnsw/ agent/controlle r/controller_ peer.cc" line = 1465 trace.log: 3667:2017- 02-28 23:54:58.624388 AgentXmppSession: peer = "10.3.135.72" event = "NOT_READY" tree_builder = "NULL" message = "BGP peer decommissioned for xmpp channel." file = "controller/ src/vnsw/ agent/controlle r/controller_ peer.cc" line = 1465 trace.log: 3698:2017- 02-28 23:54:58.624927 AgentXmppSession: peer = "10.3.135.73" event = "NOT_READY" tree_builder = "NULL" message = "BGP peer selected asconfig peer on decommission of old config peer." file = "controller/ src/vnsw/ agent/controlle r/controller_ peer.cc" line = 1491 trace.log: 3701:2017- 02-28 23:54:58.624939 AgentXmppSession: peer = "10.3.135.72" event = "NOT_READY" tree_builder = "10.3.135.73" message = "Peer elected Multicast Tree Builder" file = "controller/ src/vnsw/ agent/controlle r/controller_ peer.cc" line = 1543
controller_
controller_
controller_
controller_
controller_
controller_