This is the email from Raja earlier when I looked at it, I don’t know about the current repro.
In this he clearly mentions “have all interfaces in 65535 VRF”.
Pramodh

From: Pramodh D'Souza <email address hidden>
Date: Wednesday, September 12, 2018 at 4:43 PM
To: Rajakumar David <email address hidden>
Cc: Yuvaraja Mariappan <email address hidden>
Subject: Re: Wrong VRF and DHCP lease after re-creating VM / NEC/Juniper Engineering review of Etisalat VeCPE project 2018-0806-0353 / LP786240

Quick observation:
Noticed these logs in sdnvcpe04cn_Snh_SandeshTraceRequest_Config.log corresponding to c328ec46-2c35-4211-b560-ea5b309cdc8d.

ConfigAddPortEnqueue: op = Add port_uuid = 7a1ccbc9-de44-49fa-9183-179383792538 instance_uuid = c328ec46-2c35-4211-b560-ea5b309cdc8d vn_uuid = feb5da8e-fb2b-412f-a96c-532542be5630 ip_address = 192.168.116.10 system_name = tapsdnvcpe04

ConfigAddPortEnqueue: op = Add port_uuid = 40166ed9-592e-4e90-9622-e8edaeccabf8 instance_uuid = c328ec46-2c35-4211-b560-ea5b309cdc8d vn_uuid = 23c6b87f-b4ec-4b53-a486-305914f44587 ip_address = 10.101.178.134 system_name = tap40166ed9-59 mac_address = 02:40:24:37:aa:16 display_name = VM_IPFE_M_VNFC_Record___0000029683_000973 tx_vlan_id = -1 rx_vlan_id = -1 vm_project_uuid = f6d14e50-ca94-415a-b8df-fbd418db6783 port_type = CfgIntVMPort ip6_address = None file = controller/src/vnsw/agent/port_ipc/port_ipc_handler.cc line = 366

ConfigAddPortEnqueue: op = Add port_uuid = f9644a68-6646-43aa-9b5d-323751bed387 instance_uuid = c328ec46-2c35-4211-b560-ea5b309cdc8d vn_uuid = b12a36da-4383-4d80-801a-ad81ba
a3929c ip_address = 1.2.3.252 system_name = tapf9644a68-66 mac_address = 02:5e:e9:80:21:ca display_name = VM_IPFE_M_VNFC_Record___0000029683_000973 tx_vlan_id = -1 rx_vlan_id = -1 vm_project_uuid = f6d14e50-ca94-415a-b8df-fbd418db6783 port_type = CfgIntVMPort ip6_address = None file = controller/src/vnsw/agent/port_ipc/port_ipc_handler.cc line = 366

ConfigAddPortEnqueue: op = Add port_uuid = 7a1ccbc9-de44-49fa-9
183-179383792538 instance_uuid = c328ec46-2c35-4211-b560-ea5b309cdc8d vn_uuid = feb5da8e-fb2b-412f-a96c-532542be5630 ip_address = 192.168.116.10 system_name = tap7a1ccbc9-de mac_address = 02:39:74:b4:5f:0b display_name = VM_IPFE_M_VNFC_Record___0000029683_000973 tx_vlan_id = -1 rx_vlan_id = -1 vm_project_uuid = f6d14e50-ca94-415a-b8df-fbd418db6783 port_type = CfgIntVMPort ip6_address = None file = controller/src/vnsw/agent/port_ipc/port_ipc_handler.cc line = 366



From: Rajakumar David <email address hidden>
Date: Wednesday, September 12, 2018 at 2:55 AM
To: Anantharamu Suryanarayana <email address hidden>, Ashok Singh R <email address hidden>, Sivakumar Ganapathy <email address hidden>
Cc: contrail-emea <email address hidden>, Contrail Systems Virtual Router Team <email address hidden>, support-private <email address hidden>, Federico Toci <email address hidden>, Alois Zellner <email address hidden>, Nikhil Bansal <email address hidden>, Slobodan Blatnjak <email address hidden>, Mladen Maric <email address hidden>, Assen Tarlov <email address hidden>
Subject: Re: Wrong VRF and DHCP lease after re-creating VM / NEC/Juniper Engineering review of Etisalat VeCPE project 2018-0806-0353 / LP786240

Hello Engineering,

Net cracker were able to recreate this issue in their lab. VM was running at the sdnvcpe05cn, Hypervisor was stopped and healing process started. Now VM is located (created at 08:51:21 EDT) on the sdnvcpe04cn and have all interfaces in 65535 VRF. Logs from all three controller node and introspects can be downloaded from below link https://jtacworkbench.juniper.net/api/download-manager/download?caseId=2018-0906-0624&attId=2018-0906-06240007092484

We are evaluating the logs and we will share our findings by end of the day, it will great if engineering looks into this.

Thanks,
Raja

[root@cr1 ~(vcpe1_offsite)]# nova show c328ec46-2c35-4211-b560-ea5b309cdc8d
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value |
+--------------------------------------+----------------------------------------------------------+
| Access network                       | 192.168.116.10 |
| Network VXLAN-000796 network         | 1.2.3.252 |
| OS-DCF:diskConfig                    | MANUAL |
| OS-EXT-AZ:availability_zone          | nova |
| OS-EXT-SRV-ATTR:host                 | sdnvcpe04cn |
| OS-EXT-SRV-ATTR:hostname             | vm-ipfe-m-vnfc-record---0000029683-000973                |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | sdnvcpe04cn.netcracker.com |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000045d |
| OS-EXT-SRV-ATTR:kernel_id            | |
| OS-EXT-SRV-ATTR:launch_index         | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id           | |
| OS-EXT-SRV-ATTR:reservation_id       | r-7066yr0r |
| OS-EXT-SRV-ATTR:root_device_name     | /dev/vda |
| OS-EXT-SRV-ATTR:user_data            | - |
| OS-EXT-STS:power_state               | 1 |
| OS-EXT-STS:task_state                | - |
| OS-EXT-STS:vm_state                  | active |
| OS-SRV-USG:launched_at               | 2018-09-06T12:51:21.000000 | (means 08:51:21 in EDT timezone)
| OS-SRV-USG:terminated_at             | - |
| VNF-MGMT network                     | 10.101.178.134 |
| accessIPv4                           | |
| accessIPv6                           | |
| config_drive                         | True |
| created                              | 2018-09-06T12:51:00Z |
| description                          | VM_IPFE_M_VNFC_Record___0000029683_000973                |
| flavor                               | a1.IPFE-M (ab1dadf0-7fa9-47d1-8e45-4de3280d5ec0)         |
| hostId                               | cec54b3fb09c5871dc83a867a2ea86f682a9e663cd747bba7c075692 |
| host_status                          | UP |
| id                                   | c328ec46-2c35-4211-b560-ea5b309cdc8d                     |
| image                                | ipfe-6541-1.0a (bb4e0a04-46bb-4811-b5d4-4b495793f31b)    |
| key_name                             | - |
| locked                               | False |
| metadata                             | {} |
| name                                 | VM_IPFE_M_VNFC_Record___0000029683_000973                |
| os-extended-volumes:volumes_attached | [] |
| progress                             | 0 |
| security_groups                      | default |
| status                               | ACTIVE |
| tenant_id                            | f6d14e50ca94415ab8dffbd418db6783                         |
| updated                              | 2018-09-06T12:51:21Z |
| user_id                              | f4495c9c491d4c4d951a3c9a734836b8                         |





From: Anantharamu Suryanarayana <email address hidden>
Date: Monday, August 27, 2018 at 5:44 PM
To: Ashok Singh R <email address hidden>, Mladen Maric <email address hidden>, Sivakumar Ganapathy <email address hidden>, VinayKumar Tejavath <email address hidden>, Assen Tarlov <email address hidden>, Slobodan Blatnjak <email address hidden>, Nikhil Bansal <email address hidden>
Cc: contrail-emea <email address hidden>, Contrail Systems Virtual Router Team <email address hidden>, support-private <email address hidden>, Federico Toci <email address hidden>, Rajakumar David <email address hidden>, Himanshu Bahukhandi <email address hidden>, JTAC-Contrail-Bng <email address hidden>, Mahesh Subramaniam <email address hidden>, Robert Horner <email address hidden>, Tom Nakano <email address hidden>, Yaoxuan Wang <email address hidden>, Soumit Mishra <email address hidden>, Alois Zellner <email address hidden>, Lin Wu <email address hidden>, "Selmane B. SLAMA" <email address hidden>, "Mohammed A. Hafez" <email address hidden>, Vlad Stoica <email address hidden>, Rakesh Dubey <email address hidden>, Ragunath Subramaniam <email address hidden>, Ravi Prakash <email address hidden>, Ajay Pandey <email address hidden>, Masum Mir <email address hidden>, Umesh Kukreja <email address hidden>, PrasannaKumar Anandan Gajendran <email address hidden>, Suryanarayana MNV <email address hidden>, Stephane Blazy <email address hidden>, Tony Sarathchandra <email address hidden>, "KK (Kaushal) Agrawal" <email address hidden>, Ashbi Jose <email address hidden>, Viswanath KJ <email address hidden>, Ali Ahmed Shakir <email address hidden>, Lester Andrade <email address hidden>, Soumit Mishra <email address hidden>, Rudra Rugge <email address hidden>, Sreelakshmi Sarva <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: Re: Wrong VRF and DHCP lease after re-creating VM / NEC/Juniper Engineering review of Etisalat VeCPE project 2018-0806-0353 / LP786240


Hi,



If compute is powered off, then control-node takes at most 90 seconds (xmpp hold-time) to delete those routes. During this time, those routes will remain and will be used for forwarding. I presume there is no GR enabled here. If so, then routes will remain for even longer time.



Regards,
Ananth

________________________________
From: Ashok Singh R
Sent: Monday, August 27, 2018 1:31:25 AM
To: Mladen Maric; Sivakumar Ganapathy; VinayKumar Tejavath; Assen Tarlov; Slobodan Blatnjak; Nikhil Bansal
Cc: contrail-emea; Contrail Systems Virtual Router Team; support-private; Federico Toci; Rajakumar David; Himanshu Bahukhandi; JTAC-Contrail-Bng; Mahesh Subramaniam; Robert Horner; Tom Nakano; Yaoxuan Wang; Soumit Mishra; Alois Zellner; Lin Wu; Selmane B. SLAMA; Mohammed A. Hafez; Vlad Stoica; Rakesh Dubey; Ragunath Subramaniam; Ravi Prakash; Ajay Pandey; Masum Mir; Umesh Kukreja; PrasannaKumar Anandan Gajendran; Suryanarayana MNV; Stephane Blazy; Tony Sarathchandra; KK (Kaushal) Agrawal; Ashbi Jose; Viswanath KJ; Ali Ahmed Shakir; Lester Andrade; Soumit Mishra; Rudra Rugge; Sreelakshmi Sarva; Jeba Paulaiyan
Subject: Re: Wrong VRF and DHCP lease after re-creating VM / NEC/Juniper Engineering review of Etisalat VeCPE project 2018-0806-0353 / LP786240


+Nikhil



Hi Vinay,



I am not sure if the steps that you have provided is recommended/supported way.  The reason is since we have shutdown the compute, the vrouter-agent which is supposed to process the delete request for VM, will not be able to do so. So vrouter-agent wouldn’t have sent any delete/unsubscribe message to control-node.



Have added Nikhil to explain behavior from control-node in this scenario.



The issue that I had looked at earlier as part of 1786240 was that in which port didn’t have any VRF. Here we have wrong VRF?



Regards,

Ashok





From: Mladen Maric <email address hidden>
Date: Monday, August 27, 2018 at 1:23 PM
To: Sivakumar Ganapathy <email address hidden>, VinayKumar Tejavath <email address hidden>, Ashok Singh R <email address hidden>, Assen Tarlov <email address hidden>, Slobodan Blatnjak <email address hidden>
Cc: contrail-emea <email address hidden>, Contrail Systems Virtual Router Team <email address hidden>, support-private <email address hidden>, Federico Toci <email address hidden>, Rajakumar David <email address hidden>, Himanshu Bahukhandi <email address hidden>, JTAC-Contrail-Bng <email address hidden>, Mahesh Subramaniam <email address hidden>, Robert Horner <email address hidden>, Tom Nakano <email address hidden>, Yaoxuan Wang <email address hidden>, Soumit Mishra <email address hidden>, Alois Zellner <email address hidden>, Lin Wu <email address hidden>, "Selmane B. SLAMA" <email address hidden>, "Mohammed A. Hafez" <email address hidden>, Vlad Stoica <email address hidden>, Rakesh Dubey <email address hidden>, Ragunath Subramaniam <email address hidden>, Ravi Prakash <email address hidden>, Ajay Pandey <email address hidden>, Masum Mir <email address hidden>, Umesh Kukreja <email address hidden>, PrasannaKumar Anandan Gajendran <email address hidden>, Suryanarayana MNV <email address hidden>, Stephane Blazy <email address hidden>, Tony Sarathchandra <email address hidden>, "KK (Kaushal) Agrawal" <email address hidden>, Ashbi Jose <email address hidden>, Viswanath KJ <email address hidden>, Ali Ahmed Shakir <email address hidden>, Lester Andrade <email address hidden>, Soumit Mishra <email address hidden>, Rudra Rugge <email address hidden>, Sreelakshmi Sarva <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: RE: Wrong VRF and DHCP lease after re-creating VM / NEC/Juniper Engineering review of Etisalat VeCPE project 2018-0806-0353 / LP786240



Hi all,

This seems to be (initial) development in progress (Partner testing their Orchestrator):



1. VNF ports are created separately to VMs
2. When orchestrator creates VM, it attaches existing interafaces to it

Our case was:
1. Forcefully poweroff compute node
2. Orchestrator starts VM healing. First it tries to use normal delete, if it fails it starts using force-delete and retries it until VM gets deleted (we use force-delete because normal delete usually doesn't help if VM is hosted on shutdowned compute node).
3. Once VM is deleted Orchestrator creates a new VM on another compute node and assignes old ports which were previosly assigned to the deleted VM.
4. When process is finished in some cases (may be 20-30%) created VM has such wrong VRF issue.



Based on available info, bug report has been marked as invalid:

https://bugs.launchpad.net/juniperopenstack/r4.1/+bug/1786240



Basically we concluded that this operation breaks connectivity between vrouter and control/config nodes and operation does not complete successfully – which is in line with operations customer is performing (force delete).



That bring next major question (BU/QA on CC, feedback from Partner Sales needed as well):

  1.  Has Netcracker Orchestrator been validated/certified on Contrail platform (from JTACs perspective, it’s a black box with unknown input/outputs)?
  2.  Are described operations from above supported at all? Does not look like it, as we have force delete involved.



Regards,

Mladen



From: Sivakumar Ganapathy
Sent: Sunday, August 26, 2018 2:22 PM
To: VinayKumar Tejavath <email address hidden>; Ashok Singh R <email address hidden>; Assen Tarlov <email address hidden>
Cc: contrail-emea <email address hidden>; Contrail Systems Virtual Router Team <email address hidden>; support-private <email address hidden>; Federico Toci <email address hidden>; Mladen Maric <email address hidden>; Rajakumar David <email address hidden>; Himanshu Bahukhandi <email address hidden>; JTAC-Contrail-Bng <email address hidden>
Subject: Re: Wrong VRF and DHCP lease after re-creating VM



Vinaykumar,

Ashok had already provided the analysis and in the last email thread it was concluded that the problem is due to control node connection going down. Assen had requested more information and don’t see that provided

Anywhere.



Assen,

Do reach out to Ashok once you have the info.



========
Hi vRouter-Team,



Netcracker escalated the case as they hit the issue again and want to know how to prevent that behavior as they are running ATP for customer.



From the case notes I see that Ashok found that  in previous case the problem was the connection to control node.



“I checked the gcore file. The port was in inactive state because configuration was not downloaded. The config was not downloaded because connection to control node was down at that moment.”



The following logs confirm when the connection to control-node was established



2018-08-07 20:31:40.587568 AgentXmppSession: peer = "10.237.212.136" event = "READY" tree_builder = "NULL" message = "BGP peer ready." file = "controller/src/vnsw/agent/controller/controller_peer.cc" line = 1513



2018-08-07 20:31:40.674994 AgentXmppSession: peer = "10.237.212.138" event = "READY" tree_builder = "NULL" message = "BGP peer ready." file = "controller/src/vnsw/agent/controller/controller_peer.cc" line = 1513

“



Is it possible to revisit this issue once again as part of latest Netcracker/Etisalat escalation ?



@Curtis

We will need documentation for the underlay topology, KVM VM details, server specification and monitoring data in order to evaluate this properly



==========



Thanks,

Sivakumar



From: VinayKumar Tejavath <<email address hidden><mailto:<email address hidden>>>
Date: Saturday, 25 August 2018 at 3:49 AM
To: Sivakumar Ganapathy <<email address hidden><mailto:<email address hidden>>>, Ashok Singh R <<email address hidden><mailto:<email address hidden>>>
Cc: contrail-emea <<email address hidden><mailto:<email address hidden>>>, Contrail Systems Virtual Router Team <<email address hidden><mailto:<email address hidden>>>, support-private <<email address hidden><mailto:<email address hidden>>>, Federico Toci <<email address hidden><mailto:<email address hidden>>>, Mladen Maric <<email address hidden><mailto:<email address hidden>>>, Rajakumar David <<email address hidden><mailto:<email address hidden>>>, Himanshu Bahukhandi <<email address hidden><mailto:<email address hidden>>>, JTAC-Contrail-Bng <<email address hidden><mailto:<email address hidden>>>
Subject: Wrong VRF and DHCP lease after re-creating VM



Hi Sivakumar / Ashok,



Customer escalated this case again and LP bug 1786240 already marked as hot-case. Latest logs are uploaded to the JTAC case and the same has been updated in LP bug comments.



Customer followed below steps to recreate the problem



1. Forcefully power off compute node

2. Orchestrator starts VM healing. First it tries to use normal delete, if it fails it starts using force-delete and retries it until VM gets deleted (we use force-delete because normal delete usually doesn't help if VM is hosted on shutdowned compute node).

3. Once VM is deleted Orchestrator creates a new VM on another compute node and assignes old ports which were previously assigned to the deleted VM.

4. When process is finished in some cases (may be 20-30%) created VM has such wrong VRF issue.



Customer will be available on Monday morning in Moscow time zone (10:00 AM) if any troubleshooting session required. I am copying BLR and EMEA contrail JTAC teams in this email who can coordinate session with customer.



--

Thanks & Regards,

VinayKumar Tejavath