vcenter-only-mode:Across subnet traffic getting disrupted for 30-40 sec when vm migrates to different esxi host

[Sandip Dey]

build:3.1.1.0-45~vcenter

vcenter only setup

For vcenter only mode, the agent replies arp with the physical intf mac address for the VN's gateway ip.

When vm migrates to a different esxi host as part of vmotion,vcenter plugin binds it to the new contrail-compute-vm,whose physical intf mac is different. But the vm still hold the old mac address for the gateway ip in the arp table.

Until the old mac address gets flushed and the new mac is learnt for the gateway ip, the traffic is lost from the vm for a different subnet.

Showed the problem to Naveen/Hari.

[kratna (kratna)]

This issue was found in IT CCP ESX cloud:

VM ( part of VN1 contrail network ) was migrated from one host esxi-02 to esxi-14 . The host esxi-14 did not have any VMs that had interfaces in VN1 .

We did 2 iterations :

Iteration 1 :- downtime was 31.4 seconds

a) to program the VRF + flow - 2.5 s
b) ping to resume - 28.9 s

Iteration 2: downtime was 26.4 seconds

a) program the VRF + flow = 2.0 s
b) ping to resume - 24.4 s

I used ping to check these times - I was sending ping every 0.1 second . Before the migration the ping was taking rtt = ~34 milliseconds .

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27833
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27904
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27905
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27833
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27905
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27904
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27833
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27904
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27905
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27904
Committed: http://github.org/Juniper/contrail-controller/commit/c537a30a392a250ce728f5d5d4a7d57f89618124
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit c537a30a392a250ce728f5d5d4a7d57f89618124
Author: jayaramsatya <email address hidden>
Date: Wed Jan 11 19:33:37 2017 +0530

Problem:
For vcenter only mode, the agent replies arp with the physical intf mac address for the VN's gateway ip.
When vm migrates to a different esxi host as part of vmotion,vcenter plugin binds it to the new contrail-compute-vm,whose physical intf mac is different. But the vm still hold the old mac address for the gateway ip in the arp table.
Until the old mac address gets flushed and the new mac is learnt for the gateway ip, the traffic is lost from the vm for a different subnet.
Solution:
Whenever VM migrates to new esxi host. and upon creation of VM interface
Arp module will listen on route update. after getting the route update
send the gratious arp for the Gateway IP.
closes-bug:#1650219

Change-Id: I6bf4bab98c956bead866d22b7c8e8960419315b4

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27833
Committed: http://github.org/Juniper/contrail-controller/commit/65bfda4275e660c80805121fb0ebbdc07a88b52f
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 65bfda4275e660c80805121fb0ebbdc07a88b52f
Author: jayaramsatya <email address hidden>
Date: Wed Jan 11 19:33:37 2017 +0530

Problem:
For vcenter only mode, the agent replies arp with the physical intf mac address for the VN's gateway ip.
When vm migrates to a different esxi host as part of vmotion,vcenter plugin binds it to the new contrail-compute-vm,whose physical intf mac is different. But the vm still hold the old mac address for the gateway ip in the arp table.
Until the old mac address gets flushed and the new mac is learnt for the gateway ip, the traffic is lost from the vm for a different subnet.
Solution:
Whenever VM migrates to new esxi host. and upon creation of VM interface
Arp module will listen on route update. after getting the route update
send the gratious arp for the Gateway IP.
closes-bug:#1650219

Change-Id: I6bf4bab98c956bead866d22b7c8e8960419315b4

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27905
Committed: http://github.org/Juniper/contrail-controller/commit/5912a8bf417658f53d7878cc800b14ade7800f64
Submitter: Zuul (<email address hidden>)
Branch: master

commit 5912a8bf417658f53d7878cc800b14ade7800f64
Author: jayaramsatya <email address hidden>
Date: Wed Jan 11 19:33:37 2017 +0530

Problem:
For vcenter only mode, the agent replies arp with the physical intf mac address for the VN's gateway ip.
When vm migrates to a different esxi host as part of vmotion,vcenter plugin binds it to the new contrail-compute-vm,whose physical intf mac is different. But the vm still hold the old mac address for the gateway ip in the arp table.
Until the old mac address gets flushed and the new mac is learnt for the gateway ip, the traffic is lost from the vm for a different subnet.
Solution:
Whenever VM migrates to new esxi host. and upon creation of VM interface
Arp module will listen on route update. after getting the route update
send the gratious arp for the Gateway IP.
closes-bug:#1650219

Change-Id: I6bf4bab98c956bead866d22b7c8e8960419315b4

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27986
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27986
Committed: http://github.org/Juniper/contrail-controller/commit/3192a79712e8e2c85e2faf50005b3631bb6685e6
Submitter: Zuul (<email address hidden>)
Branch: R3.1.1.x

commit 3192a79712e8e2c85e2faf50005b3631bb6685e6
Author: jayaramsatya <email address hidden>
Date: Wed Jan 11 19:33:37 2017 +0530

Problem:
For vcenter only mode, the agent replies arp with the physical intf mac address for the VN's gateway ip.
When vm migrates to a different esxi host as part of vmotion,vcenter plugin binds it to the new contrail-compute-vm,whose physical intf mac is different. But the vm still hold the old mac address for the gateway ip in the arp table.
Until the old mac address gets flushed and the new mac is learnt for the gateway ip, the traffic is lost from the vm for a different subnet.
Solution:
Whenever VM migrates to new esxi host. and upon creation of VM interface
Arp module will listen on route update. after getting the route update
send the gratious arp for the Gateway IP.
closes-bug:#1650219

Change-Id: I6bf4bab98c956bead866d22b7c8e8960419315b4
(cherry picked from commit 5912a8bf417658f53d7878cc800b14ade7800f64)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28284
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28285
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28286
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28287
Submitter: jayaramsatya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28285
Committed: http://github.org/Juniper/contrail-controller/commit/ae3a2ee8ab31b91e186deeb505f3b3bed02dbd89
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit ae3a2ee8ab31b91e186deeb505f3b3bed02dbd89
Author: jayaramsatya <email address hidden>
Date: Mon Jan 30 18:07:13 2017 +0530

Interface Mac is Wrongly given instead of vm_mac.
closes-bug:#1650219

Change-Id: I8ad0e093ecf9255852e37c3c835e65d764fb70de

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28286
Committed: http://github.org/Juniper/contrail-controller/commit/2f406692881c2f39ddd0d39baa15c6bef210535b
Submitter: Zuul (<email address hidden>)
Branch: R3.1.1.x

commit 2f406692881c2f39ddd0d39baa15c6bef210535b
Author: jayaramsatya <email address hidden>
Date: Mon Jan 30 18:07:13 2017 +0530

Interface Mac is Wrongly given instead of vm_mac.
closes-bug:#1650219

Change-Id: I8ad0e093ecf9255852e37c3c835e65d764fb70de

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28287
Committed: http://github.org/Juniper/contrail-controller/commit/1aed8495e589dec3fe41d236896b6fc781048e09
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 1aed8495e589dec3fe41d236896b6fc781048e09
Author: jayaramsatya <email address hidden>
Date: Mon Jan 30 18:07:13 2017 +0530

Interface Mac is Wrongly given instead of vm_mac.
closes-bug:#1650219

Change-Id: I8ad0e093ecf9255852e37c3c835e65d764fb70de

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28284
Committed: http://github.org/Juniper/contrail-controller/commit/6d048b84075473156b6269b7a3e3226ab816f5c6
Submitter: Zuul (<email address hidden>)
Branch: master

commit 6d048b84075473156b6269b7a3e3226ab816f5c6
Author: jayaramsatya <email address hidden>
Date: Mon Jan 30 18:07:13 2017 +0530

Interface Mac is Wrongly given instead of vm_mac.
closes-bug:#1650219

Change-Id: I8ad0e093ecf9255852e37c3c835e65d764fb70de