ECMP tuning not effective when packet is received from a remote compute

[richard roberts]

We have the following set-up:

Two Vns: vn_right and vn_left
Two VM: VM_left and VM_right on a same compute (COMPUTE1)
A SI in scaled out with two Instances on another compute (COMPUTE2)

 VM_left ---vn_left -------[SI_scaled_out -- Instance1 -- ] ----- vn_right ---- VM_right
                                          -- Instance2 --
 [ COMPUTE1 ]============== [ COMPUTE 2 ]===================[ COMPUTE1 ]

Appropriate policy to send traffic through the SI for Rigth to/from Left communication.

From a a data plane perpective,

VM_Left ===== MPLSoUDP ====== [SI]====== ===== MPLSoUDP ====== CM_right

Testing showed that tuning ECMP settings has no effect on how traffic is send toward the scaled-out SI.

Example of per VN setting (both on right and left), where only destination IP is taken into account:

        "ecmp_hashing_include_fields": {
            "destination_ip": true, ############## only field set as true #####
            "destination_port": false,
            "hashing_configured": true,
            "ip_protocol": false,
            "source_ip": false,
            "source_port": false
        },

The problem also occurs when LB is modified on a PE port basis.

Having a same a set of flows with same IP SRC/dst and different ports, showed that traffic is still load balanced accross both instances of the scaled-out SI.

In the below trace: src = 5.5.5.3 / dst = 4.4.4.3

compute 1 vn left (Destination 4.4.4.3)

[root@compute1 qemu(keystone_admin)]# rt --get 4.4.4.3/32 --vrf 1
Match 4.4.4.3/32 in vRouter inet4 table 0/1/unicast

Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP
vRouter inet4 routing table 0/1/unicast
Destination PPL Flags Label Nexthop Stitched MAC(Index)
4.4.4.3/32 0 LP 40 20 -

[root@compute1 qemu(keystone_admin)]# nh --get 20
Id:20 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:194 Vrf:0
              Flags:Valid, MPLSoUDP,
              Oif:0 Len:14 Flags Valid, MPLSoUDP, Data:18 66 da 5e 5a c3 b8 2a 72 d6 8b 19 08 00
              Vrf:0 Sip:200.0.0.1 Dip:200.0.0.2

========== on compute 2 (packet received with MPLS label 40)

[root@compute2 ~]# mpls --get 40
MPLS Input Label Map

   Label NextHop
-------------------
      40 98
[root@compute2 ~]#

packets is popped/swapped toward the ECMP NH made up of both instances. In this case ECMP is not based on selected 5-tuples criterias.

[root@compute2 ~]# nh --get 98
Id:98 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:2
              Flags:Valid, Policy, Ecmp,
              Sub NH(label): 116(57) 81(53)

We can check that IP flows will follow the same logic (e.g. NH of flow is the ECMP NH = 98)

Listing flows matching ([4.4.4.3]:*)

    Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
    23756<=>351916 5.5.5.3:0 17 (2->1)
                         4.4.4.3:0
(Gen: 10, K(nh):98, Action:F, Flags:, E:0, S(nh):20, Stats:32745/1506270, SPort 53135)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31055
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31055
Committed: http://github.com/Juniper/contrail-controller/commit/1ad60e63baedef3eb66e02132e31fd64eca5f4f0
Submitter: Zuul (<email address hidden>)
Branch: master

commit 1ad60e63baedef3eb66e02132e31fd64eca5f4f0
Author: Manish <email address hidden>
Date: Fri May 5 19:32:52 2017 +0530

ECMP load balance is not effective on remote SI compute.

Problem:
On SI compute routes are imported from CN with ECMP NH.
These routes were not notifying change in load balance params if changed.

Solution:
Notify on change in ecmp hash fields.

Note:
Any change in hash fields gets applied to new flows. Existing flows continue to
use same index as computed before.

Change-Id: I0216e12ba90b152c58deaaf67b477876eddfc9e8
Closes-bug: #1643842

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31185
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31186
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31185
Committed: http://github.com/Juniper/contrail-controller/commit/7486c3debb8837de4625a0d4902e743108e1d4d2
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 7486c3debb8837de4625a0d4902e743108e1d4d2
Author: Manish <email address hidden>
Date: Fri May 5 19:32:52 2017 +0530

ECMP load balance is not effective on remote SI compute.

Problem:
On SI compute routes are imported from CN with ECMP NH.
These routes were not notifying change in load balance params if changed.

Solution:
Notify on change in ecmp hash fields.

Note:
Any change in hash fields gets applied to new flows. Existing flows continue to
use same index as computed before.

Closes-bug: #1643842

Conflicts:
 src/vnsw/agent/controller/controller_route_path.cc
Change-Id: I0216e12ba90b152c58deaaf67b477876eddfc9e8

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31186
Committed: http://github.com/Juniper/contrail-controller/commit/3a1022da847bffc4069290acc97bd75c4b629263
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 3a1022da847bffc4069290acc97bd75c4b629263
Author: Manish <email address hidden>
Date: Fri May 5 19:32:52 2017 +0530

ECMP load balance is not effective on remote SI compute.

Problem:
On SI compute routes are imported from CN with ECMP NH.
These routes were not notifying change in load balance params if changed.

Solution:
Notify on change in ecmp hash fields.

Note:
Any change in hash fields gets applied to new flows. Existing flows continue to
use same index as computed before.

Closes-bug: #1643842

Conflicts:
 src/vnsw/agent/controller/controller_route_path.cc

Conflicts:
 src/vnsw/agent/controller/controller_route_path.cc
Change-Id: I0216e12ba90b152c58deaaf67b477876eddfc9e8

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/32573
Submitter: Vinay Vithal Mahuli (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/32573
Committed: http://github.com/Juniper/contrail-controller/commit/b14290bf497212c4e1629f6ccbfaaf1da637a082
Submitter: Zuul (<email address hidden>)
Branch: R3.2.3.x

commit b14290bf497212c4e1629f6ccbfaaf1da637a082
Author: Manish <email address hidden>
Date: Fri May 5 19:32:52 2017 +0530

ECMP load balance is not effective on remote SI compute.

Problem:
On SI compute routes are imported from CN with ECMP NH.
These routes were not notifying change in load balance params if changed.

Solution:
Notify on change in ecmp hash fields.

Note:
Any change in hash fields gets applied to new flows. Existing flows continue to
use same index as computed before.

Closes-bug: #1643842

Conflicts:
 src/vnsw/agent/controller/controller_route_path.cc
Change-Id: I0216e12ba90b152c58deaaf67b477876eddfc9e8