vRouter: First packet of a flow ends up in a wrong destination

[Anand H. Krishnan]

The problem started manifesting in the form of delays in TCP connection establishment in a customer setup. After observations, it was reported that the first packet is being delivered to the wrong VM.

Observations:

Release 2.20

ping -c 1 <address> delivers the packet to the wrong VM but results in the correct flow entry.
ping -c 2 <address> results in the first packet being dropped.

Next-Hops were observed to be stable. The nature of the problem was intermittent.

Post L2 flow support, vRouter started using the label in the forwarding metadata to also store VXLAN identifier. For MPLS-O-X(GRE/UDP) packets, the label indicates the MPLS label of the packet. For VXLAN tunneled packets, the label indicates the VNID.

We store label in the forwarding metadata so that various sections of the code can make use of the label to add encapsulation data/lookup forwarding data when packets are cached. One such use of storing the label is to provide a way for the first packet that is held in the flow entry to find the destination. Label, logically, is not available for parsing at that point of time since we do flow lookup for the inner packet and the label would have been popped. When a packet is cached, we use flags to mark whether the label is a VNID or a MPLS label since there is no other way a cached packet can identify the label.

The logic that identifies this information has turned out to be buggy, resulting in VNID being identified as a MPLS label and thus sending the packet to a wrong interface/VM.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14635
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14635
Committed: http://github.org/Juniper/contrail-vrouter/commit/cccf1512922429fba8b687e2be2b1c9be64e0c45
Submitter: Zuul
Branch: R2.20

commit cccf1512922429fba8b687e2be2b1c9be64e0c45
Author: Anand H. Krishnan <email address hidden>
Date: Mon Oct 26 11:25:12 2015 +0530

Logic to identify the nature of label in fmd

Post L2 flow support, vRouter started using the label in the forwarding
metadata to also store VXLAN identifier. For MPLS-O-X(GRE/UDP) packets,
the label indicates the MPLS label of the packet. For VXLAN tunneled
packets, the label indicates the VNID.

The logic that identifies this information has turned out to be buggy,
resulting in VNID being identified as a MPLS label and thus sending the
packet to a wrong interface/VM. Till now, we used to mark the label as
VNID only if the packet had hit VRF translation nexthop (the vxlan kind)
AND if the packet was cached in the flow entry because of that
particular nexthop, which never happens and hence the label was always
identified as a MPLS label.

We now mark a flag indicating the type of the label, whenever the type
is identified. To be more explicit, the label is now set in the fmd with
an API that takes the type of the label, so that mistakes can be avoided.

Closes BUG: 1509939

Change-Id: I63167986b4532d6820b0c76f267c61ca788e8fd4

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/15727
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/15727
Committed: http://github.org/Juniper/contrail-vrouter/commit/bcbbf1c6fd0ca85b6fc21fc96795997d9b79bbf9
Submitter: Zuul
Branch: master

commit bcbbf1c6fd0ca85b6fc21fc96795997d9b79bbf9
Author: Anand H. Krishnan <email address hidden>
Date: Mon Oct 26 11:25:12 2015 +0530

Logic to identify the nature of label in fmd

Post L2 flow support, vRouter started using the label in the forwarding
metadata to also store VXLAN identifier. For MPLS-O-X(GRE/UDP) packets,
the label indicates the MPLS label of the packet. For VXLAN tunneled
packets, the label indicates the VNID.

The logic that identifies this information has turned out to be buggy,
resulting in VNID being identified as a MPLS label and thus sending the
packet to a wrong interface/VM. Till now, we used to mark the label as
VNID only if the packet had hit VRF translation nexthop (the vxlan kind)
AND if the packet was cached in the flow entry because of that
particular nexthop, which never happens and hence the label was always
identified as a MPLS label.

We now mark a flag indicating the type of the label, whenever the type
is identified. To be more explicit, the label is now set in the fmd with
an API that takes the type of the label, so that mistakes can be avoided.

Closes BUG: 1509939

Change-Id: I63167986b4532d6820b0c76f267c61ca788e8fd4

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/16028
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/16028
Committed: http://github.org/Juniper/contrail-vrouter/commit/25825c6e373824e9abefb9a7fa3e9a4306f3d24d
Submitter: Zuul
Branch: R2.21.x

commit 25825c6e373824e9abefb9a7fa3e9a4306f3d24d
Author: Anand H. Krishnan <email address hidden>
Date: Mon Oct 26 11:25:12 2015 +0530

Logic to identify the nature of label in fmd

Post L2 flow support, vRouter started using the label in the forwarding
metadata to also store VXLAN identifier. For MPLS-O-X(GRE/UDP) packets,
the label indicates the MPLS label of the packet. For VXLAN tunneled
packets, the label indicates the VNID.

The logic that identifies this information has turned out to be buggy,
resulting in VNID being identified as a MPLS label and thus sending the
packet to a wrong interface/VM. Till now, we used to mark the label as
VNID only if the packet had hit VRF translation nexthop (the vxlan kind)
AND if the packet was cached in the flow entry because of that
particular nexthop, which never happens and hence the label was always
identified as a MPLS label.

We now mark a flag indicating the type of the label, whenever the type
is identified. To be more explicit, the label is now set in the fmd with
an API that takes the type of the label, so that mistakes can be avoided.

Closes BUG: 1509939

Change-Id: I63167986b4532d6820b0c76f267c61ca788e8fd4