[vrouter] bad chksum on IPSEC UDP packets

[eon]

Observed on R2.21 / R3.2

We are trying to setup IPsec tunnels from contrail to other environments.

When using a FIP the vrouter is adding an incorrect UDP cksum and the packet is drop at the other end.

The VM sends packets without chksums:

tcpdump: WARNING: tap96f6ee93-e3: no IPv4 address assigned
tcpdump: listening on tap96f6ee93-e3, link-type EN10MB (Ethernet), capture size 65535 bytes
15:17:11.783321 IP (tos 0x0, ttl 64, id 7755, offset 0, flags [DF], proto UDP (17), length 160)
    172.10.1.3.4500 > 90.84.46.223.4500: [no cksum] UDP-encap: ESP(spi=0xc77ae455,seq=0x1be), length 132

On the physical interface we can see the bad cksum:

tcpdump: WARNING: bond0.1002: no IPv4 address assigned
tcpdump: listening on bond0.1002, link-type EN10MB (Ethernet), capture size 65535 bytes
15:18:07.223355 IP (tos 0x0, ttl 64, id 15097, offset 0, flags [none], proto GRE (47), length 188)
    10.12.129.65 > 10.128.0.251: GREv0, Flags [none], length 168
        MPLS (label 17, exp 0, [S], ttl 63)
        IP (tos 0x0, ttl 63, id 15097, offset 0, flags [DF], proto UDP (17), length 160)
    84.39.34.120.4500 > 90.84.46.223.4500: [bad udp cksum 0x366e -> 0x062e!] UDP-encap: ESP(spi=0xc77ae455,seq=0x1f5), length 132

Tx chksum offload is disabled on the host:

Features for bond0.1002:
rx-checksumming: off [fixed]
tx-checksumming: off
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: off
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]

This seems to only happen when the vrouter is doing N(S)/N(D) on the flow.

If we add the FIP on the guest interface, the vrouter doesn't NAT the packets and no chksum is added.

If we use a SNAT service instance, no chksum is added also.

[eon (eon-5)]

Issue is present on master also

No Tc on interfaces:

[cloud@devstack:~] 39s $ sudo vif --list
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
       Proxy=MAC Requests Proxied Always, Er=Etree Root, Mn=Mirror without Vlan Tag, Ig=Igmp Trap Enabled

vif0/0 OS: ens3
            Type:Physical HWaddr:02:17:78:da:90:6c IPaddr:0.0.0.0
            Vrf:0 Mcast Vrf:65535 Flags:L3L2VpEr QOS:-1 Ref:3
            RX packets:11900 bytes:4736349 errors:0
            TX packets:13496 bytes:5594197 errors:0
            Drops:0

vif0/1 OS: vhost0
            Type:Host HWaddr:02:17:78:da:90:6c IPaddr:192.168.40.11
            Vrf:0 Mcast Vrf:65535 Flags:L3DEr QOS:-1 Ref:8
            RX packets:13633 bytes:5610074 errors:0
            TX packets:12465 bytes:4769656 errors:0
            Drops:0

vif0/2 OS: pkt0
            Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0.0.0.0
            Vrf:65535 Mcast Vrf:65535 Flags:L3Er QOS:-1 Ref:3
            RX packets:2439 bytes:223920 errors:4
            TX packets:1729 bytes:157545 errors:0
            Drops:8

vif0/6 OS: tap7b38c53a-7c
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.24.0.4
            Vrf:4 Mcast Vrf:4 Flags:PL3L2DEr QOS:-1 Ref:6
            RX packets:2583 bytes:346981 errors:0
            TX packets:5795 bytes:4184638 errors:0
            ISID: 0 Bmac: 02:7b:38:c5:3a:7c
            Drops:15

No cksum from the VM, bad after the NAT

[cloud@devstack:~] $ sudo tcpdump -vv -i any -n port 4500
13:44:42.423143 IP (tos 0x0, ttl 64, id 32615, offset 0, flags [DF], proto UDP (17), length 160)
    10.24.0.4.4500 > 90.84.46.223.1024: [no cksum] UDP-encap: ESP(spi=0xc863117d,seq=0xef), length 132
13:44:42.423143 IP (tos 0x0, ttl 63, id 32615, offset 0, flags [DF], proto UDP (17), length 160)
    172.24.4.8.4500 > 90.84.46.223.1024: [bad udp cksum 0x59fb -> 0x1a5f!] UDP-encap: ESP(spi=0xc863117d,seq=0xef), length 132

[eon (eon-5)]

After some more investigation it appears that the vrouter is putting bad checksums during the NAT process when the packets comming from the VM have no chksum.

Taking the following setup:

VM1 (contrail 3.2) -> PHY1 -> IPsec tunnel -> PHY2 -> VM2 (external)

When looking at packets on the physical interface (PHY1), isakmp packets have a correct cksum, so the tunnel establishement is fine:

    84.39.63.145.4500 > 90.84.47.18.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000001 cookie eb0126d8cd3e7a7d->3255d616d467404a: child_sa ikev2_auth[I]:
    (v2e: len=444)

But when traffic occurs on the tunnel the cksums are bad:

    84.39.63.145.4500 > 90.84.47.18.4500: [bad udp cksum 0x1955 -> 0xed65!] UDP-encap: ESP(spi=0xc9c3730a,seq=0x1), length 132

The packets are dropped VM2 side. (netstat -su)

From the VM1 tap we can see that UDP ESP packets have no checksums:

tcpdump: listening on tap47f2864d-a7, link-type EN10MB (Ethernet), capture size 262144 bytes
07:59:34.915688 IP (tos 0x0, ttl 64, id 2471, offset 0, flags [DF], proto UDP (17), length 160)
    172.10.1.3.4500 > 90.84.47.18.4500: [no cksum] UDP-encap: ESP(spi=0xc9c3730a,seq=0x9), length 132

With the following patch in the vrouter module, we make the vrouter not change the cksum when packets have no checksum

diff --git a/dp-core/vr_proto_ip.c b/dp-core/vr_proto_ip.c
index b96fce3..d12d284 100644
--- a/dp-core/vr_proto_ip.c
+++ b/dp-core/vr_proto_ip.c
@@ -106,6 +106,9 @@ vr_ip_update_csum(struct vr_packet *pkt, unsigned int ip_inc, unsigned int inc)
     } else if (ip->ip_proto == VR_IP_PROTO_UDP) {
         udp = (struct vr_udp *)((unsigned char *)ip + ip->ip_hl * 4);
         csump = &udp->udp_csum;
+ if (*csump == 0) {
+ return;
+ }
     } else {
         return;
     }

With this patch, the tunnel establishment is ok as well as the traffic inside the tunnel. No bad cksum is set by the vrouter.

[eon (eon-5)]

Adding some captures of the tunnel establishment and ping inside the tunnel.

tx offloading is enabled inside VM1.

Without the patch:
- ovh.tap.pcap (output of VM1 tap)
- ovh.pcap (output of PHY1)

With the patch:
- ovh.new2.tap.pcap (output of VM1 tap)
- ovh.new2.pcap (output of PHY1)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/44814
Submitter: Jean-Philippe Braun (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/44814
Committed: http://github.com/Juniper/contrail-vrouter/commit/393c57f46ab587d198180f8a9e35d570a9111538
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 393c57f46ab587d198180f8a9e35d570a9111538
Author: Jean-Philippe Braun <email address hidden>
Date: Mon Jul 16 10:26:28 2018 +0200

Don't do any chksum calculation if there isn't any chksum

Change-Id: I9eae8b8c7fcc1f82c75bfa57c3a0da9e7e78ad22
Closes-Bug: #1780060

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/45046
Submitter: Jean-Philippe Braun (<email address hidden>)