TCP/UDP traffic fails on a nested compute setup if encap is MPLSoGRE in overlay

[Vedamurthy Joshi]

Was found in Reliance virtual cluster setup. Build 1.05 211

Two VMs were launched in two virtual compute-nodes in a cluster. Vrouter encap mode was MPLSoGRE
Ping between the VMs passed, but ssh traffic failed.

Divakar debugged the issue and found that the protocol field in the TCP syn packet was getting corrupted (overwritten by CRC value) and thus the packet reaching the VM was corrupted.

On the sender overlay compute node, packet as seen on fabric :

15:46:41.720940 IP (tos 0x0, ttl 64, id 140, offset 0, flags [none], proto GRE (47), length 88)
    10.0.0.252 > 10.0.0.253: GREv0, Flags [none], length 68
        MPLS (label 16, exp 0, [S], ttl 64)
        IP (tos 0x0, ttl 64, id 33008, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.253.50880 > 192.168.1.252.22: Flags [S], cksum 0x8578 (incorrect -> 0xa503), seq 2364212271, win 14600, options [mss 1460,sackOK,TS val 18844572 ecr 0,nop,wscale 5], length 0
        0x0000: 0000 5e00 0100 0298 e441 3842 0800 4500
        0x0010: 0058 008c 0000 402f 63f3 0a00 00fc 0a00
        0x0020: 00fd 0000 8847 0001 0140 4500 003c 80f0
        0x0030: 4000 4006 3482 c0a8 01fd c0a8 01fc c6c0
        0x0040: 0016 8ceb 042f 0000 0000 a002 3908 8578
        0x0050: 0000 0204 05b4 0402 080a 011f 8b9c 0000
        0x0060: 0000 0103 0305

On the receiver overlay compute node, packet as seen on fabric :

15:46:41.720301 IP (tos 0x0, ttl 64, id 140, offset 0, flags [none], proto GRE (47), length 88)
    10.0.0.252 > 10.0.0.253: GREv0, Flags [none], length 68
        MPLS (label 16, exp 0, [S], ttl 64)
        IP (tos 0x0, ttl 27, id 33008, offset 0, flags [DF], proto unknown (123), length 60, bad cksum 3482 (->590d)!)
    192.168.1.253 > 192.168.1.252: ip-proto-123 40
        0x0000: 0255 8791 4893 ac16 2db3 2b60 0800 4500
        0x0010: 0058 008c 0000 402f 63f3 0a00 00fc 0a00
        0x0020: 00fd 0000 8847 0001 0140 4500 003c 80f0
        0x0030: 4000 1b7b 3482 c0a8 01fd c0a8 01fc c6c0
        0x0040: 0016 8ceb 042f 0000 0000 a002 3908 8578
        0x0050: 0000 0204 05b4 0402 080a 011f 8b9c 0000
        0x0060: 0000 0103 0305