ovs bridge breaking TCP between two virtio net devices when checksum offload on

[bugproxy]

== Comment: #0 - Gowrishankar Muthukrishnan - 2016-09-29 13:06:10 ==
---Problem Description---

Two kvm guests, each booted with a vnic (virtio_net) in ovs bridge in host seem
to fail in establishing TCP when virtio turned on for tcp checksum offload.
Where as, these guests are able to complete TCP when vnics are over linux
bridge in host.

tcpdump on destination guest that confirms SYN retransmission, in OVS case:

12:21:56.588835 52:54:c1:a2:7c:63 > 52:54:c1:a2:7c:53, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 49483, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.126.3.51944 > 10.0.126.2.22: Flags [S], cksum 0x1034 (incorrect -> 0x0482), seq 621922575, win 29200, options [mss 1460,sackOK,TS val 329796 ecr 0,nop,wscale 7], length 0
12:21:57.588220 52:54:c1:a2:7c:63 > 52:54:c1:a2:7c:53, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 49484, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.126.3.51944 > 10.0.126.2.22: Flags [S], cksum 0x1034 (incorrect -> 0x0388), seq 621922575, win 29200, options [mss 1460,sackOK,TS val 330046 ecr 0,nop,wscale 7], length 0

tcpdump on destination guest that confirms SYN/SYN-ACK/ACK , in bridge case:

12:34:53.140513 52:54:c1:a2:7c:64 > 52:54:c1:a2:7c:54, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 57464, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.126.3.41722 > 10.1.126.2.22: Flags [S], cksum 0x1036 (incorrect -> 0xb8a4), seq 1286151909, win 29200, options [mss 1460,sackOK,TS val 523933 ecr 0,nop,wscale 7], length 0
12:34:53.140729 52:54:c1:a2:7c:54 > 52:54:c1:a2:7c:64, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.126.2.22 > 10.1.126.3.41722: Flags [S.], cksum 0x1036 (incorrect -> 0xe19a), seq 3214677586, ack 1286151910, win 28960, options [mss 1460,sackOK,TS val 524786 ecr 523933,nop,wscale 7], length 0
12:34:53.141253 52:54:c1:a2:7c:64 > 52:54:c1:a2:7c:54, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 57465, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.126.3.41722 > 10.1.126.2.22: Flags [.], cksum 0x102e (incorrect -> 0x80a1), ack 1, win 229, options [nop,nop,TS val 523934 ecr 524786], length 0

---uname output---
Linux ip9-114-219-126gate 4.4.0-9136-generic #55-Ubuntu SMP Fri Aug 26 05:56:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = P8 LE

---Steps to Reproduce---
 1. Create a ovs bridge "ovsbr0"
2. Create a kvm guest (preferably same as host - Ubuntu 16.10) carrying a vnic
   (as shown below) in ovsbr0

   <interface type='bridge'>
      <mac address='52:54:c1:a2:7c:53'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'/>
      <model type='virtio'/>
   </interface>

   Similarly create another kvm guest but with vnic of different mac address.

3. Assign a static IP over vnic inside each guest.
   For eg, if enp0s3 is vnic:
   sudo ifconfig enp0s3 10.0.126.2/24 up (in guest 1)
   sudo ifconfig enp0s3 10.0.126.3/24 up (in guest 2)

4. Ping and ensure ARP completes.
   ping 10.0.126.3 (from guest 2)

5. Check tcp scum offload ON as by default.
   ethtool -k enp0s3

   Features for enp0s3:
    rx-checksumming: on [fixed]
    tx-checksumming: on
 tx-checksum-ipv4: off [fixed]
 tx-checksum-ip-generic: on
 tx-checksum-ipv6: off [fixed]
 tx-checksum-fcoe-crc: off [fixed]
 tx-checksum-sctp: off [fixed]

6. Check if TCP establishes between two guests on above IPs, using nc (netcat).
   nc 10.0.126.2 22

   As you would not see "SSH-2.0-OpenSSH_7.3p1 Ubuntu-1" as response, this
   confirms TCP over port did not establish.

   You may run tcpdump on each port while running nc to confirm invalid
   checksum received in tcp (as shown in problem description).

[Christian Ehrhardt  (paelzer)]

Hi,
I've had a very similar setup avail atm.
Running that in the guest gave me:

ethtool -k ens4' | grep check

rx-checksumming: on [fixed]
tx-checksumming: off
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]

Just to clarify the default is "tx-checksumming: off" right - is it on for you by default?

Note: Guest and Host are Yakkety for me.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-09-30 02:40 EDT-------
Irrespective of default csum on/off, having csum=on breaks tcp as checksum_unnecessary / relevant protocol header flag not passed in higher stack , an so tcp eventually drop syn. Could you please check with tx=on enforced and see if tcp completes 3way ?.

[Christian Ehrhardt  (paelzer)]

Ok, after discussing on IRC a bit more info.

my "normal bridge" backed device is tx-checksumming: on by default and work
my "ovs-dpdk" backed devices are tx-checksumming: off by default and work
my "normal ovs" backed devices are tx-checksumming: on by default and work as well

In E/N/I my interfaces only have (ens3 is the bridge backed one, and ens4 the one with an OVS of some kind):
auto ens3
iface ens3 inet dhcp
auto ens4
iface ens4 inet dhcp
So all defaults.

Versions for me:
libvirt-bin 2.1.0-1ubuntu6
qemu-kvm 1:2.6.1+dfsg-0ubuntu4

Sharing my network definitions for those:

Normal one:
    <interface type='network'>
      <mac address='52:54:00:3e:4b:48'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

OVS-DPDK one
    <interface type='vhostuser'>
      <mac address='52:54:00:99:f7:a4'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user-2' mode='client'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>

With OVS-DPDK-multiqueue
    <interface type='vhostuser'>
      <mac address='52:54:00:d1:7e:cc'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user-2' mode='client'/>
      <model type='virtio'/>
      <driver queues='4'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface

With "pure" OVS checksumming:
    <interface type='bridge'>
      <mac address='52:54:00:b4:b4:1c'/>
      <source network='ovsbr0' bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='1c486148-a9ea-4000-8cfa-5fc9ce591f52'/>
      </virtualport>
      <target dev='vnet3'/>
      <model type='virtio'/>
      <driver name='vhost' queues='4'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>

The last should be your case, but with tx-checksumming: on it is working just nice.
I've run iperf, netperf and uperf over that connection all good.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-10-03 10:23 EDT-------
Interesting. I ensured that, ovs and kernel upto date, but of no luck. OVS backed virtual nic still fail to establish tcp with tx on. Here is xml for this port:

<interface type='bridge'>
<mac address='52:54:c1:a2:7c:53'/>
<source bridge='ovsbr0'/>
<virtualport type='openvswitch'>
<parameters interfaceid='6b0499e1-42d5-4afa-ae66-9d403aba30ac'/>
</virtualport>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

Wrt your xml, I don't think having a dhcp network as you have is anything of concern for the root cause as it is about csum that breaks the session.

pkg info:
openvswitch-common 2.6.0-0ubuntu1
linux-generic 4.8.0.17.27

[bugproxy (bugproxy)]

Default Comment by Bridge

[Manoj Iyer (manjo)]

Assigning to carlos for comments.

[Manoj Iyer (manjo)]

Based on Carlos's comments I am assigning this to James:

Hi,

I looked at it and it seems that James triaged it. He is the openvswitch expert in the team and the bug should be assigned to him. I don't seem to have the privy to change the bug...

I'll talk to James about when he gets back from the charmer summit this week.

Thanks,
-Carlos

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-08-07 08:12 EDT-------
Hi James,

Any updates on this one?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-08-08 05:28 EDT-------
This bug is reproducible with ubuntu 17.04 (host and guests of same version). Disabling tx csum is the only work around as observed earlier.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-08-31 17:44 EDT-------
Updating release and target information to reflect that this is still an issue as of 17.04.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-27 02:09 EDT-------
With ubuntu 17.10 as kvm hypervisor, I could not reproduce this problem, so it seems to be fixed.

Below is the default offload configurations for the virtio-net nic.

dev@zesty:~$ ethtool -k enp0s3
Features for enp0s3:
rx-checksumming: on [fixed]
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on

Here is an establishing session packet dump:

01:53:24.475995 52:54:c1:a2:7c:53 > 52:54:c1:a2:7c:54, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 1919, offset 0, flags [DF], proto TCP (6), length 60)
10.0.129.3.46878 > 10.0.129.4.22: Flags [S], cksum 0xe187 (correct), seq 4289434639, win 29200, options [mss 1460,sackOK,TS val 4294939504 ecr 0,nop,wscale 7], length 0

01:53:24.476020 52:54:c1:a2:7c:54 > 52:54:c1:a2:7c:53, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.0.129.4.22 > 10.0.129.3.46878: Flags [S.], cksum 0x1636 (incorrect -> 0xd213), seq 1650342985, ack 4289434640, win 28960, options [mss 1460,sackOK,TS val 4294930859 ecr 4294939504,nop,wscale 7], length 0

01:53:24.476350 52:54:c1:a2:7c:53 > 52:54:c1:a2:7c:54, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 1920, offset 0, flags [DF], proto TCP (6), length 52)
10.0.129.3.46878 > 10.0.129.4.22: Flags [.], cksum 0x162e (incorrect -> 0x711b), ack 1, win 229, options [nop,nop,TS val 4294939504 ecr 4294930859], length 0

01:53:24.492529 52:54:c1:a2:7c:54 > 52:54:c1:a2:7c:53, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 64, id 19942, offset 0, flags [DF], proto TCP (6), length 83)
10.0.129.4.22 > 10.0.129.3.46878: Flags [P.], cksum 0x164d (incorrect -> 0x0f49), seq 1:32, ack 1, win 227, options [nop,nop,TS val 4294930864 ecr 4294939504], length 31

I think, CHECKSUM_UNNECESSARY is being handled correctly for tcp syn, but I do nt have stap set currently in my env. As session gets running without any error, we can close this bug.

[Launchpad Janitor (janitor)]

[Expired for openvswitch (Ubuntu) because there has been no activity for 60 days.]