From default scenario (TSO and GRO turned ON on all devices) we were trying to change TSO and GRO on different devices and measuring performance.
Changes against default / Test result Iperf TCP default test
Default deployment 22.8 Kbits/sec
GRO off on controller private NICs(bond slaves) 1.80 Gbits/sec
TSO off on controller public interface 5.19 Gbits/sec
The only place where we are having an MTU mismatch is FIP namespace of destination compute node.
root@node-10:~# ip netns e fip-54905d4f-9b66-4c2e-ab9d-4105ea459fed ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
4: fpr-5faaca01-9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 8e:9b:95:a4:f8:1f brd ff:ff:ff:ff:ff:ff
30: fg-1336f4fb-cb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether fa:16:3e:ba:a7:cb brd ff:ff:ff:ff:ff:ff
I was dumping traffic for TSO on/off on both (destination compute and controller) public interfaces. Just to see the difference:
TSO ON (low speed)
Controller:
06:04:56.915792 IP (tos 0x0, ttl 62, id 29440, offset 0, flags [DF], proto TCP (6), length 1450)
172.20.156.140.53025 > 172.20.156.138.5001: Flags [.], cksum 0x369f (correct), seq 1398:2796, ack 1, win 441, options [nop,nop,TS val 19235505 ecr 30734267], length 1398
06:04:56.916241 IP (tos 0x0, ttl 62, id 48079, offset 0, flags [DF], proto TCP (6), length 64)
172.20.156.138.5001 > 172.20.156.140.53025: Flags [.], cksum 0x9aba (correct), seq 1, ack 2796, win 699, options [nop,nop,TS val 30734601 ecr 19235505,nop,nop,sack 1 {12582:15378}], length 0
06:04:56.916660 IP (tos 0x0, ttl 62, id 29441, offset 0, flags [DF], proto TCP (6), length 1450)
172.20.156.140.53025 > 172.20.156.138.5001: Flags [.], cksum 0xf9ce (correct), seq 18174:19572, ack 1, win 441, options [nop,nop,TS val 19235505 ecr 30734601], length 1398
06:04:56.916754 IP (tos 0x0, ttl 62, id 29442, offset 0, flags [DF], proto TCP (6), length 1450)
172.20.156.140.53025 > 172.20.156.138.5001: Flags [.], cksum 0xf256 (correct), seq 19572:20970, ack 1, win 441, options [nop,nop,TS val 19235505 ecr 30734601], length 1398
06:04:56.916884 IP (tos 0x0, ttl 62, id 48080, offset 0, flags [DF], proto TCP (6), length 72)
172.20.156.138.5001 > 172.20.156.140.53025: Flags [.], cksum 0x617f (correct), seq 1, ack 2796, win 743, options [nop,nop,TS val 30734601 ecr 19235505,nop,nop,sack 2 {18174:19572}{12582:15378}], length 0
06:04:56.916907 IP (tos 0x0, ttl 62, id 48081, offset 0, flags [DF], proto TCP (6), length 72)
172.20.156.138.5001 > 172.20.156.140.53025: Flags [.], cksum 0x5bdd (correct), seq 1, ack 2796, win 787, options [nop,nop,TS val 30734601 ecr 19235505,nop,nop,sack 2 {18174:20970}{12582:15378}], length 0
Alex, I'll duplicate all I've described in email here so it will be pinned here.
root@node-8:~# uname -r
3.13.0-88-generic
root@node-8:~# modinfo i40e 3.13.0- 88-generic/ updates/ dkms/i40e. ko
filename: /lib/modules/
version: 1.3.47
For controller node I've tried i40e 1.5.18.
From default scenario (TSO and GRO turned ON on all devices) we were trying to change TSO and GRO on different devices and measuring performance.
Changes against default / Test result Iperf TCP default test
Default deployment 22.8 Kbits/sec
GRO off on controller private NICs(bond slaves) 1.80 Gbits/sec
TSO off on controller public interface 5.19 Gbits/sec
The only place where we are having an MTU mismatch is FIP namespace of destination compute node.
root@node-10:~# ip netns e fip-54905d4f- 9b66-4c2e- ab9d-4105ea459f ed ip link UP,LOWER_ UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default MULTICAST, UP,LOWER_ UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 MULTICAST, UP,LOWER_ UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
4: fpr-5faaca01-9: <BROADCAST,
link/ether 8e:9b:95:a4:f8:1f brd ff:ff:ff:ff:ff:ff
30: fg-1336f4fb-cb: <BROADCAST,
link/ether fa:16:3e:ba:a7:cb brd ff:ff:ff:ff:ff:ff
I was dumping traffic for TSO on/off on both (destination compute and controller) public interfaces. Just to see the difference:
TSO ON (low speed)
Controller:
06:04:56.915792 IP (tos 0x0, ttl 62, id 29440, offset 0, flags [DF], proto TCP (6), length 1450) 20.156. 140.53025 > 172.20. 156.138. 5001: Flags [.], cksum 0x369f (correct), seq 1398:2796, ack 1, win 441, options [nop,nop,TS val 19235505 ecr 30734267], length 1398 20.156. 138.5001 > 172.20. 156.140. 53025: Flags [.], cksum 0x9aba (correct), seq 1, ack 2796, win 699, options [nop,nop,TS val 30734601 ecr 19235505, nop,nop, sack 1 {12582:15378}], length 0 20.156. 140.53025 > 172.20. 156.138. 5001: Flags [.], cksum 0xf9ce (correct), seq 18174:19572, ack 1, win 441, options [nop,nop,TS val 19235505 ecr 30734601], length 1398 20.156. 140.53025 > 172.20. 156.138. 5001: Flags [.], cksum 0xf256 (correct), seq 19572:20970, ack 1, win 441, options [nop,nop,TS val 19235505 ecr 30734601], length 1398 20.156. 138.5001 > 172.20. 156.140. 53025: Flags [.], cksum 0x617f (correct), seq 1, ack 2796, win 743, options [nop,nop,TS val 30734601 ecr 19235505, nop,nop, sack 2 {18174: 19572}{ 12582:15378} ], length 0 20.156. 138.5001 > 172.20. 156.140. 53025: Flags [.], cksum 0x5bdd (correct), seq 1, ack 2796, win 787, options [nop,nop,TS val 30734601 ecr 19235505, nop,nop, sack 2 {18174: 20970}{ 12582:15378} ], length 0
172.
06:04:56.916241 IP (tos 0x0, ttl 62, id 48079, offset 0, flags [DF], proto TCP (6), length 64)
172.
06:04:56.916660 IP (tos 0x0, ttl 62, id 29441, offset 0, flags [DF], proto TCP (6), length 1450)
172.
06:04:56.916754 IP (tos 0x0, ttl 62, id 29442, offset 0, flags [DF], proto TCP (6), length 1450)
172.
06:04:56.916884 IP (tos 0x0, ttl 62, id 48080, offset 0, flags [DF], proto TCP (6), length 72)
172.
06:04:56.916907 IP (tos 0x0, ttl 62, id 48081, offset 0, flags [DF], proto TCP (6), length 72)
172.
Compute: 20.156. 140.53025 > 172.20. 156.138. 5001: Flags [.], cksum 0x369f (correct), seq 2796:4194, ack 1, win 441, options [nop,nop,TS val 19235505 ecr 30734267], length 1398 20.156. 138.5001 > 172.20. 156.140. 53025: Flags [.], cksum 0x9172 (incorrect -> 0x9aba), seq 1, ack 4194, win 699, options [nop,nop,TS val 30734601 ecr 19235505, nop,nop, sack 1 {13980:16776}], length 0 20.156. 140.53025 > 172.20. 156.138. 5001: Flags [.], cksum 0xf9ce (correct), seq 19572:20970, ack 1, win 441, options [nop,nop,TS val 19235505 ecr 30734601], length 1398 20.156. 140.53025 > 172.20. 156.138. 5001: Flags [.], cksum 0xf256 (correct), seq 20970:22368, ack 1, win 441, options [nop,nop,TS val 19235505 ecr 30734601], length 1398 20.156. 138.5001 > 172.20. 156.140. 53025: Flags [.], cksum 0x917a (incorrect -> 0x617f), seq 1, ack 4194, win 743, options [nop,nop,TS val 30734601 ecr 19235505, nop,nop, sack 2 {19572: 20970}{ 13980:16776} ], length 0 20.156. 138.5001 > 172.20. 156.140. 53025: Flags [.], cksum 0x917a (incorrect -> 0x5bdd), seq 1, ack 4194, win 787, options [nop,nop,TS val 30734601 ecr 19235505, nop,nop, sack 2 {19572: 22368}{ 13980:16776} ], length 0
6:04:56.912659 IP (tos 0x0, ttl 62, id 29440, offset 0, flags [DF], proto TCP (6), length 1450)
172.
06:04:56.913036 IP (tos 0x0, ttl 62, id 48079, offset 0, flags [DF], proto TCP (6), length 64)
172.
06:04:56.913505 IP (tos 0x0, ttl 62, id 29441, offset 0, flags [DF], proto TCP (6), length 1450)
172.
06:04:56.913612 IP (tos 0x0, ttl 62, id 29442, offset 0, flags [DF], proto TCP (6), length 1450)
172.
06:04:56.913687 IP (tos 0x0, ttl 62, id 48080, offset 0, flags [DF], proto TCP (6), length 72)
172.
06:04:56.913695 IP (tos 0x0, ttl 62, id 48081, offset 0, flags [DF], proto TCP (6), length 72)
172.
TSO OFF
Controller:
172. 20.156. 140.53024 > 172.20. 156.138. 5001: Flags [.], cksum 0x96dc (incorrect -> 0x0a77), seq 241855:243253, ack 0, win 441, options [nop,nop,TS val 19198612 ecr 30697707], length 1398 20.156. 140.53024 > 172.20. 156.138. 5001: Flags [.], cksum 0x96dc (incorrect -> 0x02ff), seq 243253:244651, ack 0, win 441, options [nop,nop,TS val 19198612 ecr 30697707], length 1398 20.156. 140.53024 > 172.20. 156.138. 5001: Flags [.], cksum 0x96dc (incorrect -> 0x0591), seq 244651:246049, ack 0, win 441, options [nop,nop,TS val 19198612 ecr 30697707], length 1398 20.156. 138.5001 > 172.20. 156.140. 53024: Flags [.], cksum 0xde6a (correct), seq 0, ack 64309, win 30734, options [nop,nop,TS val 30697708 ecr 19198612], length 0 20.156. 140.53024 > 172.20. 156.138. 5001: Flags [.], cksum 0x96dc (incorrect -> 0xfe18), seq 246049:247447, ack 0, win 441, options [nop,nop,TS val 19198612 ecr 30697707], length 1398 20.156. 140.53024 > 172.20. 156.138. 5001: Flags [.], cksum 0x96dc (incorrect -> 0xf6a0), seq 247447:248845, ack 0, win 441, options [nop,nop,TS val 19198612 ecr 30697707], length 1398
06:02:29.347727 IP (tos 0x0, ttl 62, id 6084, offset 0, flags [DF], proto TCP (6), length 1450)
172.
06:02:29.347728 IP (tos 0x0, ttl 62, id 6085, offset 0, flags [DF], proto TCP (6), length 1450)
172.
06:02:29.347725 IP (tos 0x0, ttl 62, id 11529, offset 0, flags [DF], proto TCP (6), length 52)
172.
06:02:29.347728 IP (tos 0x0, ttl 62, id 6086, offset 0, flags [DF], proto TCP (6), length 1450)
172.
06:02:29.347730 IP (tos 0x0, ttl 62, id 6087, offset 0, flags [DF], proto TCP (6), length 1450)
172.
Compute:
06:02:27.823811 IP (tos 0x0, ttl 62, id 47700, offset 0, flags [DF], proto TCP (6), length 14032) 20.156. 140.53024 > 172.20. 156.138. 5001: Flags [.], cksum 0xc802 (incorrect -> 0x33f1), seq 651469:665449, ack 0, win 441, options [nop,nop,TS val 19198232 ecr 30697327], length 13980 20.156. 140.53024 > 172.20. 156.138. 5001: Flags [.], cksum 0x8c9b (incorrect -> 0xbbcd), seq 665449:729757, ack 0, win 441, options [nop,nop,TS val 19198232 ecr 30697327], length 64308 20.156. 140.53024 > 172.20. 156.138. 5001: Flags [.], cksum 0xf3b2 (incorrect -> 0x6c43), seq 729757:754921, ack 0, win 441, options [nop,nop,TS val 19198232 ecr 30697327], length 25164 20.156. 138.5001 > 172.20. 156.140. 53024: Flags [.], cksum 0x9166 (incorrect -> 0x49c9), seq 0, ack 466933, win 30734, options [nop,nop,TS val 30697328 ecr 19198232], length 0 20.156. 138.5001 > 172.20. 156.140. 53024: Flags [.], cksum 0x9166 (incorrect -> 0xe77c), seq 0, ack 492097, win 30734, options [nop,nop,TS val 30697328 ecr 19198232], length 0
172.
06:02:27.823944 IP (tos 0x0, ttl 62, id 47710, offset 0, flags [DF], proto TCP (6), length 64360)
172.
06:02:27.823949 IP (tos 0x0, ttl 62, id 47756, offset 0, flags [DF], proto TCP (6), length 25216)
172.
06:02:27.823995 IP (tos 0x0, ttl 62, id 46912, offset 0, flags [DF], proto TCP (6), length 52)
172.
06:02:27.824002 IP (tos 0x0, ttl 62, id 46913, offset 0, flags [DF], proto TCP (6), length 52)
172.
With dumping on slaves/bonds I've discovered that packets arrived on controller are 1500 size (vxlan encapsulated + 1398 payload).
Also for physical interfaces ethtool -k are attached(in default scenario with poor performance).