TCP autobind almost always returns an even numbered port
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| linux-lts-xenial (Ubuntu) |
New
|
Undecided
|
Unassigned | ||
Bug Description
Summary:
TCP source ports assigned by autobind results in even number ports almost always. This causes issues with LACP xmit_hash_policy layer3+4 policy as only the first nic in a two port bond will be utilized.
Details:
kurt@cypher:~$ lsb_release -rd
Description: Ubuntu 16.04.1 LTS
Release: 16.04
kurt@cypher:~$ uname -a
Linux cypher 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Example iperf run demonstrating source ports are sequentially moving up by 2:
kurt@cypher:~$ iperf -t 30 -c saio -P 15
-------
Client connecting to saio, TCP port 5001
TCP window size: 85.0 KByte (default)
-------
[ 9] local 172.16.2.38 port 34520 connected with 172.16.2.32 port 5001
[ 3] local 172.16.2.38 port 34496 connected with 172.16.2.32 port 5001
[ 7] local 172.16.2.38 port 34498 connected with 172.16.2.32 port 5001
[ 5] local 172.16.2.38 port 34500 connected with 172.16.2.32 port 5001
[ 6] local 172.16.2.38 port 34502 connected with 172.16.2.32 port 5001
[ 4] local 172.16.2.38 port 34504 connected with 172.16.2.32 port 5001
[ 11] local 172.16.2.38 port 34506 connected with 172.16.2.32 port 5001
[ 13] local 172.16.2.38 port 34510 connected with 172.16.2.32 port 5001
[ 12] local 172.16.2.38 port 34508 connected with 172.16.2.32 port 5001
[ 16] local 172.16.2.38 port 34512 connected with 172.16.2.32 port 5001
[ 17] local 172.16.2.38 port 34514 connected with 172.16.2.32 port 5001
[ 10] local 172.16.2.38 port 34516 connected with 172.16.2.32 port 5001
[ 15] local 172.16.2.38 port 34522 connected with 172.16.2.32 port 5001
[ 8] local 172.16.2.38 port 34524 connected with 172.16.2.32 port 5001
[ 14] local 172.16.2.38 port 34518 connected with 172.16.2.32 port 5001
[ ID] Interval Transfer Bandwidth
[ 11] 0.0-30.0 sec 304 MBytes 85.0 Mbits/sec
[ 16] 0.0-30.0 sec 306 MBytes 85.5 Mbits/sec
[ 17] 0.0-30.0 sec 299 MBytes 83.6 Mbits/sec
[ 9] 0.0-30.0 sec 323 MBytes 90.3 Mbits/sec
[ 8] 0.0-30.0 sec 305 MBytes 85.4 Mbits/sec
[ 3] 0.0-30.0 sec 300 MBytes 83.8 Mbits/sec
[ 5] 0.0-30.0 sec 166 MBytes 46.3 Mbits/sec
[ 6] 0.0-30.0 sec 305 MBytes 85.3 Mbits/sec
[ 10] 0.0-30.0 sec 315 MBytes 88.0 Mbits/sec
[ 15] 0.0-30.0 sec 104 MBytes 29.1 Mbits/sec
[ 7] 0.0-30.0 sec 99.9 MBytes 27.9 Mbits/sec
[ 13] 0.0-30.0 sec 196 MBytes 54.8 Mbits/sec
[ 14] 0.0-30.1 sec 107 MBytes 29.7 Mbits/sec
[ 4] 0.0-30.1 sec 111 MBytes 30.9 Mbits/sec
[ 12] 0.0-30.1 sec 138 MBytes 38.4 Mbits/sec
[SUM] 0.0-30.1 sec 3.30 GBytes 942 Mbits/sec
Expected behaviour:
TCP autobind assigning source ports either sequentially by 1, or more securely randomly.
Note that after multiple attempts TCP autobind may result in an odd numbered source port, however it predominately results in even numbered source ports.
Here is an example where one port was odd out of 15 ports. The result is that LACP xmit_hash_policy layer3+4 is able to direct that connection over a different nic and the resulting bandwidth is doubled (2x 1G nic in LACP):
kurt@cypher:~$ iperf -t 30 -c saio -P 15
-------
Client connecting to saio, TCP port 5001
TCP window size: 85.0 KByte (default)
-------
[ 12] local 172.16.2.38 port 34676 connected with 172.16.2.32 port 5001
[ 4] local 172.16.2.38 port 34648 connected with 172.16.2.32 port 5001
[ 6] local 172.16.2.38 port 34650 connected with 172.16.2.32 port 5001
[ 3] local 172.16.2.38 port 34654 connected with 172.16.2.32 port 5001
[ 5] local 172.16.2.38 port 34652 connected with 172.16.2.32 port 5001
[ 7] local 172.16.2.38 port 34656 connected with 172.16.2.32 port 5001
[ 9] local 172.16.2.38 port 34658 connected with 172.16.2.32 port 5001
[ 13] local 172.16.2.38 port 34660 connected with 172.16.2.32 port 5001
[ 14] local 172.16.2.38 port 34664 connected with 172.16.2.32 port 5001
[ 10] local 172.16.2.38 port 34662 connected with 172.16.2.32 port 5001
[ 11] local 172.16.2.38 port 34666 connected with 172.16.2.32 port 5001
[ 17] local 172.16.2.38 port 34668 connected with 172.16.2.32 port 5001
[ 16] local 172.16.2.38 port 34672 connected with 172.16.2.32 port 5001
[ 15] local 172.16.2.38 port 34670 connected with 172.16.2.32 port 5001
[ 8] local 172.16.2.38 port 34673 connected with 172.16.2.32 port 5001
[ ID] Interval Transfer Bandwidth
[ 12] 0.0-30.0 sec 294 MBytes 82.2 Mbits/sec
[ 8] 0.0-30.0 sec 3.28 GBytes 940 Mbits/sec
[ 7] 0.0-30.0 sec 302 MBytes 84.4 Mbits/sec
[ 10] 0.0-30.0 sec 296 MBytes 82.7 Mbits/sec
[ 4] 0.0-30.0 sec 232 MBytes 64.9 Mbits/sec
[ 3] 0.0-30.0 sec 133 MBytes 37.3 Mbits/sec
[ 14] 0.0-30.0 sec 282 MBytes 78.7 Mbits/sec
[ 9] 0.0-30.0 sec 362 MBytes 101 Mbits/sec
[ 13] 0.0-30.0 sec 273 MBytes 76.2 Mbits/sec
[ 11] 0.0-30.0 sec 342 MBytes 95.5 Mbits/sec
[ 15] 0.0-30.0 sec 230 MBytes 64.2 Mbits/sec
[ 17] 0.0-30.0 sec 196 MBytes 54.8 Mbits/sec
[ 6] 0.0-30.1 sec 133 MBytes 37.1 Mbits/sec
[ 16] 0.0-30.1 sec 188 MBytes 52.4 Mbits/sec
[ 5] 0.0-30.1 sec 117 MBytes 32.5 Mbits/sec
[SUM] 0.0-30.1 sec 6.58 GBytes 1.88 Gbits/sec
This demonstrates that the LACP xmit_hash_policy layer3+4 configuration is working correctly but due to the predominately even number source ports, it primarily utilizes one nic.
| Kurt Miller (bsdkurt) wrote | #1 |
| Kurt Miller (bsdkurt) wrote | #2 |
| affects: | kernel-package (Ubuntu) → linux-lts-xenial (Ubuntu) |
| Jan Fuchs (oposum) wrote | #3 |
| Jan Fuchs (oposum) wrote | #4 |
| Kurt Miller (bsdkurt) wrote | #5 |
More data for the report. Testing using a different program shows the same results. In this case I used a python program called getput to test the performance of an openstack swift node. All the source ports are even numbered in the netstat output which prevents LACP xmit_hash_policy layer3+4 to load balance the outgoing connections:
kurt@cypher:~$ netstat -4
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 388064 172.16.2.38:58806 172.16.2.32:http ESTABLISHED
tcp 0 217200 172.16.2.38:58548 172.16.2.32:http ESTABLISHED
tcp 0 257744 172.16.2.38:58646 172.16.2.32:http ESTABLISHED
tcp 0 492032 172.16.2.38:58926 172.16.2.32:http ESTABLISHED
tcp 0 679112 172.16.2.38:58672 172.16.2.32:http ESTABLISHED
tcp 0 267880 172.16.2.38:58730 172.16.2.32:http ESTABLISHED
tcp 0 296840 172.16.2.38:58898 172.16.2.32:http ESTABLISHED
tcp 0 865904 172.16.2.38:58580 172.16.2.32:http ESTABLISHED
tcp 0 166520 172.16.2.38:59002 172.16.2.32:http ESTABLISHED
tcp 0 1970728 172.16.2.38:58704 172.16.2.32:http ESTABLISHED
tcp 0 1424832 172.16.2.38:58594 172.16.2.32:http ESTABLISHED
tcp 0 432952 172.16.2.38:58642 172.16.2.32:http ESTABLISHED
tcp 0 80760 172.16.2.38:58804 172.16.2.32:http ESTABLISHED
tcp 0 1440760 172.16.2.38:58836 172.16.2.32:http ESTABLISHED
tcp 0 154936 172.16.2.38:58840 172.16.2.32:http ESTABLISHED
tcp 0 293944 172.16.2.38:58788 172.16.2.32:http ESTABLISHED
tcp 0 199824 172.16.2.38:58876 172.16.2.32:http ESTABLISHED
tcp 0 489424 172.16.2.38:58602 172.16.2.32:http ESTABLISHED
tcp 0 0 172.16.2.38:58794 172.16.2.32:http ESTABLISHED
tcp 0 1633344 172.16.2.38:58820 172.16.2.32:http ESTABLISHED
tcp 0 477840 172.16.2.38:58540 172.16.2.32:http ESTABLISHED
tcp 0 708240 172.16.2.38:58766 172.16.2.32:http ESTABLISHED
tcp 0 382272 172.16.2.38:58632 172.16.2.32:http ESTABLISHED
tcp 0 218320 172.16.2.38:58732 172.16.2.32:http ESTABLISHED
tcp 0 618296 172.16.2.38:58574 172.16.2.32:http ESTABLISHED
tcp 0 269328 172.16.2.38:58566 172.16.2.32:http ESTABLISHED
tcp 0 249056 172.16.2.38:58872 172.16.2.32:http ESTABLISHED
tcp 0 347520 172.16.2.38:58760 172.16.2.32:http ESTABLISHED
tcp 0 434400 172.16.2.38:58940 172.16.2.32:http ESTABLISHED
tcp 0 178104 172.16.2.38:58558 172.16.2.32:http ESTABLISHED
tcp 0 529968 172.16.2.38:58628 172.16.2.32:http ESTABLISHED
tcp 0 0 172.16.2.38:ssh 172.16.1.22:37196 ESTABLISHED
tcp 0 249056 172.16.2.38:58724 172.16.2.32:http ESTABLISHED
tcp 0 366344 172.16.2.38:58654 172.16.2.32:http EST...