Trusty kernel inbound network performance regression when GRO is enabled
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Confirmed
|
Medium
|
Unassigned |
Bug Description
After upgrading our EC2 instances from Lucid to Trusty we noticed an increase on download times, Lucid instances were able to download twice as fast as Trusty. After some investigation and testing older kernels (precise, raring and saucy) we confirmed that this only happens on trusty kernel or newer since utopic kernel shows the same result and disabling gro with `ethtool -K eth0 gro off` seems to fix the problem making download speed the same as the Lucid instances again.
The problem is easily reproducible using Apache Bench a couple times on files bigger than 100MB on 1Gb network (EC2) using HTTP or HTTPS.
Following is an example of download throughput with and without gro:
<email address hidden> ~# ethtool -K eth0 gro off
<email address hidden> ~# for i in {1..10}; do ab -n 10 $URL | grep "Transfer rate"; done
Transfer rate: 85183.40 [Kbytes/sec] received
Transfer rate: 86375.80 [Kbytes/sec] received
Transfer rate: 94720.24 [Kbytes/sec] received
Transfer rate: 84783.82 [Kbytes/sec] received
Transfer rate: 84933.09 [Kbytes/sec] received
Transfer rate: 84714.04 [Kbytes/sec] received
Transfer rate: 84795.58 [Kbytes/sec] received
Transfer rate: 84636.54 [Kbytes/sec] received
Transfer rate: 84924.26 [Kbytes/sec] received
Transfer rate: 84994.10 [Kbytes/sec] received
<email address hidden> ~# ethtool -K eth0 gro on
<email address hidden> ~# for i in {1..10}; do ab -n 10 $URL | grep "Transfer rate"; done
Transfer rate: 74193.53 [Kbytes/sec] received
Transfer rate: 56808.91 [Kbytes/sec] received
Transfer rate: 56011.58 [Kbytes/sec] received
Transfer rate: 82227.74 [Kbytes/sec] received
Transfer rate: 70806.54 [Kbytes/sec] received
Transfer rate: 72848.10 [Kbytes/sec] received
Transfer rate: 58451.94 [Kbytes/sec] received
Transfer rate: 61221.33 [Kbytes/sec] received
Transfer rate: 58620.21 [Kbytes/sec] received
Transfer rate: 69950.03 [Kbytes/sec] received
<email address hidden> ~#
Similar results can be observed using iperf and netperf as well.
Tested kernels:
Not affected: 3.8.0-44-generic (precise/raring), 3.11.0-26-generic (saucy)
Affected: 3.13.0-39-generic (trusty), 3.16.0-24-generic (utopic)
Let me know if I can provide any other information that might be helpful like perf traces and reports.
Rodrigo.
tags: | added: regression-release trusty |
Changed in linux (Ubuntu): | |
importance: | Undecided → Medium |
tags: | added: kernel-da-key |
Status changed to 'Confirmed' because the bug affects multiple users.