Intel XL710 - i40e driver does not work with kernel 4.15 (Ubuntu 18.04)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Bionic |
Fix Released
|
High
|
Nivedita Singhvi | ||
Cosmic |
Fix Released
|
High
|
Nivedita Singhvi |
Bug Description
[Impact]
The i40e driver can get stalled on tx timeouts. This can happen when
DCB is enabled on the connected switch. This can also trigger a
second situation when a tx timeout occurs before the recovery of
a previous timeout has completed due to CPU load, which is not
handled correctly. This leads to networking delays, drops and
application timeouts and hangs. Note that the first tx timeout
cause is just one of the ways to end up in the second situation.
This issue was seen on a heavily loaded Kafka broker node running
the 4.15.0-38-generic kernel on Xenial.
Symptoms include messages in the kernel log of the form:
---
[4733544.982116] i40e 0000:18:00.1 eno2: tx_timeout: VSI_seid: 390, Q 6, NTC: 0x1a0, HWB: 0x66, NTU: 0x66, TAIL: 0x66, INT: 0x0
[4733544.982119] i40e 0000:18:00.1 eno2: tx_timeout recovery level 1, hung_queue 6
----
With the test kernel provided in this LP bug which had these
two commits compiled in, the problem has not been seen again,
and has been running successfully for several months:
"i40e: Fix for Tx timeouts when interface is brought up if
DCB is enabled"
Commit: fa38e30ac73fbb0
"i40e: prevent overlapping tx_timeout recover"
Commit: d5585b7b6846a6d
* The first commit is already in Disco, Cosmic
* The second commit is already in Disco
* Bionic needs both patches and Cosmic needs the second
[Test Case]
* We are considering the case of both issues above occurring.
* Seen by reporter on a Kafka broker node with heavy traffic.
* Not easy to reproduce as it requires something like the
following example environment and heavy load:
Kernel: 4.15.0-38-generic
Network driver: i40e
version: 2.1.14-k
NIC: Intel 40Gb XL710
DCB enabled
[Regression Potential]
Low, as the first only impacts i40e DCB environment, and has
been running for several months in production-load testing
successfully.
--- Original Description
Today Ubuntu 16.04 LTS Enablement Stacks has moved from the Kernel 4.13 to the Kernel 4.15.0-24-generic.
On a "Dell PowerEdge R330" server with a network adapter "Intel Ethernet Converged Network Adapter X710-DA2" (driver i40e) the network card no longer works and permanently displays these three lines :
[ 98.012098] i40e 0000:01:00.0 enp1s0f0: tx_timeout: VSI_seid: 388, Q 8, NTC: 0x0, HWB: 0x0, NTU: 0x1, TAIL: 0x1, INT: 0x1
[ 98.012119] i40e 0000:01:00.0 enp1s0f0: tx_timeout recovery level 11, hung_queue 8
[ 98.012125] i40e 0000:01:00.0 enp1s0f0: tx_timeout recovery unsuccessful
summary: |
- i40e driver does not work with kernel 4.15 + Intel XL710 - i40e driver does not work with kernel 4.15 (Ubuntu 18.04) |
Changed in linux (Ubuntu): | |
status: | Incomplete → In Progress |
Changed in linux (Ubuntu Bionic): | |
status: | Incomplete → In Progress |
Changed in linux (Ubuntu Bionic): | |
status: | In Progress → Confirmed |
Changed in linux (Ubuntu): | |
status: | In Progress → Confirmed |
Changed in linux (Ubuntu Bionic): | |
assignee: | Joseph Salisbury (jsalisbury) → nobody |
Changed in linux (Ubuntu): | |
assignee: | Joseph Salisbury (jsalisbury) → nobody |
Changed in linux (Ubuntu Cosmic): | |
status: | New → Confirmed |
importance: | Undecided → High |
Changed in linux (Ubuntu Bionic): | |
assignee: | nobody → Nivedita Singhvi (niveditasinghvi) |
Changed in linux (Ubuntu Cosmic): | |
assignee: | nobody → Nivedita Singhvi (niveditasinghvi) |
Changed in linux (Ubuntu Bionic): | |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu Cosmic): | |
status: | Confirmed → In Progress |
description: | updated |
tags: | added: bionic cosmic |
Changed in linux (Ubuntu Bionic): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu Cosmic): | |
status: | In Progress → Fix Committed |
tags: | added: cosmic |
tags: | added: sts |
tags: | added: cscc |
dmesg | grep i40e to see error :