ixgbe{vf} - Physical Function gets IRQ when VF checks link state
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
linux (Ubuntu) | Status tracked in Focal | |||||
| Trusty |
High
|
Guilherme G. Piccoli | |||
| Xenial |
High
|
Guilherme G. Piccoli | |||
| Bionic |
High
|
Guilherme G. Piccoli | |||
| Cosmic |
High
|
Guilherme G. Piccoli | |||
| Disco |
High
|
Guilherme G. Piccoli | |||
| Eoan |
High
|
Guilherme G. Piccoli | |||
| Focal |
High
|
Guilherme G. Piccoli |
Bug Description
[Impact]
* Intel NICs that are SR-IOV capable and are managed by ixgbe driver presents a potentially harmful behavior when the ixgbevf-managed VFs (Virtual Functions) perform an ethtool link check. The ixgbevf driver issues a mailbox command in the ethtool link state handler, which induces one IRQ in the PF (Physical Function) per link check.
* This was reported as a sort of "denial-of-service" from a guest; due to some link check loop running inside a guest with PCI-PT of a ixgbevf-managed VF, the host received a huge amount of IRQs causing soft-lockups.
* The patch proposed in this SRU request fix this behavior by relying in the saved link state (obtained in the ixgbevf's watchdog routine) instead of issuing a mailbox command to the PF in every link state check request. The commit is available on Linus tree: 1e1b0c658d9b ("ixgbevf: Use cached link state instead of re-reading the value for ethtool")
http://
[Test case]
Reproducing the behavior is pretty simple; having a machine with an Intel NIC managed by ixgbe, proceed with the following steps:
a) Create one or more VFs
(echo 1 > /sys/class/net/<PF iface>/
b) In a different terminal, monitor the non-TxRx PF IRQs:
(watch -n1 "cat /proc/interrupts | grep <PF iface> | grep -v Tx")
c) Run "ethtool <VF iface>" in a loop
Without the hereby proposed patch, the PF IRQs will increase.
[Regression potential]
The patch scope is restricted to ixgbevf ethtool link-check procedure, and was developed by the vendor itself. Being a self-contained patch affecting only this driver's ethtool handler, the worst potential regression would be a wrong link state report.
summary: |
- ixgbe - Physical Function gets IRQ when VF checks link state + ixgbe{vf} - Physical Function gets IRQ when VF checks link state |
Changed in linux (Ubuntu Disco): | |
importance: | Undecided → High |
Changed in linux (Ubuntu Cosmic): | |
importance: | Undecided → High |
Changed in linux (Ubuntu Bionic): | |
importance: | Undecided → High |
Changed in linux (Ubuntu Xenial): | |
importance: | Undecided → High |
Changed in linux (Ubuntu Disco): | |
assignee: | nobody → Guilherme G. Piccoli (gpiccoli) |
Changed in linux (Ubuntu Cosmic): | |
assignee: | nobody → Guilherme G. Piccoli (gpiccoli) |
Changed in linux (Ubuntu Bionic): | |
assignee: | nobody → Guilherme G. Piccoli (gpiccoli) |
Changed in linux (Ubuntu Xenial): | |
assignee: | nobody → Guilherme G. Piccoli (gpiccoli) |
Changed in linux (Ubuntu Disco): | |
status: | New → Confirmed |
Changed in linux (Ubuntu Cosmic): | |
status: | New → Confirmed |
Changed in linux (Ubuntu Bionic): | |
status: | New → Confirmed |
Changed in linux (Ubuntu Xenial): | |
status: | New → Confirmed |
description: | updated |
Guilherme G. Piccoli (gpiccoli) wrote : | #1 |
Changed in linux (Ubuntu Xenial): | |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu Bionic): | |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu Cosmic): | |
status: | Confirmed → Won't Fix |
Changed in linux (Ubuntu Eoan): | |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu Disco): | |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu Eoan): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu Bionic): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu Disco): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu Xenial): | |
status: | In Progress → Fix Committed |
tags: | added: cscc |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/
tags: | added: verification-needed-disco |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/
tags: | added: verification-needed-bionic |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/
tags: | added: verification-needed-xenial |
Guilherme G. Piccoli (gpiccoli) wrote : | #5 |
I've validated the -proposed kernels for Xenial (4.4.0-158), Bionic (4.15.0-56) and Disco (5.0.0-22), using the test case mentioned in the description. All working fine, the issue is gone.
Also, the patch was released upstream in the 5.3.x series, so I'll mark ff-series as Released.
Cheers,
Guilherme
tags: |
added: verification-done-bionic verification-done-disco verification-done-xenial removed: verification-needed-bionic verification-needed-disco verification-needed-xenial |
Launchpad Janitor (janitor) wrote : | #6 |
This bug was fixed in the package linux - 5.2.0-10.11
---------------
linux (5.2.0-10.11) eoan; urgency=medium
* eoan/linux: 5.2.0-10.11 -proposed tracker (LP: #1838113)
* Packaging resync (LP: #1786013)
- [Packaging] resync git-ubuntu-log
* Eoan update: v5.2.4 upstream stable release (LP: #1838428)
- bnx2x: Prevent load reordering in tx completion processing
- caif-hsi: fix possible deadlock in cfhsi_exit_module()
- hv_netvsc: Fix extra rcu_read_unlock in netvsc_
- igmp: fix memory leak in igmpv3_del_delrec()
- ipv4: don't set IPv6 only flags to IPv4 addresses
- ipv6: rt6_check should return NULL if 'from' is NULL
- ipv6: Unlink sibling route in case of failure
- net: bcmgenet: use promisc for unsupported filters
- net: dsa: mv88e6xxx: wait after reset deactivation
- net: make skb_dst_force return true when dst is refcounted
- net: neigh: fix multiple neigh timer scheduling
- net: openvswitch: fix csum updates for MPLS actions
- net: phy: sfp: hwmon: Fix scaling of RX power
- net_sched: unset TCQ_F_CAN_BYPASS when adding filters
- net: stmmac: Re-work the queue selection for TSO packets
- net/tls: make sure offload also gets the keys wiped
- nfc: fix potential illegal memory access
- r8169: fix issue with confused RX unit after PHY power-down on RTL8411b
- rxrpc: Fix send on a connected, but unbound socket
- sctp: fix error handling on stream scheduler initialization
- sctp: not bind the socket in sctp_connect
- sky2: Disable MSI on ASUS P6T
- tcp: be more careful in tcp_fragment()
- tcp: fix tcp_set_
- tcp: Reset bytes_acked and bytes_received when disconnecting
- vrf: make sure skb->data contains ip header to make routing
- net/mlx5e: IPoIB, Add error path in mlx5_rdma_setup_rn
- net: bridge: mcast: fix stale nsrcs pointer in igmp3/mld2 report handling
- net: bridge: mcast: fix stale ipv6 hdr pointer when handling v6 query
- net: bridge: don't cache ether dest pointer on input
- net: bridge: stp: don't cache eth dest pointer before skb pull
- macsec: fix use-after-free of skb during RX
- macsec: fix checksumming after decryption
- netrom: fix a memory leak in nr_rx_frame()
- netrom: hold sock when setting skb->destructor
- selftests: txring_overwrite: fix incorrect test of mmap() return value
- net/tls: fix poll ignoring partially copied records
- net/tls: reject offload of TLS 1.3
- net/mlx5e: Fix port tunnel GRE entropy control
- net/mlx5e: Rx, Fix checksum calculation for new hardware
- net/mlx5e: Fix return value from timeout recover function
- net/mlx5e: Fix error flow in tx reporter diagnose
- bnxt_en: Fix VNIC accounting when enabling aRFS on 57500 chips.
- mlxsw: spectrum_dcb: Configure DSCP map as the last rule is removed
- net/mlx5: E-Switch, Fix default encap mode
- mlxsw: spectrum: Do not process learned records with a dummy FID
- dma-buf: balance refcount inbalance
- dma-buf: Discard old fence_excl on retrying get_fences_rcu for realloc
- Revert "gpio/spi: Fix spi-gpio...
Changed in linux (Ubuntu Eoan): | |
status: | Fix Committed → Fix Released |
Launchpad Janitor (janitor) wrote : | #7 |
This bug was fixed in the package linux - 5.0.0-25.26
---------------
linux (5.0.0-25.26) disco; urgency=medium
* CVE-2019-1125
- x86/cpufeatures: Carve out CQM features retrieval
- x86/cpufeatures: Combine word 11 and 12 into a new scattered features word
- x86/speculation: Prepare entry code for Spectre v1 swapgs mitigations
- x86/speculation: Enable Spectre v1 swapgs mitigations
- x86/entry/64: Use JMP instead of JMPQ
- x86/speculation
-- Kleber Sacilotto de Souza <email address hidden> Thu, 01 Aug 2019 12:04:35 +0200
Changed in linux (Ubuntu Disco): | |
status: | Fix Committed → Fix Released |
Launchpad Janitor (janitor) wrote : | #8 |
This bug was fixed in the package linux - 4.15.0-58.64
---------------
linux (4.15.0-58.64) bionic; urgency=medium
* unable to handle kernel NULL pointer dereference at 000000000000002c (IP:
iget5_
- Revert "ovl: set I_CREATING on inode being created"
- Revert "new primitive: discard_
linux (4.15.0-57.63) bionic; urgency=medium
* CVE-2019-1125
- x86/cpufeatures: Carve out CQM features retrieval
- x86/cpufeatures: Combine word 11 and 12 into a new scattered features word
- x86/speculation: Prepare entry code for Spectre v1 swapgs mitigations
- x86/speculation: Enable Spectre v1 swapgs mitigations
- x86/entry/64: Use JMP instead of JMPQ
- x86/speculation
* Packaging resync (LP: #1786013)
- update dkms package versions
linux (4.15.0-56.62) bionic; urgency=medium
* bionic/linux: 4.15.0-56.62 -proposed tracker (LP: #1837626)
* Packaging resync (LP: #1786013)
- [Packaging] resync git-ubuntu-log
- [Packaging] update helper scripts
* CVE-2019-2101
- media: uvcvideo: Fix 'type' check leading to overflow
* hibmc-drm Causes Unreadable Display for Huawei amd64 Servers (LP: #1762940)
- [Config] Set CONFIG_
- SAUCE: Make CONFIG_
* Bionic: support for Solarflare X2542 network adapter (sfc driver)
(LP: #1836635)
- sfc: make mem_bar a function rather than a constant
- sfc: support VI strides other than 8k
- sfc: add Medford2 (SFC9250) PCI Device IDs
- sfc: improve PTP error reporting
- sfc: update EF10 register definitions
- sfc: populate the timer reload field
- sfc: update MCDI protocol headers
- sfc: support variable number of MAC stats
- sfc: expose FEC stats on Medford2
- sfc: expose CTPIO stats on NICs that support them
- sfc: basic MCDI mapping of 25/50/100G link speeds
- sfc: support the ethtool ksettings API properly so that 25/50/100G works
- sfc: add bits for 25/50/100G supported/
- sfc: remove tx and MCDI handling from NAPI budget consideration
- sfc: handle TX timestamps in the normal data path
- sfc: add function to determine which TX timestamping method to use
- sfc: use main datapath for HW timestamps if available
- sfc: only enable TX timestamping if the adapter is licensed for it
- sfc: MAC TX timestamp handling on the 8000 series
- sfc: on 8000 series use TX queues for TX timestamps
- sfc: only advertise TX timestamping if we have the license for it
- sfc: simplify RX datapath timestamping
- sfc: support separate PTP and general timestamping
- sfc: support second + quarter ns time format for receive datapath
- sfc: support Medford2 frequency adjustment format
- sfc: add suffix to large constant in ptp
- sfc: mark some unexported symbols as static
- sfc: update MCDI protocol headers
- sfc: support FEC configuration through ethtool
- sfc: remove ctpio_dmabuf_start from stats
- sfc: stop the TX queue before pushing new buffers
* [18.04 FEAT] zKVM: Add hardwar...
Changed in linux (Ubuntu Bionic): | |
status: | Fix Committed → Fix Released |
Launchpad Janitor (janitor) wrote : | #9 |
This bug was fixed in the package linux - 4.4.0-159.187
---------------
linux (4.4.0-159.187) xenial; urgency=medium
* CVE-2019-1125
- x86/cpufeatures: Carve out CQM features retrieval
- x86/cpufeatures: Combine word 11 and 12 into a new scattered features word
- x86/speculation: Prepare entry code for Spectre v1 swapgs mitigations
- x86/speculation: Enable Spectre v1 swapgs mitigations
- x86/entry/64: Use JMP instead of JMPQ
- x86/speculation
linux (4.4.0-158.186) xenial; urgency=medium
* xenial/linux: 4.4.0-158.186 -proposed tracker (LP: #1837609)
* Packaging resync (LP: #1786013)
- [Packaging] resync git-ubuntu-log
- [Packaging] update helper scripts
* ixgbe{vf} - Physical Function gets IRQ when VF checks link state
(LP: #1836760)
- ixgbevf: Use cached link state instead of re-reading the value for ethtool
* CVE-2018-5383
- crypto: kpp - Key-agreement Protocol Primitives API (KPP)
- crypto: dh - Add DH software implementation
- crypto: ecdh - Add ECDH software support
- crypto: ecdh - make ecdh_shared_secret unique
- crypto: doc - add KPP documentation
- crypto: kpp, (ec)dh - fix typos
- crypto: ecc - remove unused function arguments
- crypto: ecc - remove unnecessary casts
- crypto: ecc - rename ecdh_make_pub_key()
- crypto: ecdh - add privkey generation support
- crypto: ecc - Fix NULL pointer deref. on no default_rng
- [Config] CRYPTO_ECDH=m
- Bluetooth: convert smp and selftest to crypto kpp API
- crypto: ecdh - add public key verification test
* Xenial update: 4.4.185 upstream stable release (LP: #1836668)
- fs/binfmt_flat.c: make load_flat_
- scsi: vmw_pscsi: Fix use-after-free in pvscsi_queue_lck()
- tracing: Silence GCC 9 array bounds warning
- gcc-9: silence 'address-
- usb: chipidea: udc: workaround for endpoint conflict issue
- Input: uinput - add compat ioctl number translation for UI_*_FF_UPLOAD
- apparmor: enforce nullbyte at end of tag string
- parport: Fix mem leak in parport_
- parisc: Fix compiler warnings in float emulation code
- IB/hfi1: Insure freeze_work work_struct is canceled on shutdown
- MIPS: uprobes: remove set but not used variable 'epc'
- net: hns: Fix loopback test failed at copper ports
- sparc: perf: fix updated event period in response to PERF_EVENT_
- scripts/
- scsi: ufs: Check that space was properly alloced in copy_query_response
- s390/qeth: fix VLAN attribute in bridge_hostnotify udev event
- hwmon: (pmbus/core) Treat parameters as paged if on multiple pages
- Btrfs: fix race between readahead and device replace/removal
- btrfs: start readahead also in seed devices
- can: flexcan: fix timeout when set small bitrate
- can: purge socket error queue on sock destruct
- ARM: imx: cpuidle-imx6sx: Restrict the SW2ISO increase to i.MX6SX
- Bluetooth: Align minimum encryption key size for LE and BR/EDR connections
- Bluet...
Changed in linux (Ubuntu Xenial): | |
status: | Fix Committed → Fix Released |
Changed in linux (Ubuntu Trusty): | |
assignee: | nobody → Guilherme G. Piccoli (gpiccoli) |
importance: | Undecided → High |
status: | New → In Progress |
Changed in linux (Ubuntu Trusty): | |
status: | In Progress → Fix Committed |
Mauricio Faria de Oliveira (mfo) wrote : | #10 |
Verification successful on Ubuntu 14.04 (ESM):
No interrupts are generated on the PF by the link status check of the VF.
Verified with both Link down and Link up.
# uname -rv
3.13.0-174-generic #225-Ubuntu SMP Wed Oct 2 22:08:03 UTC 2019
# ls -ld /sys/bus/
drwxr-xr-x 5 root root 0 Oct 17 20:53 /sys/bus/
drwxr-xr-x 5 root root 0 Oct 17 20:53 /sys/bus/
# echo 1 > /sys/class/
# dmesg
...
[ 1104.000023] ixgbe 0000:04:00.0 eth4: SR-IOV enabled with 1 VFs
[ 1104.100390] pci 0000:04:10.0: [8086:1515] type 00 class 0x020000
[ 1104.100981] ixgbe 0000:04:00.0: removed PHC on eth4
[ 1104.109683] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.11.3-k
[ 1104.109689] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 1104.109797] ixgbevf 0000:04:10.0: enabling device (0000 -> 0002)
[ 1104.109976] ixgbe 0000:04:00.0 eth4: VF Reset msg received from vf 0
[ 1104.110096] ixgbe 0000:04:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[ 1104.120353] ixgbevf 0000:04:10.0: PF still in reset state. Is the PF interface up?
[ 1104.120359] ixgbevf 0000:04:10.0: Assigning random MAC address
[ 1104.121016] ixgbevf 0000:04:10.0: irq 305 for MSI/MSI-X
[ 1104.121037] ixgbevf 0000:04:10.0: irq 306 for MSI/MSI-X
[ 1104.516319] ixgbe 0000:04:00.0: irq 155 for MSI/MSI-X
[ 1104.516335] ixgbe 0000:04:00.0: irq 156 for MSI/MSI-X
[ 1104.516359] ixgbe 0000:04:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
[ 1104.523447] pps pps1: new PPS source ptp4
[ 1104.523452] ixgbe 0000:04:00.0: registered PHC device on eth4
[ 1111.947668] ixgbe 0000:04:00.0 eth4: NIC Link is Up 10 Gbps, Flow Control: None
# ls -ld /sys/bus/
drwxr-xr-x 5 root root 0 Oct 17 21:11 /sys/bus/
Link down:
# ethtool enp4s16 | grep Link
Link detected: no
# while true; do ethtool enp4s16 >/dev/null; done
On another terminal:
$ while sleep 1; do echo -n "$(date) - "; cat /proc/interrupts | grep eth4 | grep -v Tx | tr -s ' '; done
Thu Oct 17 21:21:08 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:09 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:10 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:11 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:12 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:13 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-e...
Changed in linux (Ubuntu Trusty): | |
status: | Fix Committed → Fix Released |
SRU request was sent to kernel-team mailing list: https:/ /lists. ubuntu. com/archives/ kernel- team/2019- July/102282. html
Cheers,
Guilherme