Enable napi_tx for GCP/GKE kernels

Bug #1834681 reported by Chris Newcomer
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux-gcp (Ubuntu)
Invalid
Undecided
Unassigned
Xenial
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Medium
Unassigned
Disco
Fix Released
Medium
Unassigned
linux-gke-4.15 (Ubuntu)
Invalid
Undecided
Unassigned
Xenial
Invalid
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned
Disco
Invalid
Undecided
Unassigned
linux-gke-5.0 (Ubuntu)
Invalid
Undecided
Unassigned
Xenial
Invalid
Undecided
Unassigned
Bionic
Fix Released
Medium
Unassigned
Disco
Invalid
Undecided
Unassigned

Bug Description

Background: Napi_tx is a Linux kernel feature that makes the virtio driver call the skb destructor after the packets are actually “out” (i.e., at TX completion interrupt), as opposed to immediately after the packets are enqueued. This provides socket backpressure and is critical for features such as TSQ. Enabling napi_tx in Cloud guests is an indispensable link in the chain of end-to-end backpressure from USPS all the way up to the guest applications. It would help reduce bufferbloat, packet drops and/or avoid HoL blocking when the traffic from the VMs are rate limited (due to congestion/BwE/etc).

The GCP networking engineering teams have asked us to include and enable napi_tx on the major guest OS's on the platform. They have 6 months of performance and regression testing and are comfortable moving forward with this broadly.

The main request is to change this module parameter:
+++ b/drivers/net/virtio_net.c
@@ -26,7 +26,7 @@
static int napi_weight = NAPI_POLL_WEIGHT;
module_param(napi_weight, int, 0444);

-static bool csum = true, gso = true, napi_tx;
+static bool csum = true, gso = true, napi_tx = true;

That is either the above kernel change or a configuration change at module load. Note that that also gives us a simple resolution in the unlikely case that this causes a regression on some workloads.

Besides the main switch, kernels need these other prerequisite patches:

The main feature, in 4.12-rc1:

1d11e732e7d50 virtio-net: use netif_tx_napi_add for tx napi
78a57b482aa53 virtio-net: on tx, only call napi_disable if tx napi is on
bdb12e0d2ffc8 virtio-net: keep tx interrupts disabled unless kick
7b0411ef4aa69 virtio-net: clean tx descriptors from rx napi
ea7735d97ba90 virtio-net: move free_old_xmit_skbs
b92f1e6751a6a virtio-net: transmit napi
e4e8452a4ab30 virtio-net: napi helper functions

Virtio-net queue affinity, in 4.19-rc1:

2ca653d607ce5 virtio_net: Stripe queue affinities across cores.
19e226e8cc5da virtio: Make vp_set_vq_affinity() take a mask.
9af18e56d43ca cpumask: make cpumask_next_wrap available without smp

A nice to have is ethtool support to test whether the feature is enabled, in 5.1-rc1:

133bbb18ab1a2 virtio-net: per-queue RPS config

CVE References

Stefan Bader (smb)
Changed in linux-gcp (Ubuntu Bionic):
importance: Undecided → Medium
status: New → In Progress
Changed in linux-gke-5.0 (Ubuntu Bionic):
importance: Undecided → Medium
status: New → In Progress
Changed in linux-gke-5.0 (Ubuntu Disco):
status: New → Invalid
Changed in linux-gcp (Ubuntu Disco):
importance: Undecided → Medium
status: New → In Progress
Revision history for this message
Stefan Bader (smb) wrote :

This likely looks confusing but the bionic/gcp kernel (rolled to 5.0) is based on the disco/gcp kernel, so the patch(es) only need to go to disco and then come back automatically.

Changed in linux-gcp (Ubuntu Bionic):
status: In Progress → Invalid
Changed in linux-gcp (Ubuntu):
status: New → Invalid
Changed in linux-gke-5.0 (Ubuntu):
status: New → Invalid
Changed in linux-gcp (Ubuntu Xenial):
status: New → In Progress
Changed in linux-gke-5.0 (Ubuntu Xenial):
status: New → Invalid
Changed in linux-gcp (Ubuntu Xenial):
status: In Progress → Fix Committed
Changed in linux-gke-4.15 (Ubuntu Bionic):
status: New → Fix Committed
Changed in linux-gke-4.15 (Ubuntu Xenial):
status: New → Invalid
Changed in linux-gke-4.15 (Ubuntu Disco):
status: New → Invalid
Changed in linux-gke-4.15 (Ubuntu):
status: New → Invalid
Changed in linux-gcp (Ubuntu Disco):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (21.0 KiB)

This bug was fixed in the package linux-gke-4.15 - 4.15.0-1045.48

---------------
linux-gke-4.15 (4.15.0-1045.48) bionic; urgency=medium

  * bionic/linux-gke-4.15: 4.15.0-1045.48 -proposed tracker (LP: #1844382)

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts

  * Enable napi_tx for GCP/GKE kernels (LP: #1834681)
    - cpumask: make cpumask_next_wrap available without smp
    - lib: optimize cpumask_next_and()
    - net: Refactor XPS for CPUs and Rx queues
    - net: Use static_key for XPS maps
    - net-sysfs: Add interface for Rx queue(s) map per Tx queue
    - net: allow to call netif_reset_xps_queues() under cpus_read_lock
    - virtio: Make vp_set_vq_affinity() take a mask.
    - virtio_net: Stripe queue affinities across cores.
    - virtio-net: per-queue RPS config
    - virtio_net: enable napi_tx by default
    - virtio_net: ethtool tx napi configuration

  [ Ubuntu: 4.15.0-65.74 ]

  * bionic/linux: 4.15.0-65.74 -proposed tracker (LP: #1844403)
  * arm64: large modules fail to load (LP: #1841109)
    - arm64/kernel: kaslr: reduce module randomization range to 4 GB
    - arm64/kernel: don't ban ADRP to work around Cortex-A53 erratum #843419
    - arm64: fix undefined reference to 'printk'
    - arm64/kernel: rename module_emit_adrp_veneer->module_emit_veneer_for_adrp
    - [config] Remove CONFIG_ARM64_MODULE_CMODEL_LARGE
  * CVE-2018-20976
    - xfs: clear sb->s_fs_info on mount failure
  * br_netfilter: namespace sysctl operations (LP: #1836910)
    - net: bridge: add bitfield for options and convert vlan opts
    - net: bridge: convert nf call options to bits
    - netfilter: bridge: port sysctls to use brnf_net
    - netfilter: bridge: namespace bridge netfilter sysctls
    - netfilter: bridge: prevent UAF in brnf_exit_net()
  * tuntap: correctly set SOCKWQ_ASYNC_NOSPACE (LP: #1830756)
    - tuntap: correctly set SOCKWQ_ASYNC_NOSPACE
  * Bionic update: upstream stable patchset 2019-08-30 (LP: #1842114)
    - HID: Add 044f:b320 ThrustMaster, Inc. 2 in 1 DT
    - MIPS: kernel: only use i8253 clocksource with periodic clockevent
    - mips: fix cacheinfo
    - netfilter: ebtables: fix a memory leak bug in compat
    - ASoC: dapm: Fix handling of custom_stop_condition on DAPM graph walks
    - bonding: Force slave speed check after link state recovery for 802.3ad
    - can: dev: call netif_carrier_off() in register_candev()
    - ASoC: Fail card instantiation if DAI format setup fails
    - st21nfca_connectivity_event_received: null check the allocation
    - st_nci_hci_connectivity_event_received: null check the allocation
    - ASoC: ti: davinci-mcasp: Correct slot_width posed constraint
    - net: usb: qmi_wwan: Add the BroadMobi BM818 card
    - qed: RDMA - Fix the hw_ver returned in device attributes
    - isdn: mISDN: hfcsusb: Fix possible null-pointer dereferences in
      start_isoc_chain()
    - netfilter: ipset: Fix rename concurrency with listing
    - isdn: hfcsusb: Fix mISDN driver crash caused by transfer buffer on the stack
    - perf bench numa: Fix cpu0 binding
    - can: sja1000: force the string buffer NULL-terminated
    - can: peak_usb: force the string buffer NULL-terminated
    - ne...

Changed in linux-gke-4.15 (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (127.1 KiB)

This bug was fixed in the package linux-gcp - 5.0.0-1020.20

---------------
linux-gcp (5.0.0-1020.20) disco; urgency=medium

  * disco/linux-gcp: 5.0.0-1020.20 -proposed tracker (LP: #1846015)

  [ Ubuntu: 5.0.0-31.33 ]

  * disco/linux: 5.0.0-31.33 -proposed tracker (LP: #1846026)
  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts
  * /proc/self/maps paths missing on live session (was vlc won't start; eoan
    19.10 & bionic 18.04 ubuntu/lubuntu/kubuntu/xubuntu/ubuntu-mate dailies)
    (LP: #1842382)
    - SAUCE: Revert "UBUNTU: SAUCE: shiftfs: enable overlayfs on shiftfs"

linux-gcp (5.0.0-1019.19) disco; urgency=medium

  * disco/linux-gcp: 5.0.0-1019.19 -proposed tracker (LP: #1845128)

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts

  * Enable napi_tx for GCP/GKE kernels (LP: #1834681)
    - virtio-net: per-queue RPS config
    - virtio_net: enable napi_tx by default

linux-gcp (5.0.0-1018.18) disco; urgency=medium

  * disco/linux-gcp: 5.0.0-1018.18 -proposed tracker (LP: #1844351)

  * Disco update: upstream stable patchset 2019-08-20 (LP: #1840846)
    - [Config] gcp: rename module adv7511

  * Disco update: upstream stable patchset 2019-08-13 (LP: #1840076)
    - [Config] gcp: updateconfigs for CONFIG_NOUVEAU_LEGACY_CTX_SUPPORT

  [ Ubuntu: 5.0.0-30.32 ]

  * disco/linux: 5.0.0-30.32 -proposed tracker (LP: #1844362)
  * Disco update: upstream stable patchset 2019-08-20 (LP: #1840846)
    - Revert "e1000e: fix cyclic resets at link up with active tx"
    - e1000e: start network tx queue only when link is up
    - Input: synaptics - enable SMBUS on T480 thinkpad trackpad
    - nilfs2: do not use unexported cpu_to_le32()/le32_to_cpu() in uapi header
    - drivers: base: cacheinfo: Ensure cpu hotplug work is done before Intel RDT
    - firmware: improve LSM/IMA security behaviour
    - irqchip/gic-v3-its: Fix command queue pointer comparison bug
    - clk: ti: clkctrl: Fix returning uninitialized data
    - efi/bgrt: Drop BGRT status field reserved bits check
    - perf/core: Fix perf_sample_regs_user() mm check
    - ARM: dts: gemini Fix up DNS-313 compatible string
    - ARM: omap2: remove incorrect __init annotation
    - afs: Fix uninitialised spinlock afs_volume::cb_break_lock
    - x86/apic: Fix integer overflow on 10 bit left shift of cpu_khz
    - be2net: fix link failure after ethtool offline test
    - ppp: mppe: Add softdep to arc4
    - sis900: fix TX completion
    - ARM: dts: imx6ul: fix PWM[1-4] interrupts
    - pinctrl: mcp23s08: Fix add_data and irqchip_add_nested call order
    - dm table: don't copy from a NULL pointer in realloc_argv()
    - dm verity: use message limit for data block corruption message
    - x86/boot/64: Fix crash if kernel image crosses page table boundary
    - x86/boot/64: Add missing fixup_pointer() for next_early_pgt access
    - HID: chicony: add another quirk for PixArt mouse
    - pinctrl: mediatek: Ignore interrupts that are wake only during resume
    - cpu/hotplug: Fix out-of-bounds read when setting fail state
    - pinctrl: mediatek: Update cur_mask in mask/mask ops
    - linux/kernel.h: fix overflow for DIV_ROUND_UP_ULL
    - genirq...

Changed in linux-gcp (Ubuntu Disco):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (20.9 KiB)

This bug was fixed in the package linux-gcp - 4.15.0-1046.49

---------------
linux-gcp (4.15.0-1046.49) xenial; urgency=medium

  * xenial/linux-gcp: 4.15.0-1046.49 -proposed tracker (LP: #1845815)

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts

  * Enable napi_tx for GCP/GKE kernels (LP: #1834681)
    - cpumask: make cpumask_next_wrap available without smp
    - lib: optimize cpumask_next_and()
    - net: Refactor XPS for CPUs and Rx queues
    - net: Use static_key for XPS maps
    - net-sysfs: Add interface for Rx queue(s) map per Tx queue
    - net: allow to call netif_reset_xps_queues() under cpus_read_lock
    - virtio: Make vp_set_vq_affinity() take a mask.
    - virtio_net: Stripe queue affinities across cores.
    - virtio-net: per-queue RPS config
    - virtio_net: enable napi_tx by default
    - virtio_net: ethtool tx napi configuration

  [ Ubuntu: 4.15.0-65.74 ]

  * bionic/linux: 4.15.0-65.74 -proposed tracker (LP: #1844403)
  * arm64: large modules fail to load (LP: #1841109)
    - arm64/kernel: kaslr: reduce module randomization range to 4 GB
    - arm64/kernel: don't ban ADRP to work around Cortex-A53 erratum #843419
    - arm64: fix undefined reference to 'printk'
    - arm64/kernel: rename module_emit_adrp_veneer->module_emit_veneer_for_adrp
    - [config] Remove CONFIG_ARM64_MODULE_CMODEL_LARGE
  * CVE-2018-20976
    - xfs: clear sb->s_fs_info on mount failure
  * br_netfilter: namespace sysctl operations (LP: #1836910)
    - net: bridge: add bitfield for options and convert vlan opts
    - net: bridge: convert nf call options to bits
    - netfilter: bridge: port sysctls to use brnf_net
    - netfilter: bridge: namespace bridge netfilter sysctls
    - netfilter: bridge: prevent UAF in brnf_exit_net()
  * tuntap: correctly set SOCKWQ_ASYNC_NOSPACE (LP: #1830756)
    - tuntap: correctly set SOCKWQ_ASYNC_NOSPACE
  * Bionic update: upstream stable patchset 2019-08-30 (LP: #1842114)
    - HID: Add 044f:b320 ThrustMaster, Inc. 2 in 1 DT
    - MIPS: kernel: only use i8253 clocksource with periodic clockevent
    - mips: fix cacheinfo
    - netfilter: ebtables: fix a memory leak bug in compat
    - ASoC: dapm: Fix handling of custom_stop_condition on DAPM graph walks
    - bonding: Force slave speed check after link state recovery for 802.3ad
    - can: dev: call netif_carrier_off() in register_candev()
    - ASoC: Fail card instantiation if DAI format setup fails
    - st21nfca_connectivity_event_received: null check the allocation
    - st_nci_hci_connectivity_event_received: null check the allocation
    - ASoC: ti: davinci-mcasp: Correct slot_width posed constraint
    - net: usb: qmi_wwan: Add the BroadMobi BM818 card
    - qed: RDMA - Fix the hw_ver returned in device attributes
    - isdn: mISDN: hfcsusb: Fix possible null-pointer dereferences in
      start_isoc_chain()
    - netfilter: ipset: Fix rename concurrency with listing
    - isdn: hfcsusb: Fix mISDN driver crash caused by transfer buffer on the stack
    - perf bench numa: Fix cpu0 binding
    - can: sja1000: force the string buffer NULL-terminated
    - can: peak_usb: force the string buffer NULL-terminated
    - net/ethernet/qlog...

Changed in linux-gcp (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (127.3 KiB)

This bug was fixed in the package linux-gke-5.0 - 5.0.0-1020.20~18.04.1

---------------
linux-gke-5.0 (5.0.0-1020.20~18.04.1) bionic; urgency=medium

  * bionic/linux-gke-5.0: 5.0.0-1020.20~18.04.1 -proposed tracker (LP: #1846014)

  [ Ubuntu: 5.0.0-1020.20 ]

  * disco/linux-gcp: 5.0.0-1020.20 -proposed tracker (LP: #1846015)
  * disco/linux: 5.0.0-31.33 -proposed tracker (LP: #1846026)
  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts
  * /proc/self/maps paths missing on live session (was vlc won't start; eoan
    19.10 & bionic 18.04 ubuntu/lubuntu/kubuntu/xubuntu/ubuntu-mate dailies)
    (LP: #1842382)
    - SAUCE: Revert "UBUNTU: SAUCE: shiftfs: enable overlayfs on shiftfs"

linux-gke-5.0 (5.0.0-1019.19~18.04.2) bionic; urgency=medium

  * bionic/linux-gke-5.0: 5.0.0-1019.19~18.04.2 -proposed tracker (LP: #1845131)

  [ Ubuntu: 5.0.0-1019.19 ]

  * disco/linux-gcp: 5.0.0-1019.19 -proposed tracker (LP: #1845128)
  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts
  * Enable napi_tx for GCP/GKE kernels (LP: #1834681)
    - virtio-net: per-queue RPS config
    - virtio_net: enable napi_tx by default

  [ Ubuntu: 5.0.0-1018.18 ]

  * disco/linux-gcp: 5.0.0-1018.18 -proposed tracker (LP: #1844351)
  * Disco update: upstream stable patchset 2019-08-20 (LP: #1840846)
    - [Config] gcp: rename module adv7511
  * Disco update: upstream stable patchset 2019-08-13 (LP: #1840076)
    - [Config] gcp: updateconfigs for CONFIG_NOUVEAU_LEGACY_CTX_SUPPORT
  * disco/linux: 5.0.0-30.32 -proposed tracker (LP: #1844362)
  * Disco update: upstream stable patchset 2019-08-20 (LP: #1840846)
    - Revert "e1000e: fix cyclic resets at link up with active tx"
    - e1000e: start network tx queue only when link is up
    - Input: synaptics - enable SMBUS on T480 thinkpad trackpad
    - nilfs2: do not use unexported cpu_to_le32()/le32_to_cpu() in uapi header
    - drivers: base: cacheinfo: Ensure cpu hotplug work is done before Intel RDT
    - firmware: improve LSM/IMA security behaviour
    - irqchip/gic-v3-its: Fix command queue pointer comparison bug
    - clk: ti: clkctrl: Fix returning uninitialized data
    - efi/bgrt: Drop BGRT status field reserved bits check
    - perf/core: Fix perf_sample_regs_user() mm check
    - ARM: dts: gemini Fix up DNS-313 compatible string
    - ARM: omap2: remove incorrect __init annotation
    - afs: Fix uninitialised spinlock afs_volume::cb_break_lock
    - x86/apic: Fix integer overflow on 10 bit left shift of cpu_khz
    - be2net: fix link failure after ethtool offline test
    - ppp: mppe: Add softdep to arc4
    - sis900: fix TX completion
    - ARM: dts: imx6ul: fix PWM[1-4] interrupts
    - pinctrl: mcp23s08: Fix add_data and irqchip_add_nested call order
    - dm table: don't copy from a NULL pointer in realloc_argv()
    - dm verity: use message limit for data block corruption message
    - x86/boot/64: Fix crash if kernel image crosses page table boundary
    - x86/boot/64: Add missing fixup_pointer() for next_early_pgt access
    - HID: chicony: add another quirk for PixArt mouse
    - pinctrl: mediatek: Ignore interrupts that are wake only during resume
    - cpu/ho...

Changed in linux-gke-5.0 (Ubuntu Bionic):
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux-gcp - 5.0.0-1020.20~18.04.1

---------------
linux-gcp (5.0.0-1020.20~18.04.1) bionic; urgency=medium

  * bionic/linux-gcp: 5.0.0-1020.20~18.04.1 -proposed tracker (LP: #1846012)

  [ Ubuntu: 5.0.0-1020.20 ]

  * disco/linux-gcp: 5.0.0-1020.20 -proposed tracker (LP: #1846015)
  * disco/linux: 5.0.0-31.33 -proposed tracker (LP: #1846026)
  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts
  * /proc/self/maps paths missing on live session (was vlc won't start; eoan
    19.10 & bionic 18.04 ubuntu/lubuntu/kubuntu/xubuntu/ubuntu-mate dailies)
    (LP: #1842382)
    - SAUCE: Revert "UBUNTU: SAUCE: shiftfs: enable overlayfs on shiftfs"

 -- Khalid Elmously <email address hidden> Fri, 04 Oct 2019 04:26:12 -0400

Changed in linux-gcp (Ubuntu Bionic):
status: Invalid → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.