KVM: Cache CPUID at KVM.ko module init to reduce latency of VM-Enter and VM-Exit

[Matthew Ruffell]

BugLink: https://bugs.launchpad.net/bugs/2093146

[Impact]

The CPUID instruction is particularly slow on newer generation processors,
with Intel's Emerald Rapids processor taking significantly longer to execute
CPUID than Skylake or Icelake.

This introduces significant latency into the KVM subsystem, as it frequently
calls CPUID when recomputing XSTATE offsets, and especially XSAVE values, as
they need to call CPUID twice for each XSAVE call.

CPUID.0xD.[1..n] are constant and do not change during runtime, as they don't
depend on XCR0 or XSS values, and can be saved and cached for future usage.

By caching CPUID.0xD.[1..n] at kvm.ko module load, latency decreases by up to
400%.

For a round trip transition between VM-Enter and VM-Exit figures from the
commit log are:

Skylake 11650
Icelake 22350
Emerald 28850

When you add the caching in:

Skylake 6850
Icelake 9000
Emerald 7900

That's a saving of 170% for Skylake, 248% for Icelake and 365% for Emerald Rapids.

[Fix]

The fix is part of a 5 patch series. We will only SRU patch 1 for the moment, as
it is the only one in mainline, and provides a 400% latency improvement, doing
the brunt of the work. Patches 2-5 are refactors and smaller performance
improvements, not yet mainlined due to needing rework, and only account for
about 2.5% latency improvement, quite small, compared to what patch 1 does.

The fix is:

commit 1201f226c863b7da739f7420ddba818cedf372fc
Author: Sean Christopherson <email address hidden>
Date: Tue Dec 10 17:32:58 2024 -0800
Subject: KVM: x86: Cache CPUID.0xD XSTATE offsets+sizes during module init
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1201f226c863b7da739f7420ddba818cedf372fc

This landed in 6.13-rc3, and is currently queued up for upstream -stable 6.12,
6.6 and 6.1.

This applies cleanly to noble, oracular. For jammy, it requires the below
dependency, and a small backport to fix some minor context mismatches.

commit cc04b6a21d431359eceeec0d812b492088b04af5
Author: Jing Liu <email address hidden>
Date: Wed Jan 5 04:35:14 2022 -0800
Subject: kvm: x86: Fix xstate_required_size() to follow XSTATE alignment rule
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=cc04b6a21d431359eceeec0d812b492088b04af5

1) Install KVM Stack on Baremetal host
$ sudo apt-get install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils
2) Enable nested virt
$ vim /etc/modprobe.d/kvm.conf
options kvm-intel nested=1
$ sudo reboot
3) Start a VM.
4) In guest, run:
$ sudo apt-get install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils
5) In host and the guest, run:
$ sudo apt install build-essential
6) Install kvm-unit-tests and run x86/vmexit/cpuid testcase.
https://gitlab.com/kvm-unit-tests/kvm-unit-tests
$ git clone https://gitlab.com/kvm-unit-tests/kvm-unit-tests.git
$ ./configure
$ make standalone
$ cd tests
$ sudo -s
# ACCEL=kvm ./vmexit_cpuid
BUILD_HEAD=0ed2cdf3
timeout -k 1s --foreground 90s /usr/bin/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.GMVjItBglu -smp 1 -append cpuid # -initrd /tmp/tmp.uaD4VVyIqc
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]
enabling apic
smp: waiting for 0 APs
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20
pci-testdev at 0x10 membar febff000 iobar c000
cpuid 66485
PASS vmexit_cpuid

The numbers next to cpuid, is the (t2 = rdtsc) - (t1 = rdtsc) count. Smaller is
better.

A test kernel is available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf403286-test

Test data for the above test kernel is:

Sapphire Rapids Intel(R) Xeon(R) Platinum 8468
24.04 LTS Noble

6.8.0-51-generic

Outside VM
cpuid 3527
cpuid 3384
cpuid 3467
cpuid 3300
cpuid 3544

Inside VM
cpuid 68395
cpuid 58364
cpuid 65254
cpuid 68554
cpuid 66905

6.8.0-51-generic+TEST403286v20250107b1

Outside VM
cpuid 3253
cpuid 3416
cpuid 3447
cpuid 3260
cpuid 3281

Inside VM
cpuid 22852
cpuid 22890
cpuid 18168
cpuid 23462
cpuid 23281

The number of cycles of rdtsc in a nested VM is of the same order of magnitude
smaller as we are expecting.

[Where problems could occur]

This fix is related to nested virtualisation in the KVM subsystem. We are adding
a new function, called on KVM module load, which caches the CPUID instead of
fetching it every time XSAVE needs to be recomputed, which can be multiple times
on VM-Enter and VM-Exit on nested guests.

CPUID is static and should never change, so there should be no issues in saving
a value and reusing it later.

If a regression were to occur, it would affect all KVM users, and there would
be no workarounds.

[Other info]

Full mailing list series: https://<email address hidden>/T/#u

[Matthew Ruffell (mruffell)]

Sapphire Rapids Intel(R) Xeon(R) Platinum 8468
24.04 LTS Noble

6.8.0-51-generic

Outside VM
cpuid 3527
cpuid 3384
cpuid 3467
cpuid 3300
cpuid 3544

Inside VM
cpuid 68395
cpuid 58364
cpuid 65254
cpuid 68554
cpuid 66905

6.8.0-51-generic+TEST403286v20250107b1

Outside VM
cpuid 3253
cpuid 3416
cpuid 3447
cpuid 3260
cpuid 3281

Inside VM
cpuid 22852
cpuid 22890
cpuid 18168
cpuid 23462
cpuid 23281

[Matthew Ruffell (mruffell)]

Sapphire Rapids Intel(R) Xeon(R) Platinum 8468
24.10 Oracular

6.11.0-13-generic

Outside VM
cpuid 3307
cpuid 3384
cpuid 3348
cpuid 3374
cpuid 3280

Inside VM
cpuid 67659
cpuid 67748
cpuid 69626
cpuid 67772
cpuid 65224

6.11.0-13-generic+TEST403286v20250107b1

Outside VM
cpuid 3450
cpuid 3499
cpuid 3293
cpuid 3507
cpuid 3459

Inside VM
cpuid 23696
cpuid 23684
cpuid 23354
cpuid 23940
cpuid 24235

[Matthew Ruffell (mruffell)]

Sapphire Rapids Intel(R) Xeon(R) Platinum 8468
22.04 LTS Jammy

5.15.0-130-generic

Outside VM
cpuid 2700
cpuid 2822
cpuid 2699
cpuid 2698
cpuid 2698

Inside VM
cpuid 51108
cpuid 54513
cpuid 51389
cpuid 53783
cpuid 53998

5.15.0-130-generic+TEST403286v20250107b1

Outside VM
cpuid 2752
cpuid 2771
cpuid 2782
cpuid 2698
cpuid 2724

Inside VM

cpuid 20580
cpuid 20512
cpuid 20523
cpuid 20700
cpuid 20549

[Matthew Ruffell (mruffell)]

Cover Letter:
https://lists.ubuntu.com/archives/kernel-team/2025-January/156198.html
Patches:
https://lists.ubuntu.com/archives/kernel-team/2025-January/156199.html
https://lists.ubuntu.com/archives/kernel-team/2025-January/156201.html
https://lists.ubuntu.com/archives/kernel-team/2025-January/156200.html

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/5.15.0-132.143 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux' to 'verification-done-jammy-linux'. If the problem still exists, change the tag 'verification-needed-jammy-linux' to 'verification-failed-jammy-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/6.11.0-17.17 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-oracular-linux' to 'verification-done-oracular-linux'. If the problem still exists, change the tag 'verification-needed-oracular-linux' to 'verification-failed-oracular-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/6.8.0-53.55 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux' to 'verification-done-noble-linux'. If the problem still exists, change the tag 'verification-needed-noble-linux' to 'verification-failed-noble-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Matthew Ruffell (mruffell)]

Performing verification for Oracular

I deployed a bare metal host, and installed a KVM stack just like the testcase
section.

Sapphire Rapids Intel(R) Xeon(R) Platinum 8468
24.10 Oracular

6.11.0-13-generic

Outside VM
cpuid 3368
cpuid 3399
cpuid 3560
cpuid 3282
cpuid 3289

Inside VM
cpuid 66195
cpuid 67674
cpuid 69635
cpuid 70856
cpuid 64587

I then enabled -proposed on both the bare metal system and the VM, and installed
6.11.0-17-generic.

$ uname -rv
6.11.0-17-generic #17-Ubuntu SMP PREEMPT_DYNAMIC Thu Jan 16 23:58:35 UTC 2025

I then re-ran the tests:

6.11.0-17-generic

Outside VM
cpuid 3374
cpuid 3298
cpuid 3325
cpuid 3448
cpuid 3470

Inside VM
cpuid 23193
cpuid 23517
cpuid 23774
cpuid 23993
cpuid 23754

As we can see, Outside VM testcases are stable, while Inside VM performance
has improved by what we expected, going from around 68,000 to 23,000.

The kernel in -proposed solves the problem, happy to mark verified for Oracular.

[Matthew Ruffell (mruffell)]

Performing verification for Noble

I deployed a bare metal host, and installed a KVM stack just like the testcase
section.

Sapphire Rapids Intel(R) Xeon(R) Platinum 8468
24.04 LTS Noble

6.8.0-51-generic

Outside VM
cpuid 3528
cpuid 3526
cpuid 3444
cpuid 3360
cpuid 3268

Inside VM
cpuid 66278
cpuid 70955
cpuid 67065
cpuid 69456
cpuid 68781

I then enabled -proposed on both the bare metal system and the VM, and installed
6.8.0-53-generic.

$ uname -rv
6.8.0-53-generic #55-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 17 15:37:52 UTC 2025

I then re-ran the tests:

6.8.0-53-generic

Outside VM
cpuid 3353
cpuid 3367
cpuid 3270
cpuid 3304
cpuid 3245

Inside VM
cpuid 23412
cpuid 24049
cpuid 23952
cpuid 23439
cpuid 23995

As we can see, Outside VM testcases are stable, while Inside VM performance
has improved by what we expected, going from around 69,000 to 23,000.

The kernel in -proposed solves the problem, happy to mark verified for Noble.

[Matthew Ruffell (mruffell)]

Performing verification for Jammy

I deployed a bare metal host, and installed a KVM stack just like the testcase
section.

Sapphire Rapids Intel(R) Xeon(R) Platinum 8468
22.04 LTS Jammy

5.15.0-130-generic

Outside VM
cpuid 2784
cpuid 2699
cpuid 2699
cpuid 2797
cpuid 2866

Inside VM
cpuid 52891
cpuid 51100
cpuid 50767
cpuid 55294
cpuid 52449

I then enabled -proposed on both the bare metal system and the VM, and installed
5.15.0-132-generic.

$ uname -rv
5.15.0-132-generic #143-Ubuntu SMP Wed Jan 15 20:39:05 UTC 2025

I then re-ran the tests:

5.15.0-132-generic

Outside VM
cpuid 2698
cpuid 2700
cpuid 2698
cpuid 2797
cpuid 2869

Inside VM
cpuid 20481
cpuid 20541
cpuid 20494
cpuid 20444
cpuid 20474

As we can see, Outside VM testcases are stable, while Inside VM performance
has improved by what we expected, going from around 52,000 to 20,000.

The kernel in -proposed solves the problem, happy to mark verified for Jammy.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-intel/6.11.0-1007.7 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-oracular-linux-intel' to 'verification-done-oracular-linux-intel'. If the problem still exists, change the tag 'verification-needed-oracular-linux-intel' to 'verification-failed-oracular-linux-intel'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 6.8.0-53.55

---------------
linux (6.8.0-53.55) noble; urgency=medium

  * noble/linux: 6.8.0-53.55 -proposed tracker (LP: #2093677)

  * Packaging resync (LP: #1786013)
    - [Packaging] debian.master/dkms-versions -- update from kernel-versions
      (main/2025.01.13)

  * generate and ship vmlinux.h to allow packages to build BPF CO-RE
    (LP: #2050083)
    - [Packaging] add linux-bpf-dev package
    - [Packaging] do not attempt to generate BTF header on armhf

  * Unable to boot as a guest on VMware ESX (LP: #2091941)
    - ptp/vmware: Use VMware hypercall API
    - input/vmmouse: Use VMware hypercall API
    - drm/vmwgfx: Use VMware hypercall API
    - x86/vmware: Use VMware hypercall API
    - x86/vmware: Correct macro names
    - x86/vmware: Remove legacy VMWARE_HYPERCALL* macros

  * When /dev/vmbus/hv_kvp is not present, disable hv-kvp-daemon (LP: #2091744)
    - [Packaging] disable hv-kvp-daemon if needed

  * Backport "netkit: Add option for scrubbing skb meta data" to 6.8
    (LP: #2091184)
    - netkit: Add option for scrubbing skb meta data

  * KVM: Cache CPUID at KVM.ko module init to reduce latency of VM-Enter and VM-
    Exit (LP: #2093146)
    - KVM: x86: Cache CPUID.0xD XSTATE offsets+sizes during module init

  * [SRU] add support of QCA BT 0489:e0fc (LP: #2085406)
    - Bluetooth: btusb: add Foxconn 0xe0fc for Qualcomm WCN785x

  * ice driver RTNL assertion failed warning on shutdown/reboot (LP: #2091107)
    - ice: Remove and readd netdev during devlink reload

  * vfio_pci soft lockup on VM start while using PCIe passthrough (LP: #2089306)
    - SAUCE: Revert "vfio/pci: Insert full vma on mmap'd MMIO fault"
    - SAUCE: Revert "vfio/pci: Use unmap_mapping_range()"

  * [SRU] Fix error of resume on rtl8168fp (LP: #2087507)
    - r8169: avoid unsolicited interrupts

  * [SRU] power: intel_pstate: HWP interrupt support for maximum ratio changed
    (LP: #2090852)
    - x86/cpufeatures: Add HWP highest perf change feature flag
    - cpufreq: intel_pstate: Support highest performance change interrupt

  * Noble update: upstream stable patchset 2024-11-29 (LP: #2089884)
    - static_call: Handle module init failure correctly in
      static_call_del_module()
    - static_call: Replace pointless WARN_ON() in static_call_module_notify()
    - jump_label: Simplify and clarify static_key_fast_inc_cpus_locked()
    - jump_label: Fix static_key_slow_dec() yet again
    - scsi: st: Fix input/output error on empty drive reset
    - scsi: pm8001: Do not overwrite PCI queue mapping
    - drm/amdgpu: Fix get each xcp macro
    - mailbox: rockchip: fix a typo in module autoloading
    - mailbox: bcm2835: Fix timeout during suspend mode
    - ceph: remove the incorrect Fw reference check when dirtying pages
    - ieee802154: Fix build error
    - net: sparx5: Fix invalid timestamps
    - net/mlx5: Fix error path in multi-packet WQE transmit
    - net/mlx5: Added cond_resched() to crdump collection
    - net/mlx5e: Fix NULL deref in mlx5e_tir_builder_alloc()
    - net/mlx5e: Fix crash caused by calling __xfrm_state_delete() twice
    - netfilter: uapi: NFTA_FLOWTABLE_HOOK is NLA_NESTED
    - net...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 6.11.0-18.18

---------------
linux (6.11.0-18.18) oracular; urgency=medium

  * CVE-2025-0927
    - SAUCE: fs: hfs/hfsplus: add key_len boundary check to hfs_bnode_read_key

linux (6.11.0-17.17) oracular; urgency=medium

  * oracular/linux: 6.11.0-17.17 -proposed tracker (LP: #2093643)

  * Packaging resync (LP: #1786013)
    - [Packaging] debian.master/dkms-versions -- update from kernel-versions
      (main/2025.01.13)

  * When /dev/vmbus/hv_kvp is not present, disable hv-kvp-daemon (LP: #2091744)
    - [Packaging] disable hv-kvp-daemon if needed

  * Backport "netkit: Add option for scrubbing skb meta data" to 6.8
    (LP: #2091184)
    - netkit: Add option for scrubbing skb meta data

  * KVM: Cache CPUID at KVM.ko module init to reduce latency of VM-Enter and VM-
    Exit (LP: #2093146)
    - KVM: x86: Cache CPUID.0xD XSTATE offsets+sizes during module init

  * [SRU] add support of QCA BT 0489:e0fc (LP: #2085406)
    - Bluetooth: btusb: add Foxconn 0xe0fc for Qualcomm WCN785x

  * oracular: ubuntu_boot lib/dynamic_queue_limits.c:99! (LP: #2089684)
    - virtio_net: correct netdev_tx_reset_queue() invocation point
    - virtio_ring: add a func argument 'recycle_done' to virtqueue_resize()
    - virtio_net: ensure netdev_tx_reset_queue is called on tx ring resize

  * Failed to probe for OVTI02C1: chip id mismatch: 560243!=0 (LP: #2090932)
    - SAUCE: ACPI: scan: Update HID for new platform

  * Bluetooth[8086:a876] crash with "hci0: Failed to read MSFT supported
    features (-110)" (LP: #2085485)
    - Bluetooth: btintel_pcie: Add recovery mechanism

  * Poor bluetooth performance on Lenovo X13s (LP: #2089357)
    - SAUCE: Bluetooth: qca: Support downloading board ID specific NVM for WCN6855

  * vfio_pci soft lockup on VM start while using PCIe passthrough (LP: #2089306)
    - SAUCE: Revert "mm: use rwsem assertion macros for mmap_lock"
    - SAUCE: Revert "vfio/pci: Insert full vma on mmap'd MMIO fault"
    - SAUCE: Revert "vfio/pci: Use unmap_mapping_range()"

  * Oracular update: v6.11.11 upstream stable release (LP: #2091655)
    - wifi: mac80211: Fix setting txpower with emulate_chanctx
    - wifi: cfg80211: Add wiphy_delayed_work_pending()
    - wifi: mac80211: Convert color collision detection to wiphy work
    - wifi: radiotap: Avoid -Wflex-array-member-not-at-end warnings
    - spi: stm32: fix missing device mode capability in stm32mp25
    - ASoC: codecs: rt5640: Always disable IRQs from rt5640_cancel_work()
    - ASoC: Intel: bytcr_rt5640: Add support for non ACPI instantiated codec
    - ASoC: Intel: bytcr_rt5640: Add DMI quirk for Vexia Edu Atla 10 tablet
    - ASoC: Intel: sst: Support LPE0F28 ACPI HID
    - wifi: iwlwifi: mvm: Use the sync timepoint API in suspend
    - wifi: iwlwifi: mvm: SAR table alignment
    - mac80211: fix user-power when emulating chanctx
    - usb: add support for new USB device ID 0x17EF:0x3098 for the r8152 driver
    - usb: typec: use cleanup facility for 'altmodes_node'
    - selftests/watchdog-test: Fix system accidentally reset after watchdog-test
    - ALSA: hda/realtek: Add subwoofer quirk for Infinix ZERO BOOK 13
    - ASoC: codecs: wcd937x: ...

[Timo Aaltonen (tjaalton)]

The verification of the Stable Release Update for linux has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.15.0-133.144

---------------
linux (5.15.0-133.144) jammy; urgency=medium

  * CVE-2025-0927
    - SAUCE: fs: hfs/hfsplus: add key_len boundary check to hfs_bnode_read_key

linux (5.15.0-132.143) jammy; urgency=medium

  * jammy/linux: 5.15.0-132.143 -proposed tracker (LP: #2093735)

  * Packaging resync (LP: #1786013)
    - [Packaging] debian.master/dkms-versions -- update from kernel-versions
      (main/2025.01.13)

  * KVM: Cache CPUID at KVM.ko module init to reduce latency of VM-Enter and VM-
    Exit (LP: #2093146)
    - kvm: x86: Fix xstate_required_size() to follow XSTATE alignment rule
    - KVM: x86: Cache CPUID.0xD XSTATE offsets+sizes during module init

  * Jammy update: v5.15.173 upstream stable release (LP: #2089541)
    - 9p: Avoid creating multiple slab caches with the same name
    - irqchip/ocelot: Fix trigger register address
    - block: Fix elevator_get_default() checking for NULL q->tag_set
    - HID: multitouch: Add support for B2402FVA track point
    - HID: multitouch: Add quirk for HONOR MagicBook Art 14 touchpad
    - bpf: use kvzmalloc to allocate BPF verifier environment
    - crypto: marvell/cesa - Disable hash algorithms
    - sound: Make CONFIG_SND depend on INDIRECT_IOMEM instead of UML
    - drm/vmwgfx: Limit display layout ioctl array size to
      VMWGFX_NUM_DISPLAY_UNITS
    - powerpc/powernv: Free name on error in opal_event_init()
    - vDPA/ifcvf: Fix pci_read_config_byte() return code handling
    - fs: Fix uninitialized value issue in from_kuid and from_kgid
    - HID: multitouch: Add quirk for Logitech Bolt receiver w/ Casa touchpad
    - HID: lenovo: Add support for Thinkpad X1 Tablet Gen 3 keyboard
    - net: usb: qmi_wwan: add Fibocom FG132 0x0112 composition
    - md/raid10: improve code of mrdev in raid10_sync_request
    - mm/memory: add non-anonymous page check in the copy_present_page()
    - udf: Allocate name buffer in directory iterator on heap
    - udf: Avoid directory type conversion failure due to ENOMEM
    - 9p: fix slab cache name creation for real
    - Linux 5.15.173

  * Jammy update: v5.15.173 upstream stable release (LP: #2089541) //
    CVE-2024-41080
    - io_uring: fix possible deadlock in io_register_iowq_max_workers()

  * Jammy update: v5.15.172 upstream stable release (LP: #2089533)
    - arm64: dts: rockchip: Fix rt5651 compatible value on rk3399-sapphire-
      excavator
    - arm64: dts: rockchip: Remove hdmi's 2nd interrupt on rk3328
    - arm64: dts: rockchip: Fix bluetooth properties on Rock960 boards
    - arm64: dts: rockchip: Remove #cooling-cells from fan on Theobroma lion
    - arm64: dts: rockchip: Fix LED triggers on rk3308-roc-cc
    - arm64: dts: imx8mp: correct sdhc ipg clk
    - ARM: dts: rockchip: fix rk3036 acodec node
    - ARM: dts: rockchip: drop grf reference from rk3036 hdmi
    - ARM: dts: rockchip: Fix the spi controller on rk3036
    - ARM: dts: rockchip: Fix the realtek audio codec on rk3036-kylin
    - NFSv3: only use NFS timeout for MOUNT when protocols are compatible
    - NFS: Add a tracepoint to show the results of nfs_set_cache_invalid()
    - NFSv3: handle out-of-order write replies.
    - ...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra/6.8.0-1003.3 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-nvidia-tegra' to 'verification-done-noble-linux-nvidia-tegra'. If the problem still exists, change the tag 'verification-needed-noble-linux-nvidia-tegra' to 'verification-failed-noble-linux-nvidia-tegra'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra-pvw/6.8.0-1004.4 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-nvidia-tegra-pvw' to 'verification-done-noble-linux-nvidia-tegra-pvw'. If the problem still exists, change the tag 'verification-needed-noble-linux-nvidia-tegra-pvw' to 'verification-failed-noble-linux-nvidia-tegra-pvw'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-gcp-6.11/6.11.0-1011.11~24.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-gcp-6.11' to 'verification-done-noble-linux-gcp-6.11'. If the problem still exists, change the tag 'verification-needed-noble-linux-gcp-6.11' to 'verification-failed-noble-linux-gcp-6.11'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure/6.8.0-1025.30 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-azure' to 'verification-done-noble-linux-azure'. If the problem still exists, change the tag 'verification-needed-noble-linux-azure' to 'verification-failed-noble-linux-azure'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-6.11/6.11.0-1004.4 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-nvidia-6.11' to 'verification-done-noble-linux-nvidia-6.11'. If the problem still exists, change the tag 'verification-needed-noble-linux-nvidia-6.11' to 'verification-failed-noble-linux-nvidia-6.11'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure-6.11/6.11.0-1012.12~24.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-azure-6.11' to 'verification-done-noble-linux-azure-6.11'. If the problem still exists, change the tag 'verification-needed-noble-linux-azure-6.11' to 'verification-failed-noble-linux-azure-6.11'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra/5.15.0-1033.33 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia-tegra' to 'verification-done-jammy-linux-nvidia-tegra'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia-tegra' to 'verification-failed-jammy-linux-nvidia-tegra'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra-igx/5.15.0-1021.21 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia-tegra-igx' to 'verification-done-jammy-linux-nvidia-tegra-igx'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia-tegra-igx' to 'verification-failed-jammy-linux-nvidia-tegra-igx'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure-nvidia/6.8.0-1014.15 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-azure-nvidia' to 'verification-done-noble-linux-azure-nvidia'. If the problem still exists, change the tag 'verification-needed-noble-linux-azure-nvidia' to 'verification-failed-noble-linux-azure-nvidia'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws-6.11/6.11.0-1013.14~24.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-aws-6.11' to 'verification-done-noble-linux-aws-6.11'. If the problem still exists, change the tag 'verification-needed-noble-linux-aws-6.11' to 'verification-failed-noble-linux-aws-6.11'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-mtk/5.15.0-1039.47 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-mtk' to 'verification-done-jammy-linux-mtk'. If the problem still exists, change the tag 'verification-needed-jammy-linux-mtk' to 'verification-failed-jammy-linux-mtk'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra-5.15/5.15.0-1039.39~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-nvidia-tegra-5.15' to 'verification-done-focal-linux-nvidia-tegra-5.15'. If the problem still exists, change the tag 'verification-needed-focal-linux-nvidia-tegra-5.15' to 'verification-failed-focal-linux-nvidia-tegra-5.15'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-fips/6.8.0-72.72+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-fips' to 'verification-done-noble-linux-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-fips' to 'verification-failed-noble-linux-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-gcp-fips/6.8.0-1035.37+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-gcp-fips' to 'verification-done-noble-linux-gcp-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-gcp-fips' to 'verification-failed-noble-linux-gcp-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws-fips/6.8.0-1034.36+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-aws-fips' to 'verification-done-noble-linux-aws-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-aws-fips' to 'verification-failed-noble-linux-aws-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-xilinx/6.8.0-1017.18 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-xilinx' to 'verification-done-noble-linux-xilinx'. If the problem still exists, change the tag 'verification-needed-noble-linux-xilinx' to 'verification-failed-noble-linux-xilinx'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure-fips/6.8.0-1034.39+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-azure-fips' to 'verification-done-noble-linux-azure-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-azure-fips' to 'verification-failed-noble-linux-azure-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!