Introduce and use sendpages_ok() instead of sendpage_ok() in nvme-tcp and drbd

[Meir Elisha]

BugLink: https://bugs.launchpad.net/bugs/2093871

[Impact]

Currently the nvme-tcp and drbd subsystems try to enable the MSG_SPLICE_PAGES
flag on pages to be written, and when MSG_SPLICE_PAGES is set, eventually it
calls skb_splice_from_iter(), which then checks all pages with sendpage_ok()
to see if all the pages are sendable.

At the moment, both subsystems only check the first page in a potentially
contiguous block of pages, if they are sendpage_ok(), and if the first page is,
then it just assumes all the rest are sendpage_ok() too, and sends the I/O off
to eventually be found out by skb_splice_from_iter(). If one or more of the
pages in the contiguous block is not sendpage_ok(), then we get a warn printed,
data transfer is aborted. In the nvme-tcp case, IO then hangs.

This patchset introduces sendpages_ok() which iterates over each page in a
contiguous block, checks if it is sendpage_ok(), and only returns true if all
of them are.

This resolves the whole MSG_SPLICE_PAGES flag situation, since you can now
depend on the result of sendpages_ok(), instead of just assuming everything is
okay.

This issue is what caused bug 2075110 [0] to be discovered in the first place,
since it was responsible for contigious blocks of pages where the first was
sendpage_ok(), but pages further into the block were not.

[0] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2075110

Even with "md/md-bitmap: fix writing non bitmap pages" applied, the issue can
still happen, e.g. with merged IO pages, so this fix is still needed to
eliminate the issue.

[Fix]

The fixes landed in mainline 6.12-rc1:

commit 23a55f4492fcf868d068da31a2cd30c15f46207d
Author: Ofir Gal <email address hidden>
Date: Thu Jul 18 11:45:12 2024 +0300
Subject: net: introduce helper sendpages_ok()
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=23a55f4492fcf868d068da31a2cd30c15f46207d

commit 6af7331a70b4888df43ec1d7e1803ae2c43b6981
Author: Ofir Gal <email address hidden>
Date: Thu Jul 18 11:45:13 2024 +0300
Subject: nvme-tcp: use sendpages_ok() instead of sendpage_ok()
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6af7331a70b4888df43ec1d7e1803ae2c43b6981

commit 7960af373ade3b39e10106ef415e43a1d2aa48c6
Author: Ofir Gal <email address hidden>
Date: Thu Jul 18 11:45:14 2024 +0300
Subject: drbd: use sendpages_ok() instead of sendpage_ok()
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7960af373ade3b39e10106ef415e43a1d2aa48c6

They are needed for noble and oracular.

[Testcase]

This is the same testcase as the original bug 2075110 [0], as the fix is
designed to prevent it or similar other bugs from happening again.

[0] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2075110

Because of this, the fix:

commit ab99a87542f194f28e2364a42afbf9fb48b1c724
Author: Ofir Gal <email address hidden>
Date: Fri Jun 7 10:27:44 2024 +0300
Subject: md/md-bitmap: fix writing non bitmap pages
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ab99a87542f194f28e2364a42afbf9fb48b1c724

needs to be reverted during your test runs, or you won't see the issue
reproduce.

You can use this ppa for updated kernels with the revert to trigger the issue:

https://launchpad.net/~mruffell/+archive/ubuntu/sf404844-revert

This can be reproduced by running blktests md/001 [1], which the author of the fix created to act as a regression test for this issue.

[1] https://github.com/osandov/blktests/commit/a24a7b462816fbad7dc6c175e53fcc764ad0a822

Deploy a fresh Noble VM, that has a scratch NVME disk.

$ sudo apt install build-essential fio
$ git clone https://github.com/osandov/blktests.git
$ cd blktests
$ make
$ echo "TEST_DEVS=(/dev/nvme0n1)" > config
$ sudo ./check md/001

The md/001 test will hang an affected system, and the above oops message will be visible in dmesg.

A test kernel is available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf404844-test

This has both the fixes for this bug, and also bug 2075110. The issue will not
reproduce.

There is also a test kernel available with the fix for this bug present, and the
fix for bug 2075110 reverted, so you can see the impact of these patches only:

https://launchpad.net/~mruffell/+archive/ubuntu/sf404844-repro

This will also not reproduce the issue anymore.

[Where problems could occur]

What we are changing is rather simple. Instead of checking the first page and
assuming all the rest in the contiguous block are sendpage_ok(), we now
check each page in the contiguous block to see if all of them are sendpage_ok().

If any aren't, then we abort the write to the driver, and try again later. This
saves us time.

However, it does take longer to call sendpage_ok() on each of the pages in the
contiguous block, so there will be a minor performance hit.

Small performance hit for correctness should be okay.

Currently we are only applying to nvme-tcp and drbd subsystems. If a regression
were to occur, it would affect users of those subsystems only.

[Other info]

Upstream mailing list:
https://<email address hidden>/T/#u

[Masahiro Yamada (myamada)]

Does it fix any bug?

[Meir Elisha (meireli)]

Yes. Without this fix, we might use unsendable pages.

[Masahiro Yamada (myamada)]

Which Ubuntu kernels do you intend to back-port them for, Ubuntu 24.04, 22.04, or anything else?

I do not see Fixes: tag on those commits.
If you believe this is a critical issue, how about requesting back-port in the stable community?
(<email address hidden>)

Once they are back-ported to the stable kernels, they will ported to Ubuntu kernels as well.

[Meir Elisha (meireli)]

Masahiro Yamada, Thanks for the quick response.
I'm using Ubuntu 22.04 (Ubuntu 22.04 6.5.0-1017-azure), but it's needed for Ubuntu 24.04 as well. I would rather not back-port to a stable community since it may take longer. Is this the only way?

[Masahiro Yamada (myamada)]

I am not saying it is the only way.
However, I do not think the Ubuntu kernel is necessarily a quicker path than the stable community because we have a certain criteria to determine which patches should be ported.

I'd like to be clear whether this is a performance improvement (if so, how big is the improvement?), or this is a bug fix for something broken.

[Matthew Ruffell (mruffell)]

Patches are on Ubuntu Kernel Mailing List

Cover Letter
https://lists.ubuntu.com/archives/kernel-team/2025-January/156684.html
Patches
https://lists.ubuntu.com/archives/kernel-team/2025-January/156685.html
https://lists.ubuntu.com/archives/kernel-team/2025-January/156686.html
https://lists.ubuntu.com/archives/kernel-team/2025-January/156687.html

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/6.8.0-58.60 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux' to 'verification-done-noble-linux'. If the problem still exists, change the tag 'verification-needed-noble-linux' to 'verification-failed-noble-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/6.11.0-24.24 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-oracular-linux' to 'verification-done-oracular-linux'. If the problem still exists, change the tag 'verification-needed-oracular-linux' to 'verification-failed-oracular-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Matthew Ruffell (mruffell)]

Performing verification for noble.

Firstly, I started with 6.8.0-57-generic with the following commit reverted:

commit ab99a87542f194f28e2364a42afbf9fb48b1c724
Author: Ofir Gal <email address hidden>
Date: Fri Jun 7 10:27:44 2024 +0300
Subject: md/md-bitmap: fix writing non bitmap pages
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ab99a87542f194f28e2364a42afbf9fb48b1c724

This is because the testcases are the same, as the above bug was fixed due to
this bug existing in the first place.

I ran the reproducer:

sudo[1591]: mruffell : TTY=pts/0 ; PWD=/home/mruffell/blktests ; USER=root ; COMMAND=./check md/001
sudo[1591]: pam_unix(sudo:session): session opened for user root(uid=0) by mruffell(uid=1003)
unknown: run blktests md/001 at 2025-04-07 03:19:12
root[1627]: run blktests md/001
kernel: brd: module loaded
(udev-worker)[1632]: dm-0: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-0' failed with exit code 1.
kernel: Key type psk registered
kernel: nvmet: adding nsid 1 to subsystem blktests-subsystem-1
kernel: nvmet_tcp: enabling port 0 (127.0.0.1:4420)
kernel: nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
kernel: nvme nvme1: creating 2 I/O queues.
kernel: nvme nvme1: mapped 2/0/0 default/read/poll queues.
kernel: nvme nvme1: new ctrl: NQN "blktests-subsystem-1", addr 127.0.0.1:4420, hostnqn: nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
(udev-worker)[1632]: nvme1n1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/nvme1n1' failed with exit code 1.
(udev-worker)[1632]: md127: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/md127' failed with exit code 1.
kernel: md/raid1:md127: active with 1 out of 2 mirrors
kernel: ------------[ cut here ]------------
kernel: WARNING: CPU: 1 PID: 58 at net/core/skbuff.c:7006 skb_splice_from_iter+0x139/0x370
kernel: Modules linked in: nvme_tcp nvmet_tcp nvmet nvme_keyring brd raid1 tls cfg80211 8021q garp mrp stp llc binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common intel_uncore_frequency_common isst_if_common skx_edac_common nfit crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd rapl psmouse nvme pvpanic_mmio pvpanic i2c_piix4 input_leds mac_hid serio_raw sch_fq_codel dm_multipath nvme_fabrics nvme_core nvme_auth efi_pstore nfnetlink dmi_sysfs virtio_rng ip_tables x_tables autofs4
kernel: CPU: 1 PID: 58 Comm: kworker/1:1H Not tainted 6.8.0-57-generic #52+TEST404844v20250402b1-Ubuntu
kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2025
kernel: Workqueue: nvme_tcp_wq nvme_tcp_io_work [nvme_tcp]
kernel: RIP: 0010:skb_splice_from_iter+0x139/0x370
kernel: Code: 39 e1 48 8b 53 08 49 0f 47 cc 49 89 cd f6 c2 01 0f 85 c0 01 00 00 66 90 48 89 da 48 8b 12 80 e6 08 0f 84 8e 00 00 00 4d 89 fe <0f> 0b 49 c7 c0 fb ff ff ff 48 8b 85 68 ff ff ff 41 01 46 70 41 01
kernel: RSP: 0018:ffffb1e9401fba38 EFLAGS: 00010246
kernel: RAX: 0000000000000000 RBX: ffffe43105245700...

[Matthew Ruffell (mruffell)]

Performing verification for oracular.

Firstly, I started with 6.11.0-21-generic with the following commit reverted:

commit ab99a87542f194f28e2364a42afbf9fb48b1c724
Author: Ofir Gal <email address hidden>
Date: Fri Jun 7 10:27:44 2024 +0300
Subject: md/md-bitmap: fix writing non bitmap pages
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ab99a87542f194f28e2364a42afbf9fb48b1c724

This is because the testcases are the same, as the above bug was fixed due to
this bug existing in the first place.

I ran the reproducer:

sudo[1244]: mruffell : TTY=pts/0 ; PWD=/home/mruffell/blktests ; USER=root ; COMMAND=./check md/001
sudo[1244]: pam_unix(sudo:session): session opened for user root(uid=0) by mruffell(uid=1008)
unknown: run blktests md/001 at 2025-04-07 04:17:58
root[1280]: run blktests md/001
brd: module loaded
(udev-worker)[1270]: dm-0: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-0' failed with exit code 1.
Key type psk registered
nvmet: adding nsid 1 to subsystem blktests-subsystem-1
nvmet_tcp: enabling port 0 (127.0.0.1:4420)
nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
nvme nvme1: creating 2 I/O queues.
nvme nvme1: mapped 2/0/0 default/read/poll queues.
nvme nvme1: new ctrl: NQN "blktests-subsystem-1", addr 127.0.0.1:4420, hostnqn: nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
(udev-worker)[1270]: nvme1n1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/nvme1n1' failed with exit code 1.
(udev-worker)[1270]: md127: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/md127' failed with exit code 1.
md/raid1:md127: active with 1 out of 2 mirrors
------------[ cut here ]------------
WARNING: CPU: 0 PID: 60 at net/core/skbuff.c:7140 skb_splice_from_iter+0x1b5/0x370
Modules linked in: nvme_tcp nvmet_tcp nvmet nvme_keyring brd raid1 tls cfg80211 8021q garp mrp stp llc binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common intel_uncore_frequency_common skx_edac_common nfit crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd rapl pvpanic_mmio pvpanic psmouse i2c_piix4 i2c_smbus nvme input_leds mac_hid serio_raw sch_fq_codel nvme_fabrics nvme_core nvme_auth efi_pstore dm_multipath nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci dmi_sysfs virtio_rng ip_tables x_tables autofs4
CPU: 0 UID: 0 PID: 60 Comm: kworker/0:1H Not tainted 6.11.0-21-generic #20+TEST404844v20250403b1-Ubuntu
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2025
Workqueue: nvme_tcp_wq nvme_tcp_io_work [nvme_tcp]
RIP: 0010:skb_splice_from_iter+0x1b5/0x370
Code: cc 49 89 cd f6 c2 01 0f 85 20 01 00 00 66 90 48 89 da 8b 52 30 81 e2 00 00 00 82 81 fa 00 00 00 80 0f 85 41 ff ff ff 4d 89 fe <0f> 0b 49 c7 c5 fb ff ff ff 48 8b 85 68 ff ff ff 41 01 46 70 41 01
RSP: 0018:ffffac5c0020ba18 EFLAGS: 00010246
RAX: 0000000000000000 RBX: fffff04bc4c1edc0 RCX: 0000000000001000
RDX: 0000000080000000 RSI: 000000...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 6.11.0-24.24

---------------
linux (6.11.0-24.24) oracular; urgency=medium

  * oracular/linux: 6.11.0-24.24 -proposed tracker (LP: #2102476)

  * Packaging resync (LP: #1786013)
    - [Packaging] debian.master/dkms-versions -- update from kernel-versions
      (main/2025.03.17)

  * ipsec_offload in rtnetlink.sh from ubunsu_kselftests_net fails on O/J
    (LP: #2096976)
    - SAUCE: selftest: netfilter: fix null IP field in kci_test_ipsec_offload

  * Add additional PCI ids for BMG support (LP: #2098969)
    - drm/xe/bmg: Add new PCI IDs

  * wdat_wdt.ko should be pulled in by linux-image-virtual (LP: #2098554)
    - [Packaging]: wdat_wdt.ko is moved from "linux-modules-extra-*-generic" to
      "linux-modules-*-generic"

  * CVE-2025-21756
    - vsock: Keep the binding until socket destruction
    - vsock: Orphan socket after transport release

  * Oracular update: upstream stable patchset 2025-03-05 (LP: #2100983)
    - ASoC: wm8994: Add depends on MFD core
    - ASoC: samsung: Add missing selects for MFD_WM8994
    - seccomp: Stub for !CONFIG_SECCOMP
    - scsi: iscsi: Fix redundant response for ISCSI_UEVENT_GET_HOST_STATS request
    - of/unittest: Add test that of_address_to_resource() fails on non-
      translatable address
    - irqchip/sunxi-nmi: Add missing SKIP_WAKE flag
    - hwmon: (drivetemp) Set scsi command timeout to 10s
    - gfs2: Truncate address space when flipping GFS2_DIF_JDATA flag
    - smb: client: handle lack of EA support in smb2_query_path_info()
    - net: sched: fix ets qdisc OOB Indexing
    - Revert "HID: multitouch: Add support for lenovo Y9000P Touchpad"
    - cachestat: fix page cache statistics permission checking
    - scsi: storvsc: Ratelimit warning logs to prevent VM denial of service
    - USB: serial: quatech2: fix null-ptr-deref in qt2_process_read_urb()
    - ALSA: usb-audio: Add delay quirk for USB Audio Device
    - Input: xpad - add support for Nacon Pro Compact
    - Input: atkbd - map F23 key to support default copilot shortcut
    - Input: xpad - add unofficial Xbox 360 wireless receiver clone
    - Input: xpad - add QH Electronics VID/PID
    - Input: xpad - improve name of 8BitDo controller 2dc8:3106
    - Input: xpad - add support for Nacon Evol-X Xbox One Controller
    - Input: xpad - add support for wooting two he (arm)
    - drm/v3d: Assign job pointer to NULL before signaling the fence
    - ASoC: codecs: es8316: Fix HW rate calculation for 48Mhz MCLK
    - ASoC: cs42l43: Add codec force suspend/resume ops
    - drm/amd/display: Initialize denominator defaults to 1
    - ALSA: hda/realtek: Fix volume adjustment issue on Lenovo ThinkBook 16P Gen5
    - drm/connector: hdmi: Validate supported_formats matches ycbcr_420_allowed
    - ASoC: samsung: Add missing depends on I2C
    - mm: zswap: properly synchronize freeing resources during CPU hotunplug
    - mm: zswap: move allocations during CPU init outside the lock
    - libfs: Return ENOSPC when the directory offset range is exhausted
    - Revert "libfs: Add simple_offset_empty()"
    - Revert "libfs: fix infinite directory reads for offset dir"
    - libfs: Replace simple_offset end-of-direc...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 6.8.0-58.60

---------------
linux (6.8.0-58.60) noble; urgency=medium

  * noble/linux: 6.8.0-58.60 -proposed tracker (LP: #2102529)

  * Packaging resync (LP: #1786013)
    - [Packaging] update variants
    - [Packaging] debian.master/dkms-versions -- update from kernel-versions
      (main/2025.03.17)

  * wdat_wdt.ko should be pulled in by linux-image-virtual (LP: #2098554)
    - [Packaging]: wdat_wdt.ko is moved from "linux-modules-extra-*-generic" to
      "linux-modules-*-generic"

  * Noble update: upstream stable patchset 2025-03-12 (LP: #2102118)
    - openrisc: Use asm-generic's version of fix_to_virt() & virt_to_fix()
    - iTCO_wdt: mask NMI_NOW bit for update_no_reboot_bit() call
    - watchdog: xilinx_wwdt: Calculate max_hw_heartbeat_ms using clock frequency
    - watchdog: apple: Actually flush writes after requesting watchdog restart
    - watchdog: mediatek: Make sure system reset gets asserted in
      mtk_wdt_restart()
    - can: gs_usb: add VID/PID for Xylanta SAINT3 product family
    - can: gs_usb: add usb endpoint address detection at driver probe step
    - can: sun4i_can: sun4i_can_err(): call can_change_state() even if cf is NULL
    - can: m_can: m_can_handle_lec_err(): fix {rx,tx}_errors statistics
    - can: ifi_canfd: ifi_canfd_handle_lec_err(): fix {rx,tx}_errors statistics
    - can: hi311x: hi3110_can_ist(): fix {rx,tx}_errors statistics
    - can: sja1000: sja1000_err(): fix {rx,tx}_errors statistics
    - can: sun4i_can: sun4i_can_err(): fix {rx,tx}_errors statistics
    - can: ems_usb: ems_usb_rx_err(): fix {rx,tx}_errors statistics
    - can: f81604: f81604_handle_can_bus_errors(): fix {rx,tx}_errors statistics
    - ipvs: fix UB due to uninitialized stack access in ip_vs_protocol_init()
    - selftests: hid: fix typo and exit code
    - ptp: Add error handling for adjfine callback in ptp_clock_adjtime
    - net/sched: tbf: correct backlog statistic for GSO packets
    - net: hsr: fix fill_frame_info() regression vs VLAN packets
    - platform/x86: asus-wmi: add support for vivobook fan profiles
    - platform/x86: asus-wmi: Fix inconsistent use of thermal policies
    - platform/x86: asus-wmi: Ignore return value when writing thermal policy
    - net/smc: mark optional smcd_ops and check for support when called
    - net/smc: add operations to merge sndbuf with peer DMB
    - net/smc: {at|de}tach sndbuf to peer DMB if supported
    - net/smc: refactoring initialization of smc sock
    - net/qed: allow old cards not supporting "num_images" to work
    - ixgbevf: stop attempting IPSEC offload on Mailbox API 1.5
    - ixgbe: downgrade logging of unsupported VF API version to debug
    - net: sched: fix erspan_opt settings in cls_flower
    - netfilter: nft_set_hash: skip duplicated elements pending gc run
    - netfilter: nft_set_hash: unaligned atomic read on struct nft_set_ext
    - ethtool: Fix wrong mod state in case of verbose and no_mask bitset
    - mlxsw: spectrum_acl_flex_keys: Constify struct mlxsw_afk_element_inst
    - mlxsw: spectrum_acl_flex_keys: Use correct key block on Spectrum-4
    - net/mlx5e: Remove workaround to avoid syndrome for internal port...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-6.11/6.11.0-1006.6 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-nvidia-6.11' to 'verification-done-noble-linux-nvidia-6.11'. If the problem still exists, change the tag 'verification-needed-noble-linux-nvidia-6.11' to 'verification-failed-noble-linux-nvidia-6.11'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws-6.11/6.11.0-1013.14~24.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-aws-6.11' to 'verification-done-noble-linux-aws-6.11'. If the problem still exists, change the tag 'verification-needed-noble-linux-aws-6.11' to 'verification-failed-noble-linux-aws-6.11'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-raspi/6.8.0-1028.32 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-raspi' to 'verification-done-noble-linux-raspi'. If the problem still exists, change the tag 'verification-needed-noble-linux-raspi' to 'verification-failed-noble-linux-raspi'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-intel/6.11.0-1010.10 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-oracular-linux-intel' to 'verification-done-oracular-linux-intel'. If the problem still exists, change the tag 'verification-needed-oracular-linux-intel' to 'verification-failed-oracular-linux-intel'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra/6.8.0-1007.7 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-nvidia-tegra' to 'verification-done-noble-linux-nvidia-tegra'. If the problem still exists, change the tag 'verification-needed-noble-linux-nvidia-tegra' to 'verification-failed-noble-linux-nvidia-tegra'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-fips/6.8.0-72.72+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-fips' to 'verification-done-noble-linux-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-fips' to 'verification-failed-noble-linux-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws-fips/6.8.0-1034.36+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-aws-fips' to 'verification-done-noble-linux-aws-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-aws-fips' to 'verification-failed-noble-linux-aws-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-gcp-fips/6.8.0-1035.37+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-gcp-fips' to 'verification-done-noble-linux-gcp-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-gcp-fips' to 'verification-failed-noble-linux-gcp-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-xilinx/6.8.0-1017.18 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-xilinx' to 'verification-done-noble-linux-xilinx'. If the problem still exists, change the tag 'verification-needed-noble-linux-xilinx' to 'verification-failed-noble-linux-xilinx'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure-fips/6.8.0-1034.39+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-azure-fips' to 'verification-done-noble-linux-azure-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-azure-fips' to 'verification-failed-noble-linux-azure-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!