TCP memory leak, slow network (arm64)

[Lev Petrushchak]

Hello! 👋

We have Ubuntu OS-based servers running in both AWS and Azure clouds. These servers are handling thousands of connections, and we've been experiencing issues with TCP memory usage since upgrading to Ubuntu 22.04.3 from 22.04.2.

$ cat /proc/net/sockstat
sockets: used 6642
TCP: inuse 5962 orphan 0 tw 292 alloc 6008 mem 128989
UDP: inuse 5 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

As shown in the output below, even after stopping all possible services and closing all open connections, the TCP memory usage remains high and only decreases very slowly.

$ cat /proc/net/sockstat
sockets: used 138
TCP: inuse 2 orphan 0 tw 0 alloc 3 mem 128320
UDP: inuse 3 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

I have attached a screenshot of linear TCP memory usage growth, which we believe may indicate a TCP memory leak

When net.ipv4.tcp_mem limit is reached, it causes network slowdown

We've never had these issues before, and the only solution we've found so far is to reboot the node. Do you have any suggestions on how to troubleshoot further?

Thank you for any help or guidance you can provide!

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: linux-image-6.2.0-1015-aws 6.2.0-1015.15~22.04.1
ProcVersionSignature: Ubuntu 6.2.0-1015.15~22.04.1-aws 6.2.16
Uname: Linux 6.2.0-1015-aws aarch64
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: arm64
CasperMD5CheckResult: unknown
CloudArchitecture: aarch64
CloudID: aws
CloudName: aws
CloudPlatform: ec2
CloudRegion: us-west-2
CloudSubPlatform: metadata (http://169.254.169.254)
Date: Mon Dec 4 13:13:08 2023
Ec2AMI: ami-095a68e28e781dfe1
Ec2AMIManifest: (unknown)
Ec2Architecture: arm64
Ec2AvailabilityZone: us-west-2b
Ec2Imageid: ami-095a68e28e781dfe1
Ec2InstanceType: m7g.large
Ec2Instancetype: m7g.large
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
Ec2Region: us-west-2
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
RebootRequiredPkgs: Error: path contained symlinks.
SourcePackage: linux-signed-aws-6.2
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ryan Huddleston (rshuddleston)]

affects 6.2.0-1011-aws -> 6.2.0-1016-aws, 6.2.0-1011-azure -> 6.2.0-1016-azure kernels, did not affect 5.19 and prior kernels in 22.04.2

[Lev Petrushchak (sabretus)]

We tried to downgrade the kernel in Ubuntu 22.04.3 from 6.2 to 5.15 by installing linux-image-{azure|aws}-lts-22.04 and it works, LTS 5.15 kernel is not affected by this problem.

[Lev Petrushchak (sabretus)]

We have confirmed that this issue only related to ARM64 architecture

[Jonathan Heathcote (mossblaser)]

As another "me too" situation, I'm seeing the same phenomenon, though on Rocky 9 rather than Ubuntu and on older kernels (5.14). Reporting details here on the off chance this provides some insight.

Hardware: Ampere Altra Max 128 cores (aarch64), ConnectX6-DX NICs (2 x dual 100G port)
Kernel versions tested: 5.15 (Rocky 9 native kernel) and 6.8 (elrepo kernel), both configured with 64 KB pages
OS: Rocky Linux 9
Software: nginx serving ~90k HTTPS clients at ~350 GBit/s (a synthetic load test)
Bare-metal (no virtualisation).

In my test environment, ~90k HTTPS connections are opened (and reused via keepalive) and used to stream ~350 GBit/s of traffic to a cluster of load generators. In this scenario, TCP memory gradually creeps up until reaching the memory pressure threshold in /proc/sys/net/ipv4/tcp_mem (243890 pages, or 15.6 GB in this system). At this point memory usage growth actually increases slightly (and increased CPU load and response times are also observed). The system eventually reaches the ultimate limit (365832 pages, or 23.4 GB) at which point most connections fail and all requests receive very slow responses.

Closing all connections or restarting nginx does not free up the memory, only a reboot resolves the situation -- as reported above already.

Leaked memory appears to persist even if all connections are closed prior to hitting any of the above limits.

Unfortunately I don't yet have any ideas on how to fix this but would be glad to hear (and will share) any insights about what might be going on here!

[Jonathan Heathcote (mossblaser)]

I've been digging into this and this appears to be a regression introduced by the following patch https://github.com/torvalds/linux/commit/3cd3399dd7a84 which was first released in Linux 6.0.0.

The bug is not a memory leak but rather a bug in how memory usage is counted. Excess memory is not actually being consumed, though the bug is still fatal since the counter controls Linux's memory pressure logic.

The (apparently) responsible patch is a performance optimisation which attempts to reduce the frequency of writes to the system-wide counter which (I suspect) is subtly misusing some atomic operation on ARM. If you undo this patch in a recent kernel, the bug disappears.

I am currently working on a detailed bug report for the relevant Kernel maintainers.

NB: It appears that the "5.15" kernel shipped by Rocky (and RHEL) includes a back-port of this bug, hence my seeing the bug in that kernel version on Rocky Linux. A non-RedHat-patched vanilla build of 5.15 does not exhibit the bug in my system either.

[Lev Petrushchak (sabretus)]

Hi Jonathan,

Thanks for all the information and the deep dive into this issue! Please post the link to the kernel bug report here when it's ready.

[Jonathan Heathcote (mossblaser)]

The bug report can be found here:

https://<email address hidden>/

The subsequently produced patch (not by me!) to fix this can be found here:

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=3584718cf2ec

I've also verified that the above patch does fix the problem at least in my case!

[Philip Cox (philcox)]

I've looked at this issue in some detail now. The initial memory leak was introduced in linux kernel version v6.0, and the fix was introduced in the v6.9 kernel version. This means that the 6.5, and 6.8 based ubuntu kernels will contain this memory leak.

I will submit a fix for the generic 6.8, and generic 6.5 kernels so all ubuntu kernels will receive the fix.

For reference: The commit that introduces this issue is 3cd3399dd7a84ada85cb839989cdf7310e302c7d

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=3cd3399dd7a84ada85cb839989cdf7310e302c7d

And the commit I will be working with to provide the fix is the stable backport to the 6.8.y stable kernel which is: d2fa3493811ecd49f1581940111ccf475fa5cd38

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=d2fa3493811ecd49f1581940111ccf475fa5cd38

[Philip Cox (philcox)]

Two patches are actually needed for this fix. From the 6.8.y stable branch, the following two patches cherry-pick and build clean on the generic ubuntu mantic (6.5), and noble (6.8) kernels. I will perform some more testing and send these out for review.

The two patches are:

d2fa3493811ecd49f1581940111ccf475fa5cd38 net: fix sk_memory_allocated_{add|sub} vs softirqs
e830c804e26733fb09219cb9b92f167715c3b8a0 net: make SK_MEMORY_PCPU_RESERV tunable

[Philip Cox (philcox)]

The review for this change has been sent out. It is: https://lists.ubuntu.com/archives/kernel-team/2024-May/151087.html

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/6.5.0-44.44 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-mantic-linux' to 'verification-done-mantic-linux'. If the problem still exists, change the tag 'verification-needed-mantic-linux' to 'verification-failed-mantic-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/6.8.0-38.38 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux' to 'verification-done-noble-linux'. If the problem still exists, change the tag 'verification-needed-noble-linux' to 'verification-failed-noble-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Nicolai Antiferov (nklya)]

Hi,
I’m wondering if this bugfix will be backported for Jammy / 22.04?
Right now as I see only Mantic/Noble is planned in status.

Jammy has 6.2/6.5 kernels and they’re affected by this.

P.S. Also, it feels that priority might be higher. In some cases it creates pretty serious network related problems .

[Philip Cox (philcox)]

@nklya

The 6.5 and 6.8 based jammy kernels have received this patch as well. The 6.2 based jammy kernel is no longer supported so it will not be receiving these patches.

The way that the jammy 6.5/6.8 kernels work are, they are backports of the newer series kernels. So, for noble (24.04) for example, the noble 6.8 kernel has been back ported to jammy and is the jammy 6.8 based kernel. So, when changes go in to the noble 6.8 kernel, they automatically make it into the jammy 6.8 kernel.

So this was targeted to noble and mantic, with the understanding that the changes would end up in the jammy 6.8 and 6.5 based kernel backports.

[Nicolai Antiferov (nklya)]

Cool, thank you for details, all clear now.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-lowlatency/6.8.0-1009.9.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-nvidia-lowlatency' to 'verification-done-noble-linux-nvidia-lowlatency'. If the problem still exists, change the tag 'verification-needed-noble-linux-nvidia-lowlatency' to 'verification-failed-noble-linux-nvidia-lowlatency'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 6.5.0-44.44

---------------
linux (6.5.0-44.44) mantic; urgency=medium

  * mantic/linux: 6.5.0-44.44 -proposed tracker (LP: #2068341)

  * Packaging resync (LP: #1786013)
    - [Packaging] debian.master/dkms-versions -- update from kernel-versions
      (main/2024.06.10)

  * Some DUTs can't boot up after installing the proposed kernel on Mantic
    (LP: #2061940)
    - SAUCE: Revert "x86/efistub: Use 1:1 file:memory mapping for PE/COFF .compat
      section"
    - SAUCE: Revert "x86/boot: Increase section and file alignment to 4k/512"
    - SAUCE: Revert "x86/boot: Split off PE/COFF .data section"
    - SAUCE: Revert "x86/boot: Drop PE/COFF .reloc section"
    - SAUCE: Revert "x86/boot: Construct PE/COFF .text section from assembler"
    - SAUCE: Revert "x86/boot: Derive file size from _edata symbol"
    - SAUCE: Revert "x86/boot: Define setup size in linker script"
    - SAUCE: Revert "x86/boot: Set EFI handover offset directly in header asm"
    - SAUCE: Revert "x86/boot: Grab kernel_info offset from zoffset header
      directly"
    - SAUCE: Revert "x86/boot: Drop redundant code setting the root device"
    - SAUCE: Revert "x86/boot: Drop references to startup_64"
    - SAUCE: Revert "x86/boot: Omit compression buffer from PE/COFF image memory
      footprint"
    - SAUCE: Revert "x86/boot: Remove the 'bugger off' message"
    - SAUCE: Revert "x86/efi: Drop alignment flags from PE section headers"
    - SAUCE: Revert "x86/efi: Drop EFI stub .bss from .data section"

  * CVE-2023-52880
    - tty: n_gsm: require CAP_NET_ADMIN to attach N_GSM0710 ldisc

  * i915 cannot probe successfully on HP ZBook Power 16 G11 (LP: #2067883)
    - drm/i915/mtl: Remove the 'force_probe' requirement for Meteor Lake

  * CVE-2024-26838
    - RDMA/irdma: Fix KASAN issue with tasklet

  * mtk_t7xx WWAN module fails to probe with: Invalid device status 0x1
    (LP: #2049358)
    - Revert "UBUNTU: SAUCE: net: wwan: t7xx: PCIe reset rescan"
    - Revert "UBUNTU: SAUCE: net: wwan: t7xx: Add AP CLDMA"
    - net: wwan: t7xx: Add AP CLDMA
    - wwan: core: Add WWAN fastboot port type
    - net: wwan: t7xx: Add sysfs attribute for device state machine
    - net: wwan: t7xx: Infrastructure for early port configuration
    - net: wwan: t7xx: Add fastboot WWAN port

  * TCP memory leak, slow network (arm64) (LP: #2045560)
    - net: make SK_MEMORY_PCPU_RESERV tunable
    - net: fix sk_memory_allocated_{add|sub} vs softirqs

  * CVE-2024-26923
    - af_unix: Do not use atomic ops for unix_sk(sk)->inflight.
    - af_unix: Fix garbage collector racing against connect()

  * Add support for Quectel EM160R-GL modem [1eac:100d] (LP: #2063399)
    - Add support for Quectel EM160R-GL modem

  * Add support for Quectel RM520N-GL modem [1eac:1007] (LP: #2063529)
    - Add support for Quectel RM520N-GL modem
    - Add support for Quectel RM520N-GL modem

  * [SRU][22.04.4]: megaraid_sas: Critical Bug Fixes (LP: #2046722)
    - scsi: megaraid_sas: Log message when controller reset is requested but not
      issued
    - scsi: megaraid_sas: Driver version update to 07.727.03.00-rc1

  * Fix the RTL8852CE BT FW Crash based on SER false alar...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 6.8.0-38.38

---------------
linux (6.8.0-38.38) noble; urgency=medium

  * noble/linux: 6.8.0-38.38 -proposed tracker (LP: #2068318)

  * race_sched in ubuntu_stress_smoke_test will cause kernel panic on 6.8 with
    Azure Standard_A2_v2 instance (LP: #2068024)
    - sched/eevdf: Prevent vlag from going out of bounds in reweight_eevdf()

  * Noble: btrfs: re-introduce 'norecovery' mount option (LP: #2068591)
    - btrfs: re-introduce 'norecovery' mount option

  * Fix system hang while entering suspend with AMD Navi3x graphics
    (LP: #2063417)
    - drm/amdgpu/mes: fix use-after-free issue

  * Noble update: v6.8.8 upstream stable release (LP: #2068087)
    - io_uring: Fix io_cqring_wait() not restoring sigmask on get_timespec64()
      failure
    - drm/i915/cdclk: Fix voltage_level programming edge case
    - Revert "vmgenid: emit uevent when VMGENID updates"
    - SUNRPC: Fix rpcgss_context trace event acceptor field
    - selftests/ftrace: Limit length in subsystem-enable tests
    - random: handle creditable entropy from atomic process context
    - scsi: core: Fix handling of SCMD_FAIL_IF_RECOVERING
    - net: usb: ax88179_178a: avoid writing the mac address before first reading
    - btrfs: do not wait for short bulk allocation
    - btrfs: zoned: do not flag ZEROOUT on non-dirty extent buffer
    - r8169: fix LED-related deadlock on module removal
    - r8169: add missing conditional compiling for call to r8169_remove_leds
    - scsi: ufs: qcom: Add missing interconnect bandwidth values for Gear 5
    - netfilter: nf_tables: Fix potential data-race in __nft_expr_type_get()
    - netfilter: nf_tables: Fix potential data-race in __nft_obj_type_get()
    - netfilter: br_netfilter: skip conntrack input hook for promisc packets
    - netfilter: nft_set_pipapo: constify lookup fn args where possible
    - netfilter: nft_set_pipapo: walk over current view on netlink dump
    - netfilter: flowtable: validate pppoe header
    - netfilter: flowtable: incorrect pppoe tuple
    - af_unix: Call manage_oob() for every skb in unix_stream_read_generic().
    - af_unix: Don't peek OOB data without MSG_OOB.
    - net: sparx5: flower: fix fragment flags handling
    - net/mlx5: Lag, restore buckets number to default after hash LAG deactivation
    - net/mlx5: Restore mistakenly dropped parts in register devlink flow
    - net/mlx5e: Prevent deadlock while disabling aRFS
    - net: change maximum number of UDP segments to 128
    - octeontx2-pf: fix FLOW_DIS_IS_FRAGMENT implementation
    - selftests/tcp_ao: Make RST tests less flaky
    - selftests/tcp_ao: Zero-init tcp_ao_info_opt
    - selftests/tcp_ao: Fix fscanf() call for format-security
    - selftests/tcp_ao: Printing fixes to confirm with format-security
    - net: stmmac: Apply half-duplex-less constraint for DW QoS Eth only
    - net: stmmac: Fix max-speed being ignored on queue re-init
    - net: stmmac: Fix IP-cores specific MAC capabilities
    - ice: tc: check src_vsi in case of traffic from VF
    - ice: tc: allow zero flags in parsing tc flower
    - ice: Fix checking for unsupported keys on non-tunnel device
    - tun: limit printing rate whe...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-gke/6.8.0-1006.9 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-gke' to 'verification-done-noble-linux-gke'. If the problem still exists, change the tag 'verification-needed-noble-linux-gke' to 'verification-failed-noble-linux-gke'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia/6.8.0-1009.9 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-nvidia' to 'verification-done-noble-linux-nvidia'. If the problem still exists, change the tag 'verification-needed-noble-linux-nvidia' to 'verification-failed-noble-linux-nvidia'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-ibm/6.8.0-1008.8 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-ibm' to 'verification-done-noble-linux-ibm'. If the problem still exists, change the tag 'verification-needed-noble-linux-ibm' to 'verification-failed-noble-linux-ibm'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-hwe-6.8/6.8.0-38.38~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-hwe-6.8' to 'verification-done-jammy-linux-hwe-6.8'. If the problem still exists, change the tag 'verification-needed-jammy-linux-hwe-6.8' to 'verification-failed-jammy-linux-hwe-6.8'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure/6.5.0-1024.25 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-mantic-linux-azure' to 'verification-done-mantic-linux-azure'. If the problem still exists, change the tag 'verification-needed-mantic-linux-azure' to 'verification-failed-mantic-linux-azure'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws/6.5.0-1023.23 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-mantic-linux-aws' to 'verification-done-mantic-linux-aws'. If the problem still exists, change the tag 'verification-needed-mantic-linux-aws' to 'verification-failed-mantic-linux-aws'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-laptop/6.5.0-1019.22 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-mantic-linux-laptop' to 'verification-done-mantic-linux-laptop'. If the problem still exists, change the tag 'verification-needed-mantic-linux-laptop' to 'verification-failed-mantic-linux-laptop'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-oracle/6.5.0-1026.26 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-mantic-linux-oracle' to 'verification-done-mantic-linux-oracle'. If the problem still exists, change the tag 'verification-needed-mantic-linux-oracle' to 'verification-failed-mantic-linux-oracle'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-6.5/6.5.0-1023.24 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia-6.5' to 'verification-done-jammy-linux-nvidia-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia-6.5' to 'verification-failed-jammy-linux-nvidia-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-oem-6.5/6.5.0-1026.27 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-oem-6.5' to 'verification-done-jammy-linux-oem-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-oem-6.5' to 'verification-failed-jammy-linux-oem-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-hwe-6.5/6.5.0-44.44~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-hwe-6.5' to 'verification-done-jammy-linux-hwe-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-hwe-6.5' to 'verification-failed-jammy-linux-hwe-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure-6.8/6.8.0-1010.10~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-azure-6.8' to 'verification-done-jammy-linux-azure-6.8'. If the problem still exists, change the tag 'verification-needed-jammy-linux-azure-6.8' to 'verification-failed-jammy-linux-azure-6.8'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-lowlatency-hwe-6.8/6.8.0-38.38.1~22.04.2 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-lowlatency-hwe-6.8' to 'verification-done-jammy-linux-lowlatency-hwe-6.8'. If the problem still exists, change the tag 'verification-needed-jammy-linux-lowlatency-hwe-6.8' to 'verification-failed-jammy-linux-lowlatency-hwe-6.8'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-gcp-6.5/6.5.0-1024.26~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-gcp-6.5' to 'verification-done-jammy-linux-gcp-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-gcp-6.5' to 'verification-failed-jammy-linux-gcp-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-raspi-realtime/6.8.0-2006.6 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-raspi-realtime' to 'verification-done-noble-linux-raspi-realtime'. If the problem still exists, change the tag 'verification-needed-noble-linux-raspi-realtime' to 'verification-failed-noble-linux-raspi-realtime'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-ibm-6.8/6.8.0-1008.8~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-ibm-6.8' to 'verification-done-jammy-linux-ibm-6.8'. If the problem still exists, change the tag 'verification-needed-jammy-linux-ibm-6.8' to 'verification-failed-jammy-linux-ibm-6.8'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-lowlatency/6.5.0-44.44.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-mantic-linux-lowlatency' to 'verification-done-mantic-linux-lowlatency'. If the problem still exists, change the tag 'verification-needed-mantic-linux-lowlatency' to 'verification-failed-mantic-linux-lowlatency'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-starfive/6.5.0-1017.18 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-mantic-linux-starfive' to 'verification-done-mantic-linux-starfive'. If the problem still exists, change the tag 'verification-needed-mantic-linux-starfive' to 'verification-failed-mantic-linux-starfive'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-riscv/6.5.0-44.44.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-mantic-linux-riscv' to 'verification-done-mantic-linux-riscv'. If the problem still exists, change the tag 'verification-needed-mantic-linux-riscv' to 'verification-failed-mantic-linux-riscv'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws-6.8/6.8.0-1011.12~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-aws-6.8' to 'verification-done-jammy-linux-aws-6.8'. If the problem still exists, change the tag 'verification-needed-jammy-linux-aws-6.8' to 'verification-failed-jammy-linux-aws-6.8'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-6.8/6.8.0-1009.9~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia-6.8' to 'verification-done-jammy-linux-nvidia-6.8'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia-6.8' to 'verification-failed-jammy-linux-nvidia-6.8'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-riscv/6.8.0-38.38.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-riscv' to 'verification-done-noble-linux-riscv'. If the problem still exists, change the tag 'verification-needed-noble-linux-riscv' to 'verification-failed-noble-linux-riscv'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-oracle/6.8.0-1008.8 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-oracle' to 'verification-done-noble-linux-oracle'. If the problem still exists, change the tag 'verification-needed-noble-linux-oracle' to 'verification-failed-noble-linux-oracle'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Nicolai Antiferov (nklya)]

@philcox you mentioned here that jammy 6.5 kernel will be backported as well.
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2045560/comments/17

However, in list of updates coming to this ticket I see only “jammy-linux-aws-6.8”. Will it be added later?

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-riscv-6.8/6.8.0-38.38.1~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-riscv-6.8' to 'verification-done-jammy-linux-riscv-6.8'. If the problem still exists, change the tag 'verification-needed-jammy-linux-riscv-6.8' to 'verification-failed-jammy-linux-riscv-6.8'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-oracle-6.5/6.5.0-1026.26~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-oracle-6.5' to 'verification-done-jammy-linux-oracle-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-oracle-6.5' to 'verification-failed-jammy-linux-oracle-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-lowlatency-hwe-6.5/6.5.0-44.44.1~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-lowlatency-hwe-6.5' to 'verification-done-jammy-linux-lowlatency-hwe-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-lowlatency-hwe-6.5' to 'verification-failed-jammy-linux-lowlatency-hwe-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws-6.5/6.5.0-1023.23~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-aws-6.5' to 'verification-done-jammy-linux-aws-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-aws-6.5' to 'verification-failed-jammy-linux-aws-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Philip Cox (philcox)]

@nklya

> you mentioned here that jammy 6.5 kernel will be backported as well.
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2045560/comments/17
>
> However, in list of updates coming to this ticket I see only “jammy-linux-aws-6.8”. Will it be added later?

Funnily enough, I looked this up yesterday for someone else, so I have all the info handy...

So, for 6.5 based kernels, it is in 6.5.0-44.44 and for 6.8 based kernels, it is in 6.8.0-38.38 of the generic Ubuntu kernels.

If you would like the AWS specific kernels, those versions are 6.5.0-1023.23 for the 6.5 based mantic kernel, and 6.8.0-1011.11 for the 6.8 based noble kernel.

If you would like the 6.5 or 6.8 based jammy kernel, look for the versionjammy version that matches or is newer than the mantic or noble equivalent. For the 6.5 jammy kernel, it looks like you want: 6.5.0-1023.23~22.04.1

And it looks like this is the version in the -updates pocket. You should be able to install the AWS kernel, or the generic version of the kernel to test this.

----

And I just noticed that the kernel-bot posted much of the same info I did.