MGLRU: kswapd uses 100% CPU when MGLRU is enabled and under memory pressure

Bug #2087886 reported by Matthew Ruffell
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Undecided
Unassigned
Noble
Fix Released
Medium
Matthew Ruffell
Oracular
Fix Released
Medium
Matthew Ruffell

Bug Description

BugLink: https://bugs.launchpad.net/bugs/2087886

[Impact]

On systems with MGLRU enabled, which it is by default, if the system is under
memory pressure, and some pages are then allocated, such that it wakes up kswapd
to attempt page reclaim, but the system has enough memory that kswapd doesn't
get OOM killed, and then there are no pages to actually reclaim, kswapd can spin
at 100% endlessly, causing severe performance issues.

What's happening is that lru_gen_shrink_node() unconditionally clears
kswapd_failures, which can prevent kswapd from sleeping and cause 100% kswapd
cpu usage even when kswapd repeatedly fails to make progress in reclaim.

A workaround is to disable MGLRU, and the issue does not occur.

[Fix]

The fix is to only clear kswap_failures in lru_gen_shrink_node() if reclaim
makes some progress, similar to shrink_node().

This was fixed in 6.12-rc4 by the commit:

commit b130ba4a6259f6b64d8af15e9e7ab1e912bcb7ad
Author: Wei Xu <email address hidden>
Date: Mon Oct 14 22:12:11 2024 +0000
Subject: mm/mglru: only clear kswapd_failures if reclaimable
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b130ba4a6259f6b64d8af15e9e7ab1e912bcb7ad

Both Noble and Oracular need this fix.

[Testcase]

The systems with this issue are DPDK compute nodes running OpenStack Yoga on
Jammy. If you leave them for several days, they will hit this issue and their
kswap processes will go to 100% and never drop.

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    846 root 20 0 0 0 0 R 100.0 0.0 1915:16 kswapd0
    846 root 20 0 0 0 0 R 100.0 0.0 1915:16 kswapd0
    846 root 20 0 0 0 0 R 100.0 0.0 1915:17 kswapd0
    846 root 20 0 0 0 0 R 99.0 0.0 1915:18 kswapd0

If you disable MGLRU however, they drop to 0% and sleep immediately.

There is a test kernel available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf390959-test

If you install the test kernel, the system should remain stable, and kswapd will
not go to 100% cpu after some time.

[Where problems could occur]

We are adding a small requirement to clearing kswapd_failures, that we actually
reclaim some memory. If we do actually reclaim some memory, the behaviour is the
same as what we have now, we clear kswapd_failures and continue.

If we don't manage to reclaim any memory, we leave kswapd_failures as is, and
let kswapd sleep, to try again at a later time.

If a regression were to occur, it would affect MGLRU users, which is the default.
A regression would look like issues with kswapd memory reclaim, or lead to high
cpu usage with kswapd.

description: updated
Changed in linux (Ubuntu Noble):
status: New → In Progress
Changed in linux (Ubuntu Oracular):
status: New → In Progress
Changed in linux (Ubuntu Noble):
importance: Undecided → Medium
Changed in linux (Ubuntu Oracular):
importance: Undecided → Medium
Changed in linux (Ubuntu Noble):
assignee: nobody → Matthew Ruffell (mruffell)
Changed in linux (Ubuntu Oracular):
assignee: nobody → Matthew Ruffell (mruffell)
tags: added: noble oracular sts
Changed in linux (Ubuntu):
status: New → Fix Released
Revision history for this message
Matthew Ruffell (mruffell) wrote :
Stefan Bader (smb)
Changed in linux (Ubuntu Oracular):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Noble):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/6.11.0-13.14 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-oracular-linux' to 'verification-done-oracular-linux'. If the problem still exists, change the tag 'verification-needed-oracular-linux' to 'verification-failed-oracular-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-oracular-linux-v2 verification-needed-oracular-linux
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/6.8.0-51.52 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux' to 'verification-done-noble-linux'. If the problem still exists, change the tag 'verification-needed-noble-linux' to 'verification-failed-noble-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-noble-linux-v2 verification-needed-noble-linux
Revision history for this message
Matthew Ruffell (mruffell) wrote :

Performing verification for Noble.

We installed 6.8.0-51-generic to the same DPDK compute nodes running OpenStack Yoga on Jammy. We enabled MGLRU and left them running, and checked 2 days later. Things are still stable, all kswapd processes are normal at 0% CPU.

Things are looking very good.

Marking as verified for Noble, the kernel in proposed2 solves the issue.

tags: added: verification-done-noble-linux
removed: verification-needed-noble-linux
Revision history for this message
Matthew Ruffell (mruffell) wrote :

Performing verification for Oracular.

I didn't have the same real-world testbed for oracular, so I just set up a lab system and enabled MGLRU and left it running stress-ng with a memory stressor for 3 days.

Its still up after 3 days, and kswapd processes are at 0%, so we will call this smoke tested, and likely fixes the issue.

Marking as verified for Oracular.

tags: added: verification-done-oracular-linux
removed: verification-needed-oracular-linux
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 6.11.0-13.14

---------------
linux (6.11.0-13.14) oracular; urgency=medium

  * oracular/linux: 6.11.0-13.14 -proposed tracker (LP: #2090463)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log

  * MGLRU: kswapd uses 100% CPU when MGLRU is enabled and under memory pressure
    (LP: #2087886)
    - mm/mglru: only clear kswapd_failures if reclaimable

  * CVE-2024-50264
    - vsock/virtio: Initialization of the dangling pointer occurring in vsk->trans

  * CVE-2024-53057
    - net/sched: stop qdisc_tree_reduce_backlog on TC_H_ROOT

  * CVE-2024-49967
    - ext4: no need to continue when the number of entries is 1

 -- Manuel Diewald <email address hidden> Sat, 30 Nov 2024 23:09:01 +0100

Changed in linux (Ubuntu Oracular):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 6.8.0-51.52

---------------
linux (6.8.0-51.52) noble; urgency=medium

  * noble/linux: 6.8.0-51.52 -proposed tracker (LP: #2090369)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log
    - [Packaging] update variants

  * MGLRU: kswapd uses 100% CPU when MGLRU is enabled and under memory pressure
    (LP: #2087886)
    - mm/mglru: only clear kswapd_failures if reclaimable

  * CVE-2024-50264
    - vsock/virtio: Initialization of the dangling pointer occurring in vsk->trans

  * CVE-2024-53057
    - net/sched: stop qdisc_tree_reduce_backlog on TC_H_ROOT

  * CVE-2024-49967
    - ext4: no need to continue when the number of entries is 1

 -- Manuel Diewald <email address hidden> Sat, 30 Nov 2024 19:21:46 +0100

Changed in linux (Ubuntu Noble):
status: Fix Committed → Fix Released
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-realtime-6.11/6.11.0-1003.3~24.04.2 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-realtime-6.11' to 'verification-done-noble-linux-realtime-6.11'. If the problem still exists, change the tag 'verification-needed-noble-linux-realtime-6.11' to 'verification-failed-noble-linux-realtime-6.11'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-noble-linux-realtime-6.11-v2 verification-needed-noble-linux-realtime-6.11
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-nvidia-6.11/6.11.0-1003.3 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-nvidia-6.11' to 'verification-done-noble-linux-nvidia-6.11'. If the problem still exists, change the tag 'verification-needed-noble-linux-nvidia-6.11' to 'verification-failed-noble-linux-nvidia-6.11'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-noble-linux-nvidia-6.11-v2 verification-needed-noble-linux-nvidia-6.11
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-intel/6.11.0-1007.7 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-oracular-linux-intel' to 'verification-done-oracular-linux-intel'. If the problem still exists, change the tag 'verification-needed-oracular-linux-intel' to 'verification-failed-oracular-linux-intel'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-oracular-linux-intel-v2 verification-needed-oracular-linux-intel
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.