Regular D-state processes impacting LXD containers
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| linux (Ubuntu) |
Undecided
|
Unassigned | ||
| Xenial |
Medium
|
Unassigned | ||
| Bionic |
Medium
|
Unassigned |
Bug Description
[Impact]
* Systems running under memory pressure may hit stalls in the
order of seconds to minutes in systemd-logind and lxd mount
operations (e.g., ZFS backend), which get stuck in D state.
* The processes stuck in D state have a common stack trace,
(cat /proc/PID/stack) all blocked in register_
* The fix checks in shrink_slab() (shrinkers are called under
memory pressure) for contention/usage of the semaphore used
by register_shrinker() and returns early in that case.
This allows the register_shrinker() callers to unblock,
and not stall until the shrink operation releases that lock.
[Test Case]
* In a system under memory pressure, specifically having the
memory shrinkers being called often and taking time to run,
perform mount operations (or other operations that acquire
the shrinker_rwsem semaphore).
* The user who reported the problem has verified the fix in
systems that exhibted the problem often (sometimes daily),
and tells it resolves the problem.
[Regression Potential]
* Low. The fix just returns early from slab memory shrinker
if there's usage/contention for 'shrinker_rwsem'.
* In some scenarios, this may cause the slab memory shrinker
to require more invocations to actually finish and potentially
release memory, but this seems minor since other shrinkers can
release memory as well, and compared to the fact that this fix
allows other applications to make progress / continue to run,
which would otherwise be stalled.
[Other Info]
* This patch is already applied in Cosmic and later (v4.16+).
It is needed only in Xenial and Bionic at this time.
Changed in linux (Ubuntu): | |
status: | New → Invalid |
Changed in linux (Ubuntu Xenial): | |
status: | New → Confirmed |
Changed in linux (Ubuntu Bionic): | |
status: | New → Confirmed |
Mauricio Faria de Oliveira (mfo) wrote : | #1 |
Mauricio Faria de Oliveira (mfo) wrote : | #2 |
[X/B][PATCH] mm: do not stall register_shrinker()
https:/
Changed in linux (Ubuntu Xenial): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Bionic): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Xenial): | |
status: | Confirmed → Fix Committed |
Changed in linux (Ubuntu Bionic): | |
status: | Confirmed → Fix Committed |
Brad Figg (brad-figg) wrote : | #3 |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/
tags: | added: verification-needed-xenial |
Brad Figg (brad-figg) wrote : | #4 |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/
tags: | added: verification-needed-bionic |
Mauricio Faria de Oliveira (mfo) wrote : | #5 |
Marking X/B as verification done.
The user reports the issue occurs much less often now.
Apparently that environment hits some corner case or this may still be expected sometimes under memory pressure (i.e., one big shrinking operation acquired the lock and must finish).
Nonetheless, the fix reduced the frequency the problem occurs, so it does address most of that and is beneficial on its own.
As discussed with @klebers on IRC, in this scenario we agreed to ship the fix, and address the pending behavior (if an actual issue) with additional fixes later.
cheers,
Mauricio
tags: |
added: verification-done-bionic verification-done-xenial removed: verification-needed-bionic verification-needed-xenial |
Launchpad Janitor (janitor) wrote : | #6 |
This bug was fixed in the package linux - 4.4.0-145.171
---------------
linux (4.4.0-145.171) xenial; urgency=medium
* linux: 4.4.0-145.171 -proposed tracker (LP: #1821724)
* linux-generic should depend on linux-base >=4.1 (LP: #1820419)
- [Packaging] Fix linux-base dependency
linux (4.4.0-144.170) xenial; urgency=medium
* linux: 4.4.0-144.170 -proposed tracker (LP: #1819660)
* Packaging resync (LP: #1786013)
- [Packaging] resync getabis
- [Packaging] update helper scripts
- [Packaging] resync retpoline extraction
* C++ demangling support missing from perf (LP: #1396654)
- [Packaging] fix a mistype
* CVE-2019-9213
- mm: enforce min addr even if capable() in expand_downwards()
* CVE-2019-3460
- Bluetooth: Check L2CAP option sizes returned from l2cap_get_conf_opt
* Xenial update: 4.4.176 upstream stable release (LP: #1818815)
- net: fix IPv6 prefix route residue
- vsock: cope with memory allocation failure at socket creation time
- hwmon: (lm80) Fix missing unlock on error in set_fan_div()
- net: Fix for_each_
- net: Add header for usage of fls64()
- tcp: tcp_v4_err() should be more careful
- net: Do not allocate page fragments that are not skb aligned
- tcp: clear icsk_backoff in tcp_write_
- vxlan: test dev->flags & IFF_UP before calling netif_rx()
- net: stmmac: Fix a race in EEE enable callback
- net: ipv4: use a dedicated counter for icmp_v4 redirect packets
- x86: livepatch: Treat R_X86_64_PLT32 as R_X86_64_PC32
- mfd: as3722: Handle interrupts on suspend
- mfd: as3722: Mark PM functions as __maybe_unused
- net/x25: do not hold the cpu too long in x25_new_lci()
- mISDN: fix a race in dev_expire_timer()
- ax25: fix possible use-after-free
- Linux 4.4.176
* sky2 ethernet card don't work after returning from suspension
(LP: #1798921) // Xenial update: 4.4.176 upstream stable release
(LP: #1818815)
- sky2: Increase D3 delay again
* Xenial update: 4.4.175 upstream stable release (LP: #1818813)
- drm/bufs: Fix Spectre v1 vulnerability
- staging: iio: adc: ad7280a: handle error from __ad7280_read32()
- ASoC: Intel: mrfld: fix uninitialized variable access
- scsi: lpfc: Correct LCB RJT handling
- ARM: 8808/1: kexec:offline panic_smp_self_stop CPU
- dlm: Don't swamp the CPU with callbacks queued during recovery
- x86/PCI: Fix Broadcom CNB20LE unintended sign extension (redux)
- powerpc/pseries: add of_node_put() in dlpar_detach_node()
- serial: fsl_lpuart: clear parity enable bit when disable parity
- ptp: check gettime64 return code in PTP_SYS_OFFSET ioctl
- staging:iio:ad2s90: Make probe handle spi_setup failure
- staging: iio: ad7780: update voltage on read
- ARM: OMAP2+: hwmod: Fix some section annotations
- modpost: validate symbol names also in find_elf_symbol
- perf tools: Add Hygon Dhyana support
- soc/tegra: Don't leak device tree node reference
- f2fs: move dir data flush to write checkpoint process
- f2fs: fix wrong return value of f2fs_acl_create
- sunvdc: Do not spin in an infin...
Changed in linux (Ubuntu Xenial): | |
status: | Fix Committed → Fix Released |
Launchpad Janitor (janitor) wrote : | #7 |
This bug was fixed in the package linux - 4.15.0-47.50
---------------
linux (4.15.0-47.50) bionic; urgency=medium
* linux: 4.15.0-47.50 -proposed tracker (LP: #1819716)
* Packaging resync (LP: #1786013)
- [Packaging] resync getabis
- [Packaging] update helper scripts
- [Packaging] resync retpoline extraction
* C++ demangling support missing from perf (LP: #1396654)
- [Packaging] fix a mistype
* arm-smmu-v3 arm-smmu-v3.3.auto: CMD_SYNC timeout (LP: #1818162)
- iommu/arm-smmu-v3: Fix unexpected CMD_SYNC timeout
* Crash in nvme_irq_check() when using threaded interrupts (LP: #1818747)
- nvme-pci: fix out of bounds access in nvme_cqe_pending
* CVE-2019-9213
- mm: enforce min addr even if capable() in expand_downwards()
* CVE-2019-3460
- Bluetooth: Check L2CAP option sizes returned from l2cap_get_conf_opt
* amdgpu with mst WARNING on blanking (LP: #1814308)
- drm/amd/display: Don't use dc_link in link_encoder
- drm/amd/display: Move wait for hpd ready out from edp power control.
- drm/amd/display: eDP sequence BL off first then DP blank.
- drm/amd/display: Fix unused variable compilation error
- drm/amd/display: Fix warning about misaligned code
- drm/amd/display: Fix MST dp_blank REG_WAIT timeout
* tun/tap: unable to manage carrier state from userland (LP: #1806392)
- tun: implement carrier change
* CVE-2019-8980
- exec: Fix mem leak in kernel_read_file
* raw_skew in timer from the ubuntu_
(LP: #1811194)
- selftest: timers: Tweak raw_skew to SKIP when ADJ_OFFSET/other clock
adjustments are in progress
* [Packaging] Allow overlay of config annotations (LP: #1752072)
- [Packaging] config-check: Add an include directive
* CVE-2019-7308
- bpf: move {prev_,}insn_idx into verifier env
- bpf: move tmp variable into ax register in interpreter
- bpf: enable access to ax register also from verifier rewrite
- bpf: restrict map value pointer arithmetic for unprivileged
- bpf: restrict stack pointer arithmetic for unprivileged
- bpf: restrict unknown scalars of mixed signed bounds for unprivileged
- bpf: fix check_map_access smin_value test when pointer contains offset
- bpf: prevent out of bounds speculation on pointer arithmetic
- bpf: fix sanitation of alu op with pointer / scalar type from different
paths
- bpf: add various test cases to selftests
* CVE-2017-5753
- bpf: properly enforce index mask to prevent out-of-bounds speculation
- bpf: fix inner map masking to prevent oob under speculation
* BPF: kernel pointer leak to unprivileged userspace (LP: #1815259)
- bpf/verifier: disallow pointer subtraction
* squashfs hardening (LP: #1816756)
- squashfs: more metadata hardening
- squashfs metadata 2: electric boogaloo
- squashfs: more metadata hardening
- Squashfs: Compute expected length from inode size rather than block length
* efi/arm/arm64: Allow SetVirtualAddre
- efi/arm/arm64: Allow SetVirtualAddre
* Update ENA driver to version 2.0.3K (LP: #1816806)...
Changed in linux (Ubuntu Bionic): | |
status: | Fix Committed → Fix Released |
Also verified this with stress-ng's memory stressor class.
The system gets under memory pressure to the point of
calling shrink_slab() 10s to 100s thousand times a second
(verified with ftrace; function tracer; shrink_slab filter).
$ sudo stress-ng --class memory --all -1 &
# echo shrink_slab > /sys/kernel/ debug/tracing/ set_ftrace_ filter debug/tracing/ current_ tracer debug/tracing/ trace_pipe | awk '{print $4}' | cut -d. -f1 | uniq -c
# echo function > /sys/kernel/
# cat /sys/kernel/
24384 998
38792 999
1 EVENTS]
22883 999
1 EVENTS]
1450 999
113849 1000
117025 1001
67285 1002
75768 1003
1 EVENTS]
14496 1003
170817 1004
98136 1005