Regular D-state processes impacting LXD containers

[Mauricio Faria de Oliveira]

[Impact]

 * Systems running under memory pressure may hit stalls in the
   order of seconds to minutes in systemd-logind and lxd mount
   operations (e.g., ZFS backend), which get stuck in D state.

 * The processes stuck in D state have a common stack trace,
   (cat /proc/PID/stack) all blocked in register_shrinker().

 * The fix checks in shrink_slab() (shrinkers are called under
   memory pressure) for contention/usage of the semaphore used
   by register_shrinker() and returns early in that case.

   This allows the register_shrinker() callers to unblock,
   and not stall until the shrink operation releases that lock.

[Test Case]

 * In a system under memory pressure, specifically having the
   memory shrinkers being called often and taking time to run,
   perform mount operations (or other operations that acquire
   the shrinker_rwsem semaphore).

 * The user who reported the problem has verified the fix in
   systems that exhibted the problem often (sometimes daily),
   and tells it resolves the problem.

[Regression Potential]

 * Low. The fix just returns early from slab memory shrinker
   if there's usage/contention for 'shrinker_rwsem'.

 * In some scenarios, this may cause the slab memory shrinker
   to require more invocations to actually finish and potentially
   release memory, but this seems minor since other shrinkers can
   release memory as well, and compared to the fact that this fix
   allows other applications to make progress / continue to run,
   which would otherwise be stalled.

[Other Info]

 * This patch is already applied in Cosmic and later (v4.16+).
   It is needed only in Xenial and Bionic at this time.

[Mauricio Faria de Oliveira (mfo)]

Also verified this with stress-ng's memory stressor class.

The system gets under memory pressure to the point of
calling shrink_slab() 10s to 100s thousand times a second
(verified with ftrace; function tracer; shrink_slab filter).

$ sudo stress-ng --class memory --all -1 &

# echo shrink_slab > /sys/kernel/debug/tracing/set_ftrace_filter
# echo function > /sys/kernel/debug/tracing/current_tracer
# cat /sys/kernel/debug/tracing/trace_pipe | awk '{print $4}' | cut -d. -f1 | uniq -c
  24384 998
  38792 999
      1 EVENTS]
  22883 999
      1 EVENTS]
   1450 999
 113849 1000
 117025 1001
  67285 1002
  75768 1003
      1 EVENTS]
  14496 1003
 170817 1004
  98136 1005

[Mauricio Faria de Oliveira (mfo)]

[X/B][PATCH] mm: do not stall register_shrinker()
https://lists.ubuntu.com/archives/kernel-team/2019-February/098791.html

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Mauricio Faria de Oliveira (mfo)]

Marking X/B as verification done.

The user reports the issue occurs much less often now.
Apparently that environment hits some corner case or this may still be expected sometimes under memory pressure (i.e., one big shrinking operation acquired the lock and must finish).

Nonetheless, the fix reduced the frequency the problem occurs, so it does address most of that and is beneficial on its own.

As discussed with @klebers on IRC, in this scenario we agreed to ship the fix, and address the pending behavior (if an actual issue) with additional fixes later.

cheers,
Mauricio

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-145.171

---------------
linux (4.4.0-145.171) xenial; urgency=medium

  * linux: 4.4.0-145.171 -proposed tracker (LP: #1821724)

  * linux-generic should depend on linux-base >=4.1 (LP: #1820419)
    - [Packaging] Fix linux-base dependency

linux (4.4.0-144.170) xenial; urgency=medium

  * linux: 4.4.0-144.170 -proposed tracker (LP: #1819660)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync getabis
    - [Packaging] update helper scripts
    - [Packaging] resync retpoline extraction

  * C++ demangling support missing from perf (LP: #1396654)
    - [Packaging] fix a mistype

  * CVE-2019-9213
    - mm: enforce min addr even if capable() in expand_downwards()

  * CVE-2019-3460
    - Bluetooth: Check L2CAP option sizes returned from l2cap_get_conf_opt

  * Xenial update: 4.4.176 upstream stable release (LP: #1818815)
    - net: fix IPv6 prefix route residue
    - vsock: cope with memory allocation failure at socket creation time
    - hwmon: (lm80) Fix missing unlock on error in set_fan_div()
    - net: Fix for_each_netdev_feature on Big endian
    - net: Add header for usage of fls64()
    - tcp: tcp_v4_err() should be more careful
    - net: Do not allocate page fragments that are not skb aligned
    - tcp: clear icsk_backoff in tcp_write_queue_purge()
    - vxlan: test dev->flags & IFF_UP before calling netif_rx()
    - net: stmmac: Fix a race in EEE enable callback
    - net: ipv4: use a dedicated counter for icmp_v4 redirect packets
    - x86: livepatch: Treat R_X86_64_PLT32 as R_X86_64_PC32
    - mfd: as3722: Handle interrupts on suspend
    - mfd: as3722: Mark PM functions as __maybe_unused
    - net/x25: do not hold the cpu too long in x25_new_lci()
    - mISDN: fix a race in dev_expire_timer()
    - ax25: fix possible use-after-free
    - Linux 4.4.176

  * sky2 ethernet card don't work after returning from suspension
    (LP: #1798921) // Xenial update: 4.4.176 upstream stable release
    (LP: #1818815)
    - sky2: Increase D3 delay again

  * Xenial update: 4.4.175 upstream stable release (LP: #1818813)
    - drm/bufs: Fix Spectre v1 vulnerability
    - staging: iio: adc: ad7280a: handle error from __ad7280_read32()
    - ASoC: Intel: mrfld: fix uninitialized variable access
    - scsi: lpfc: Correct LCB RJT handling
    - ARM: 8808/1: kexec:offline panic_smp_self_stop CPU
    - dlm: Don't swamp the CPU with callbacks queued during recovery
    - x86/PCI: Fix Broadcom CNB20LE unintended sign extension (redux)
    - powerpc/pseries: add of_node_put() in dlpar_detach_node()
    - serial: fsl_lpuart: clear parity enable bit when disable parity
    - ptp: check gettime64 return code in PTP_SYS_OFFSET ioctl
    - staging:iio:ad2s90: Make probe handle spi_setup failure
    - staging: iio: ad7780: update voltage on read
    - ARM: OMAP2+: hwmod: Fix some section annotations
    - modpost: validate symbol names also in find_elf_symbol
    - perf tools: Add Hygon Dhyana support
    - soc/tegra: Don't leak device tree node reference
    - f2fs: move dir data flush to write checkpoint process
    - f2fs: fix wrong return value of f2fs_acl_create
    - sunvdc: Do not spin in an infin...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.15.0-47.50

---------------
linux (4.15.0-47.50) bionic; urgency=medium

  * linux: 4.15.0-47.50 -proposed tracker (LP: #1819716)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync getabis
    - [Packaging] update helper scripts
    - [Packaging] resync retpoline extraction

  * C++ demangling support missing from perf (LP: #1396654)
    - [Packaging] fix a mistype

  * arm-smmu-v3 arm-smmu-v3.3.auto: CMD_SYNC timeout (LP: #1818162)
    - iommu/arm-smmu-v3: Fix unexpected CMD_SYNC timeout

  * Crash in nvme_irq_check() when using threaded interrupts (LP: #1818747)
    - nvme-pci: fix out of bounds access in nvme_cqe_pending

  * CVE-2019-9213
    - mm: enforce min addr even if capable() in expand_downwards()

  * CVE-2019-3460
    - Bluetooth: Check L2CAP option sizes returned from l2cap_get_conf_opt

  * amdgpu with mst WARNING on blanking (LP: #1814308)
    - drm/amd/display: Don't use dc_link in link_encoder
    - drm/amd/display: Move wait for hpd ready out from edp power control.
    - drm/amd/display: eDP sequence BL off first then DP blank.
    - drm/amd/display: Fix unused variable compilation error
    - drm/amd/display: Fix warning about misaligned code
    - drm/amd/display: Fix MST dp_blank REG_WAIT timeout

  * tun/tap: unable to manage carrier state from userland (LP: #1806392)
    - tun: implement carrier change

  * CVE-2019-8980
    - exec: Fix mem leak in kernel_read_file

  * raw_skew in timer from the ubuntu_kernel_selftests failed on Bionic
    (LP: #1811194)
    - selftest: timers: Tweak raw_skew to SKIP when ADJ_OFFSET/other clock
      adjustments are in progress

  * [Packaging] Allow overlay of config annotations (LP: #1752072)
    - [Packaging] config-check: Add an include directive

  * CVE-2019-7308
    - bpf: move {prev_,}insn_idx into verifier env
    - bpf: move tmp variable into ax register in interpreter
    - bpf: enable access to ax register also from verifier rewrite
    - bpf: restrict map value pointer arithmetic for unprivileged
    - bpf: restrict stack pointer arithmetic for unprivileged
    - bpf: restrict unknown scalars of mixed signed bounds for unprivileged
    - bpf: fix check_map_access smin_value test when pointer contains offset
    - bpf: prevent out of bounds speculation on pointer arithmetic
    - bpf: fix sanitation of alu op with pointer / scalar type from different
      paths
    - bpf: add various test cases to selftests

  * CVE-2017-5753
    - bpf: properly enforce index mask to prevent out-of-bounds speculation
    - bpf: fix inner map masking to prevent oob under speculation

  * BPF: kernel pointer leak to unprivileged userspace (LP: #1815259)
    - bpf/verifier: disallow pointer subtraction

  * squashfs hardening (LP: #1816756)
    - squashfs: more metadata hardening
    - squashfs metadata 2: electric boogaloo
    - squashfs: more metadata hardening
    - Squashfs: Compute expected length from inode size rather than block length

  * efi/arm/arm64: Allow SetVirtualAddressMap() to be omitted (LP: #1814982)
    - efi/arm/arm64: Allow SetVirtualAddressMap() to be omitted

  * Update ENA driver to version 2.0.3K (LP: #1816806)...