user space process hung in 'D' state waiting for disk io to complete

[Dragan S.]

== SRU Justification ==

[Impact]
Occasionally an application gets stuck in "D" state on NFS reads/sync and close system calls. All the subsequent operations on the NFS mounts are stuck and reboot is required to rectify the situation.

[Fix]
Use GPF_NOIO in some allocations in writeback to avoid a deadlock. This is upstream in:
ae97aa524ef4 ("NFS: Use GFP_NOIO for two allocations in writeback")

[Testcase]
See Test scenario in previous description.

A test kernel with this patch was tested heavily (>100hrs of test suite) without issue.

[Regression Potential]
This changes memory allocation in NFS to use a different policy. This could potentially affect NFS.

However, the patch is already in Artful and Bionic without issue.

The patch does not apply to Trusty.

== Previous Description ==

Using Ubuntu Xenial user reports processes hang in D state waiting for disk io.

Ocassionally one of the applications gets into "D" state on NFS reads/sync and close system calls. based on the kernel backtraces seems to be stuck in kmalloc allocation during cleanup of dirty NFS pages.

All the subsequent operations on the NFS mounts are stuck and reboot is required to rectify the situation.

[Test scenario]

1) Applications running in Docker environment
2) Application have cgroup limits --cpu-shares --memory -shm-limit
3) python and C++ based applications (torch and caffe)
4) Applications read big lmdb files and write results to NFS shares
5) use NFS v3 , hard and fscache is enabled
6) now swap space is configured

This prevents all other I/O activity on that mount to hang.

we are running into this issue more frequently and identified few applications causing this problem.

As updated in the description, the problem seems to be happening when exercising the stack

try_to_free_mem_cgroup_pages+0xba/0x1a0

we see this with docker containers with cgroup option --memory <USER_SPECIFIED_MEM>.

whenever there is a deadlock, we see that the process that is hung has reached the maximum cgroup limit, multiple times and typically cleans up dirty data and caches to bring the usage under the limit.

This reclaim path happens many times and finally we hit probably a race get into deadlock

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1750038

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.16 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc2

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[David Coronel (davecore)]

I have confirmation from the user who reported this that they have tested the new kernel 4.4.0-128.154 in their test environment for a 24 hours stress test and could not reproduce the issue. I changed the tag to verification done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-128.154

---------------
linux (4.4.0-128.154) xenial; urgency=medium

  * linux: 4.4.0-128.154 -proposed tracker (LP: #1772960)

  * CVE-2018-3639 (x86)
    - x86/cpu: Make alternative_msr_write work for 32-bit code
    - x86/bugs: Fix the parameters alignment and missing void
    - KVM: SVM: Move spec control call after restore of GS
    - x86/speculation: Use synthetic bits for IBRS/IBPB/STIBP
    - x86/cpufeatures: Disentangle MSR_SPEC_CTRL enumeration from IBRS
    - x86/cpufeatures: Disentangle SSBD enumeration
    - x86/cpu/AMD: Fix erratum 1076 (CPB bit)
    - x86/cpufeatures: Add FEATURE_ZEN
    - x86/speculation: Handle HT correctly on AMD
    - x86/bugs, KVM: Extend speculation control for VIRT_SPEC_CTRL
    - x86/speculation: Add virtualized speculative store bypass disable support
    - x86/speculation: Rework speculative_store_bypass_update()
    - x86/bugs: Unify x86_spec_ctrl_{set_guest,restore_host}
    - x86/bugs: Expose x86_spec_ctrl_base directly
    - x86/bugs: Remove x86_spec_ctrl_set()
    - x86/bugs: Rework spec_ctrl base and mask logic
    - x86/speculation, KVM: Implement support for VIRT_SPEC_CTRL/LS_CFG
    - KVM: SVM: Implement VIRT_SPEC_CTRL support for SSBD
    - x86/bugs: Rename SSBD_NO to SSB_NO
    - KVM: VMX: Expose SSBD properly to guests.

  * [i915_bpo] Fix flickering issue after panel change (LP: #1770565)
    - drm/i915: Fix iboost setting for DDI with 4 lanes on SKL
    - drm/i915: Name the "iboost bit"
    - drm/i915: Program iboost settings for HDMI/DVI on SKL
    - drm/i915: Move bxt_ddi_vswing_sequence() call into intel_ddi_pre_enable()
      for HDMI
    - drm/i915: Explicitly use ddi buf trans entry 9 for hdmi
    - drm/i915: Split DP/eDP/FDI and HDMI/DVI DDI buffer programming apart
    - drm/i915: Get the iboost setting based on the port type
    - drm/i915: Simplify intel_ddi_get_encoder_port()
    - drm/i915: Fix iboost setting for SKL Y/U DP DDI buffer translation entry 2
    - drm/i915: KBL - Recommended buffer translation programming for DisplayPort
    - drm/i915: Ignore OpRegion panel type except on select machines

  * [SRU][Bionic/Artful] fix false positives in W+X checking (LP: #1769696)
    - init: fix false positives in W+X checking

  * [Ubuntu 16.04] kernel: fix rwlock implementation (LP: #1761674)
    - SAUCE: (no-up) s390: fix rwlock implementation

  * linux < 4.11: unable to use netfilter logging from non-init namespaces
    (LP: #1766573)
    - netfilter: allow logging from non-init namespaces

  * [LTC Test] Ubuntu 18.04: tm_sigreturn failed on P8 compat mode 16.04.04
    guest (LP: #1771439)
    - powerpc: signals: Discard transaction state from signal frames

  * QCA9377 requires more IRAM banks for its new firmware (LP: #1748345)
    - ath10k: update the IRAM bank number for QCA9377

  * i915/kbl_dmc_ver1.bin failed with error -2 package 1.157.17 kernel
    4.4.0-116-generic (LP: #1752536)
    - ubuntu: i915_bpo - Add MODULE_FIRMWARE for Geminilake's DMC

  * Xenial update to 4.4.131 stable release (LP: #1768825)
    - ext4: prevent right-shifting extents beyond EXT_MAX_BLOCKS
    - ext4: set h_journal if there is a failure...