unnecessary request_queue freeze

[Aleksei]

Hi all,

In linux-4.15, which is used as hwe-edge in xenial, there's an unexpected behavior. It results in latency spikes of generic_make_request() function call.
How to reproduce:
  1. Run fio:
  fio --name=test --rw=randwrite --direct=1 --filename=/dev/sdt --iodepth=1 --fsync=1 --ioengine=libaio --write_lat_log loglat
  2. While fio is writing to the device, change nr_requests setting to the same value as current:
  echo 256 > /sys/block/sdt/queue/nr_requests
  or: if scheduler is already none:
  echo none > /sys/block/sdt/queue/scheduler
  3. Stop fio and check slat log:
  ...
  3980, 16, 1, 4096
  3999, 17, 1, 4096
  4010, 19, 1, 4096
  4020, 18, 1, 4096
  4045, 10821, 1, 4096
  4054, 25, 1, 4096
  4070, 12, 1, 4096
  4083, 17, 1, 4096
  4094, 16, 1, 4096
  4111, 17, 1, 4096
  ...

There're a few patches to fix this behavior:
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?h=for-5.1/block&id=e5fa81408fb43ebabde65938ef8b20ae879017e7
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?h=for-5.1/block&id=fbd72127c975dc8e532ecc73d52f3b1b00935bec

This behavior affects production systems, because every change uevent triggers udev to change settings prior to udev rules. Some software, for example arcconf (tool for managing adaptec HBA adapters) triggers uevents for every block device. So, every call to arcconf utility results in latency spike.

Could this be backported to 4.15 kernel for xenial/bionic?

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1815733/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1815733

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Kai-Heng Feng (kaihengfeng)]

Seems to be not that trivial without commit a79050434b45959f397042080fd1d70ffa9bd9df.

Since you are the patch author, and I am not familiar with block subsystem, so please check my patches are correct.

[Kai-Heng Feng (kaihengfeng)]

A test kernel can be found at https://people.canonical.com/~khfeng/lp1815733/

[Aleksei (zakharov-a-g)]

Hi, thank you!
0002-block-avoid-setting-none-scheduler-if-it-s-already-n.patch - looks good.

0001-block-avoid-setting-wbt_lat_usec-to-current-value.patch - I didn't mention this one:) it is not needed for 4.15, because blk_mq_freeze_queue() in queue_wb_lat_store() was added only at 4.19 due to c125311d96b1bfcce0f5930a4f0fdfe39ea14f7c
I think, we can skip it for 4.15.

So, i'm asking to backport this one - e5fa81408fb43ebabde65938ef8b20ae879017e7

[Kai-Heng Feng (kaihengfeng)]

Thanks for pointing that out. I thought they belong to the same series.

Please test this kernel:
https://people.canonical.com/~khfeng/lp1815733/

[Aleksei (zakharov-a-g)]

I've tested this kernel - there's no latency spike when setting none scheduler while it's already none - nice.

Waiting for e5fa81408fb43ebabde65938ef8b20ae879017e7 backport:)

[Kai-Heng Feng (kaihengfeng)]

I'd like to wait till they at least land to v5.1-rc and get more thoroughly test.
I'll request an SRU once they are there.

[Kai-Heng Feng (kaihengfeng)]

We can still put it into Disco (19.04)'s kernel (v5.0) and see how it goes.
If no regression happens for a while, we can back port it to v4.15.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.0.0-7.8

---------------
linux (5.0.0-7.8) disco; urgency=medium

  * linux: 5.0.0-7.8 -proposed tracker (LP: #1818519)

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts

  * unnecessary request_queue freeze (LP: #1815733)
    - block: avoid setting nr_requests to current value
    - block: avoid setting none scheduler if it's already none

  * Miscellaneous Ubuntu changes
    - SAUCE: selftests: net: Don't fail test_vxlan_under_vrf on xfail
    - update dkms package versions

  [ Upstream Kernel Changes ]

  * Rebase to v5.0

 -- Seth Forshee <email address hidden> Mon, 04 Mar 2019 08:46:10 -0600

[Aleksei (zakharov-a-g)]

Hello, colleagues,
These patches are merged into linux-5.1-rc1 now.
Is it ready to be backported to 4.15 ubuntu kernel?

[Kleber Sacilotto de Souza (kleber-souza)]

Hi,

The SRU request has been sent for Bionic (4.15) and Cosmic (4.18):
https://lists.ubuntu.com/archives/kernel-team/2019-April/100380.html

[AceLan Kao (acelankao)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Aleksei (zakharov-a-g)]

Thanks, i see it is fixed in 5.0.0-14 bionic hwe-edge kernel.

[Aleksei (zakharov-a-g)]

But i don't see changes for linux-4.15.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-oem - 4.15.0-1038.43

---------------
linux-oem (4.15.0-1038.43) bionic; urgency=medium

  [ Ubuntu: 4.15.0-50.54 ]

  * CVE-2018-12126 // CVE-2018-12127 // CVE-2018-12130
    - Documentation/l1tf: Fix small spelling typo
    - x86/cpu: Sanitize FAM6_ATOM naming
    - kvm: x86: Report STIBP on GET_SUPPORTED_CPUID
    - locking/atomics, asm-generic: Move some macros from <linux/bitops.h> to a
      new <linux/bits.h> file
    - tools include: Adopt linux/bits.h
    - x86/msr-index: Cleanup bit defines
    - x86/speculation: Consolidate CPU whitelists
    - x86/speculation/mds: Add basic bug infrastructure for MDS
    - x86/speculation/mds: Add BUG_MSBDS_ONLY
    - x86/kvm: Expose X86_FEATURE_MD_CLEAR to guests
    - x86/speculation/mds: Add mds_clear_cpu_buffers()
    - x86/speculation/mds: Clear CPU buffers on exit to user
    - x86/kvm/vmx: Add MDS protection when L1D Flush is not active
    - x86/speculation/mds: Conditionally clear CPU buffers on idle entry
    - x86/speculation/mds: Add mitigation control for MDS
    - x86/speculation/mds: Add sysfs reporting for MDS
    - x86/speculation/mds: Add mitigation mode VMWERV
    - Documentation: Move L1TF to separate directory
    - Documentation: Add MDS vulnerability documentation
    - x86/speculation/mds: Add mds=full,nosmt cmdline option
    - x86/speculation: Move arch_smt_update() call to after mitigation decisions
    - x86/speculation/mds: Add SMT warning message
    - x86/speculation/mds: Fix comment
    - x86/speculation/mds: Print SMT vulnerable on MSBDS with mitigations off
    - x86/speculation/mds: Add 'mitigations=' support for MDS
  * CVE-2017-5715 // CVE-2017-5753
    - s390/speculation: Support 'mitigations=' cmdline option
  * CVE-2017-5715 // CVE-2017-5753 // CVE-2017-5754 // CVE-2018-3639
    - powerpc/speculation: Support 'mitigations=' cmdline option
  * CVE-2017-5715 // CVE-2017-5754 // CVE-2018-3620 // CVE-2018-3639 //
    CVE-2018-3646
    - cpu/speculation: Add 'mitigations=' cmdline option
    - x86/speculation: Support 'mitigations=' cmdline option
  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log

linux-oem (4.15.0-1037.42) bionic; urgency=medium

  * linux-oem: 4.15.0-1037.42 -proposed tracker (LP: #1826336)

  * unnecessary request_queue freeze (LP: #1815733)
    - block: avoid setting nr_requests to current value
    - block: avoid setting none scheduler if it's already none

  * Screen freeze after resume from S3 when HDMI monitor plugged on Dell
    Precision 7740 (LP: #1825958)
    - PCI: Restore resized BAR state on resume

  [ Ubuntu: 4.15.0-49.53 ]

  * linux: 4.15.0-49.53 -proposed tracker (LP: #1826358)
  * Backport support for software count cache flush Spectre v2 mitigation. (CVE)
    (required for POWER9 DD2.3) (LP: #1822870)
    - powerpc/64s: Add support for ori barrier_nospec patching
    - powerpc/64s: Patch barrier_nospec in modules
    - powerpc/64s: Enable barrier_nospec based on firmware settings
    - powerpc: Use barrier_nospec in copy_from_user()
    - powerpc/64: Use barrier_nospec in syscall entry
    - powerpc/64s: Enhance the information in cpu_show_spectre_v1()
    - p...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-cosmic' to 'verification-done-cosmic'. If the problem still exists, change the tag 'verification-needed-cosmic' to 'verification-failed-cosmic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.15.0-51.55

---------------
linux (4.15.0-51.55) bionic; urgency=medium

  * linux: 4.15.0-51.55 -proposed tracker (LP: #1829219)

  * disable a.out support (LP: #1818552)
    - [Config] Disable a.out support

  * [UBUNTU] qdio: clear intparm during shutdown (LP: #1828394)
    - s390/qdio: clear intparm during shutdown

  * ftrace in ubuntu_kernel_selftests hang with Cosmic kernel (LP: #1826385)
    - kprobes/x86: Fix instruction patching corruption when copying more than one
      RIP-relative instruction

  * touchpad not working on lenovo yoga 530 (LP: #1787775)
    - Revert "UBUNTU: SAUCE: i2c:amd Depends on ACPI"
    - Revert "UBUNTU: SAUCE: i2c:amd move out pointer in union i2c_event_base"
    - Revert "UBUNTU: SAUCE: i2c:amd I2C Driver based on PCI Interface for
      upcoming platform"
    - i2c: add helpers to ease DMA handling
    - i2c: add a message flag for DMA safe buffers
    - i2c: add extra check to safe DMA buffer helper
    - i2c: Add drivers for the AMD PCIe MP2 I2C controller
    - [Config] Update config for AMD MP2 I2C driver
    - [Config] Update I2C_AMD_MP2 annotations

  * tm-unavailable in powerpc/tm failed on Bionic Power9 (LP: #1813129)
    - selftests/powerpc: Check for pthread errors in tm-unavailable
    - selftests/powerpc: Skip tm-unavailable if TM is not enabled

  * cp_abort in powerpc/context_switch from ubunut_kernel_selftests failed on
    Bionic P9 (LP: #1813134)
    - selftests/powerpc: Remove redundant cp_abort test

  * bionic/linux: completely remove snapdragon files from sources (LP: #1827880)
    - [Packaging] remove snapdragon dead files
    - [Config] update configs after snapdragon removal

  * The noise keeps occurring when Headset is plugged in on a Dell machine
    (LP: #1827972)
    - ALSA: hda/realtek - Fixed Dell AIO speaker noise

  * Geneve tunnels don't work when ipv6 is disabled (LP: #1794232)
    - geneve: correctly handle ipv6.disable module parameter

  * There are 4 HDMI/Displayport audio output listed in sound setting without
    attach any HDMI/DP monitor (LP: #1827967)
    - ALSA: hda/hdmi - Read the pin sense from register when repolling
    - ALSA: hda/hdmi - Consider eld_valid when reporting jack event

  * Headphone jack switch sense is inverted: plugging in headphones disables
    headphone output (LP: #1824259)
    - ASoC: rt5645: Headphone Jack sense inverts on the LattePanda board

  * CTAUTO:DevOps:860.50:devops4fp1:Error occurred during LINUX Dmesg error
    Checking for all LINUX clients for devops4p10 (LP: #1766201)
    - SAUCE: integrity: downgrade error to warning

  * Screen freeze after resume from S3 when HDMI monitor plugged on Dell
    Precision 7740 (LP: #1825958)
    - PCI: Restore resized BAR state on resume

  * potential memory corruption on arm64 on dev release (LP: #1827437)
    - driver core: Postpone DMA tear-down until after devres release

  * powerpc/pmu/ebb test in ubuntu_kernel_selftest failed with "error while
    loading shared libraries" on Bionic/Cosmic PowerPC (LP: #1812805)
    - selftests/powerpc/pmu: Link ebb tests with -no-pie

  * unnecessary request_queue freeze (LP: #1815733)
    - block: av...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.18.0-21.22

---------------
linux (4.18.0-21.22) cosmic; urgency=medium

  * linux: 4.18.0-21.22 -proposed tracker (LP: #1829186)

  * disable a.out support (LP: #1818552)
    - [Config] Turn off a.out support

  * ftrace in ubuntu_kernel_selftests hang with Cosmic kernel (LP: #1826385)
    - kprobes/x86: Fix instruction patching corruption when copying more than one
      RIP-relative instruction

  * touchpad not working on lenovo yoga 530 (LP: #1787775)
    - Revert "UBUNTU: SAUCE: i2c:amd Depends on ACPI"
    - Revert "UBUNTU: SAUCE: i2c:amd move out pointer in union i2c_event_base"
    - Revert "UBUNTU: SAUCE: i2c:amd I2C Driver based on PCI Interface for
      upcoming platform"
    - i2c: add extra check to safe DMA buffer helper
    - i2c: Add drivers for the AMD PCIe MP2 I2C controller
    - [Config] Update config for AMD MP2 I2C driver
    - [Config] Update I2C_AMD_MP2 annotations

  * Geneve tunnels don't work when ipv6 is disabled (LP: #1794232)
    - geneve: correctly handle ipv6.disable module parameter

  * There are 4 HDMI/Displayport audio output listed in sound setting without
    attach any HDMI/DP monitor (LP: #1827967)
    - ALSA: hda/hdmi - Read the pin sense from register when repolling
    - ALSA: hda/hdmi - Consider eld_valid when reporting jack event

  * Headphone jack switch sense is inverted: plugging in headphones disables
    headphone output (LP: #1824259)
    - ASoC: rt5645: Headphone Jack sense inverts on the LattePanda board

  * CTAUTO:DevOps:860.50:devops4fp1:Error occurred during LINUX Dmesg error
    Checking for all LINUX clients for devops4p10 (LP: #1766201)
    - SAUCE: integrity: downgrade error to warning

  * potential memory corruption on arm64 on dev release (LP: #1827437)
    - driver core: Postpone DMA tear-down until after devres release

  * powerpc/pmu/ebb test in ubuntu_kernel_selftest failed with "error while
    loading shared libraries" on Bionic/Cosmic PowerPC (LP: #1812805)
    - selftests/powerpc/pmu: Link ebb tests with -no-pie

  * unnecessary request_queue freeze (LP: #1815733)
    - block: avoid setting nr_requests to current value
    - block: avoid setting none scheduler if it's already none

  * Kprobe event string type argument failed in ftrace from
    ubuntu_kernel_selftests on B/C i386 (LP: #1825780)
    - selftests/ftrace: Fix kprobe string testcase to not probe notrace function

  * False positive test result in run_netsocktests from net in
    ubuntu_kernel_selftest (LP: #1825777)
    - selftests/net: correct the return value for run_netsocktests

 -- Stefan Bader <email address hidden> Wed, 15 May 2019 13:18:36 +0200