Ubuntu xenial - 4.4.0-59-generic i3 I/O performance issue

Bug #1657281 reported by Dragan S. on 2017-01-17
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Undecided
Dragan S.
Xenial
Undecided
Thadeu Lima de Souza Cascardo

Bug Description

[Impact]

 * When running block device I/O on an Amazon i3 system there
   is a performance degradation. A patch with git change id
   87c279e613f848c691111b29 that is in Ubuntu-lts-4.8.0 kernel
   series increases I/O performance.

 * Fix should be backported to xenial 4.4 series to avoid more
   support issues being filed.

 * This change gets rid of uneccessary work being performed.

[Test Case]

 * Steps to reproduce below
   1) partition ephemeral disks
      /sbin/parted -s --align optimal /dev/nvme0n1 mklabel gpt mkpart primary 0% 100%
      /sbin/parted -s --align optimal /dev/nvme1n1 mklabel gpt mkpart primary 0% 100%
      /sbin/parted -s --align optimal /dev/nvme2n1 mklabel gpt mkpart primary 0% 100%
      /sbin/parted -s --align optimal /dev/nvme3n1 mklabel gpt mkpart primary 0% 100%

   2) create raid array
      /sbin/mdadm --create /dev/md0 --assume-clean --chunk=2048 --level=10 --raid-devices=4 /dev/nvme0n1p1 /dev/nvme1n1p1 /dev/nvme2n1p1 /dev/nvme3n1p1

   3) create pv
      /sbin/pvcreate --force --metadatasize 4092k /dev/md0

   4) create volume group
      /sbin/vgcreate RDSVG /dev/md0

   5) create lv
      /sbin/lvcreate -L 2.5T -n RDSRAIDLV RDSVG

   5) test I/O
      sudo dd if=/dev/zero of=/dev/RDSVG/RDSRAIDLV bs=8k count=1000000 && sync

[Regression Potential]

 * No known regression potential.

[Original Description]

When we were doing testing on i3, we noticed that it is taking significantly longer to perform operations when using software RAID than without, we believe this is resolved in an upstream commit: http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=87c279e613f848c691111b29d49de8df3f4f56da

So stock 4.4.0-59-generic performs ok:
$ sudo dd if=/dev/zero of=/dev/RDSVG/RDSRAIDLV bs=8k count=1000000 && sync
sudo: unable to resolve host ip-10-0-85-167
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB, 7.6 GiB) copied, 54.3711 s, 151 MB/s

and the patch you originally providing works a little bit better (as expected):
$ uname -a
Linux ip-10-0-85-167 4.4.0-57-generic #78hf000000v20170110b0h3199a6e718db-Ubuntu SMP Tue Jan 10 02:53: x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ip-10-0-85-167:~$ sudo dd if=/dev/zero of=/dev/RDSVG/RDSRAIDLV bs=8k count=1000000 && sync
sudo: unable to resolve host ip-10-0-85-167
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB, 7.6 GiB) copied, 31.4108 s, 261 MB/s

Dragan S. (dragan-s) wrote :
tags: added: sts
description: updated
Changed in linux (Ubuntu):
assignee: nobody → Dragan S. (dragan-s)

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1657281

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: patch
Brad Figg (brad-figg) on 2017-01-18
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Tim Gardner (timg-tpi) wrote :
Changed in linux (Ubuntu Xenial):
assignee: nobody → Thadeu Lima de Souza Cascardo (cascardo)
status: New → In Progress
Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
John Donnelly (jpdonnelly) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-xenial
Dragan S. (dragan-s) wrote :

Verified:

4.4.0-59-generic
$ sudo dd if=/dev/zero of=/dev/RDSVG/RDSRAIDLV bs=8k count=1000000 && sync
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB, 7.6 GiB) copied, 56.7741 s, 144 MB/s

4.4.0-62-generic
$ sudo dd if=/dev/zero of=/dev/RDSVG/RDSRAIDLV bs=8k count=1000000 && sync
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB, 7.6 GiB) copied, 31.3783 s, 261 MB/s

tags: added: verification-done-xenial
removed: verification-needed-xenial
Launchpad Janitor (janitor) wrote :
Download full text (10.8 KiB)

This bug was fixed in the package linux - 4.4.0-62.83

---------------
linux (4.4.0-62.83) xenial; urgency=low

  [ Thadeu Lima de Souza Cascardo ]

  * Release Tracking Bug
    - LP: #1657430

  * Backport DP MST fixes to i915 (LP: #1657353)
    - SAUCE: i915_bpo: Fix DP link rate math
    - SAUCE: i915_bpo: Validate mode against max. link data rate for DP MST

  * Ubuntu xenial - 4.4.0-59-generic i3 I/O performance issue (LP: #1657281)
    - blk-mq: really fix plug list flushing for nomerge queues

linux (4.4.0-61.82) xenial; urgency=low

  [ Thadeu Lima de Souza Cascardo ]

  * Release Tracking Bug
    - LP: #1656810

  * Xen MSI setup code incorrectly re-uses cached pirq (LP: #1656381)
    - SAUCE: xen: do not re-use pirq number cached in pci device msi msg data

  * nvme drive probe failure (LP: #1626894)
    - nvme: revert NVMe: only setup MSIX once

linux (4.4.0-60.81) xenial; urgency=low

  [ John Donnelly ]

  * Release Tracking Bug
    - LP: #1656084

  * Couldn't emulate instruction 0x7813427c (LP: #1634129)
    - KVM: PPC: Book3S PR: Fix illegal opcode emulation

  * perf: 24x7: Eliminate domain name suffix in event names (LP: #1560482)
    - powerpc/perf/hv-24x7: Fix usage with chip events.
    - powerpc/perf/hv-24x7: Display change in counter values
    - powerpc/perf/hv-24x7: Display domain indices in sysfs
    - powerpc/perf/24x7: Eliminate domain suffix in event names

  * i386 ftrace tests hang on ADT testing (LP: #1655040)
    - ftrace/x86_32: Set ftrace_stub to weak to prevent gcc from using short jumps
      to it

  * VMX module autoloading if available (LP: #1651322)
    - powerpc: Add module autoloading based on CPU features
    - crypto: vmx - Convert to CPU feature based module autoloading

  * ACPI probe support for AD5592/3 configurable multi-channel converter
    (LP: #1654497)
    - SAUCE: iio: dac: ad5592r: Add ACPI support
    - SAUCE: iio: dac: ad5593r: Add ACPI support

  * Xenial update to v4.4.40 stable release (LP: #1654602)
    - btrfs: limit async_work allocation and worker func duration
    - Btrfs: fix tree search logic when replaying directory entry deletes
    - btrfs: store and load values of stripes_min/stripes_max in balance status
      item
    - Btrfs: fix qgroup rescan worker initialization
    - USB: serial: option: add support for Telit LE922A PIDs 0x1040, 0x1041
    - USB: serial: option: add dlink dwm-158
    - USB: serial: kl5kusb105: fix open error path
    - USB: cdc-acm: add device id for GW Instek AFG-125
    - usb: hub: Fix auto-remount of safely removed or ejected USB-3 devices
    - usb: gadget: f_uac2: fix error handling at afunc_bind
    - usb: gadget: composite: correctly initialize ep->maxpacket
    - USB: UHCI: report non-PME wakeup signalling for Intel hardware
    - ALSA: usb-audio: Add QuickCam Communicate Deluxe/S7500 to
      volume_control_quirks
    - ALSA: hiface: Fix M2Tech hiFace driver sampling rate change
    - ALSA: hda/ca0132 - Add quirk for Alienware 15 R2 2016
    - ALSA: hda - ignore the assoc and seq when comparing pin configurations
    - ALSA: hda - fix headset-mic problem on a Dell laptop
    - ALSA: hda - Gate the mic jack on HP Z1 Gen3 AiO
    - ALSA: hd...

Changed in linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers