[Hyper-V] mkfs regression in kernel 4.4+

[Stephen A. Zarkos]

Related bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1657539

This issue was fixed in Zesty, but also impacts the Xenial and Yakkety kernels. Can you please pull this patch back to Xenial+?

https://git.kernel.org/cgit/linux/kernel/git/axboe/linux-block.git/commit/?h=for-4.11/block&id=729204ef49ec00b788ce23deb9eb922a5769f55d

author
Ming Lei <email address hidden>
2016-12-17 10:49:09 (GMT)
committer
Jens Axboe <email address hidden>
2017-01-12 03:47:08 (GMT)
commit
729204ef49ec00b788ce23deb9eb922a5769f55d (patch)
tree
ab5cc3c0aa17fa8b988ed7dfefe888000fac0be3
parent
1661f2e21c8bbf922dcb76faf2126a33ffe4cddb (diff)
block: relax check on sg gap
If the last bvec of the 1st bio and the 1st bvec of the next
bio are physically contigious, and the latter can be merged
to last segment of the 1st bio, we should think they don't
violate sg gap(or virt boundary) limit.

Both Vitaly and Dexuan reported lots of unmergeable small bios
are observed when running mkfs on Hyper-V virtual storage, and
performance becomes quite low. This patch fixes that performance
issue.

The same issue should exist on NVMe, since it sets virt boundary too.

Reported-by: Vitaly Kuznetsov <email address hidden>
Reported-by: Dexuan Cui <email address hidden>
Tested-by: Dexuan Cui <email address hidden>
Cc: Keith Busch <email address hidden>
Signed-off-by: Ming Lei <email address hidden>
Signed-off-by: Jens Axboe <email address hidden>

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1682215

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

I built a test kernel with commit 729204ef49. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1682215/

Can you test this kernel and see if it resolves this bug?

Thanks in advance!

[Stephen A. Zarkos (stevez)]

Hi,

The test looks good, here are my repro steps:

 1) Create an Ubuntu 16.04 VM in Azure
 2) Attach 128GB data disk
 3) Partition and format the disk
    # wipefs --all /dev/sdc
    # sgdisk -Z /dev/sdc
    # sgdisk -n '1:2048::264191' /dev/sdc
    # mkfs.ext4 /dev/sdc1

Expected result: mkfs should return very quickly because lazy_itable_init is enabled.

Actual test results:

  - Kernel 4.4.0-71.92: mkfs blocks and formatting takes about 8-minutes.

  - Kernel 4.4.0-73.94~lp1682215: mkfs returns immediately.

Thanks!
Steve

[Joseph Salisbury (jsalisbury)]

SRU request submitted for Xenial and Yakkety:
https://lists.ubuntu.com/archives/kernel-team/2017-April/083605.html

[Kleber Sacilotto de Souza (kleber-souza)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Kleber Sacilotto de Souza (kleber-souza)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-yakkety' to 'verification-done-yakkety'. If the problem still exists, change the tag 'verification-needed-yakkety' to 'verification-failed-yakkety'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Stephen A. Zarkos (stevez)]

Test case from comment #3 passed using the -proposed kernel for Xenial and Yakkey.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.8.0-52.55

---------------
linux (4.8.0-52.55) yakkety; urgency=low

  * linux: 4.8.0-52.55 -proposed tracker (LP: #1686976)

  * CVE-2017-7477: macsec: avoid heap overflow in skb_to_sgvec (LP: #1685892)
    - macsec: avoid heap overflow in skb_to_sgvec
    - macsec: dynamically allocate space for sglist

  * net/ipv4: original ingress device index set as the loopback interface.
    (LP: #1683982)
    - net: fix incorrect original ingress device index in PKTINFO

  * Touchpad not working correctly after kernel upgrade (LP: #1662589)
    - Input: ALPS - fix V8+ protocol handling (73 03 28)

  * ifup service of network device stay active after driver stop (LP: #1672144)
    - net: use net->count to check whether a netns is alive or not

  * [Hyper-V] mkfs regression in kernel 4.4+ (LP: #1682215)
    - block: relax check on sg gap

  * Potential memory corruption with capi adapters (LP: #1681469)
    - powerpc/mm: Add missing global TLB invalidate if cxl is active

  * [Hyper-V/Azure] Please include Mellanox OFED drivers in Azure kernel and
    image (LP: #1650058)
    - net/mlx4_en: Fix bad WQE issue
    - net/mlx4_core: Fix racy CQ (Completion Queue) free
    - net/mlx4_core: Fix when to save some qp context flags for dynamic VST to VGT
      transitions
    - net/mlx4_core: Avoid command timeouts during VF driver device shutdown

 -- Stefan Bader <email address hidden> Fri, 28 Apr 2017 12:17:12 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-78.99

---------------
linux (4.4.0-78.99) xenial; urgency=low

  * linux: 4.4.0-78.99 -proposed tracker (LP: #1686645)

  * Please backport fix to reference leak in cgroup blkio throttle
    (LP: #1683976)
    - block: fix module reference leak on put_disk() call for cgroups throttle

  * UbuntuKVM guest crashed while running I/O stress test with Ubuntu kernel
    4.4.0-47-generic (LP: #1659111)
    - block: Unhash block device inodes on gendisk destruction
    - block: Use pointer to backing_dev_info from request_queue
    - block: Dynamically allocate and refcount backing_dev_info
    - block: Make blk_get_backing_dev_info() safe without open bdev
    - block: Get rid of blk_get_backing_dev_info()
    - block: Move bdev_unhash_inode() after invalidate_partition()
    - block: Unhash also block device inode for the whole device
    - block: Revalidate i_bdev reference in bd_aquire()
    - block: Initialize bd_bdi on inode initialization
    - block: Move bdi_unregister() to del_gendisk()
    - block: Allow bdi re-registration
    - bdi: Fix use-after-free in wb_congested_put()
    - block: Make del_gendisk() safer for disks without queues
    - block: Fix bdi assignment to bdev inode when racing with disk delete
    - bdi: Mark congested->bdi as internal
    - bdi: Make wb->bdi a proper reference
    - bdi: Unify bdi->wb_list handling for root wb_writeback
    - bdi: Shutdown writeback on all cgwbs in cgwb_bdi_destroy()
    - bdi: Do not wait for cgwbs release in bdi_unregister()
    - bdi: Rename cgwb_bdi_destroy() to cgwb_bdi_unregister()
    - block: Fix oops in locked_inode_to_wb_and_lock_list()
    - kobject: Export kobject_get_unless_zero()
    - block: Fix oops scsi_disk_get()

  * Touchpad not working correctly after kernel upgrade (LP: #1662589)
    - Input: ALPS - fix V8+ protocol handling (73 03 28)

  * Xenial update to v4.4.62 stable release (LP: #1683728)
    - drm/i915: Avoid tweaking evaluation thresholds on Baytrail v3
    - drm/i915: Stop using RP_DOWN_EI on Baytrail
    - usb: dwc3: gadget: delay unmap of bounced requests
    - mtd: bcm47xxpart: fix parsing first block after aligned TRX
    - MIPS: Introduce irq_stack
    - MIPS: Stack unwinding while on IRQ stack
    - MIPS: Only change $28 to thread_info if coming from user mode
    - MIPS: Switch to the irq_stack in interrupts
    - MIPS: Select HAVE_IRQ_EXIT_ON_IRQ_STACK
    - MIPS: IRQ Stack: Fix erroneous jal to plat_irq_dispatch
    - crypto: caam - fix RNG deinstantiation error checking
    - Linux 4.4.62

  * ifup service of network device stay active after driver stop (LP: #1672144)
    - net: use net->count to check whether a netns is alive or not

  * [Hyper-V] mkfs regression in kernel 4.4+ (LP: #1682215)
    - block: relax check on sg gap

  * [Feature] KBL: intel_powerclamp driver support (LP: #1591641)
    - thermal/powerclamp: remove cpu whitelist
    - thermal/powerclamp: correct cpu support check
    - thermal/powerclamp: add back module device table

  * sysfs channel reads of lps22hb pressure sensor are stale (LP: #1682103)
    - iio: st_pressure: initialize lps22hb bootime

  * Backlight control does no...