[linux] Patch to prevent possible data corruption

Bug #1848739 reported by Joseph Salisbury
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Invalid
Undecided
Unassigned
Bionic
Fix Released
Medium
Marcelo Cerri
linux-azure (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Undecided
Marcelo Cerri
Bionic
Invalid
Undecided
Unassigned

Bug Description

There are three patches that prevent possible data corruption. The three commits are:

aef1897cd36d ("blk-mq: insert rq with DONTPREP to hctx dispatch list when requeue")
c616cbee97ae ("blk-mq: punt failed direct issue to dispatch list")
923218f6166a ("blk-mq: don't allocate driver tag upfront for flush rq")

18.04 has all three of these patches. 16.04 has two out of the three, but it is missing commit c616cbee97ae.

We would like to request commit c616cbee97ae be included in the 16.04 kernel:
c616cbee97ae ("blk-mq: punt failed direct issue to dispatch list")

Revision history for this message
Marcelo Cerri (mhcerri) wrote :

Commit c616cbee97ae fixes ffe81d45322c ("blk-mq: fix corruption with direct issue") that is not present. Are you sure c616cbee97ae is really need in xenial:linux-azure?

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Yes we need that patch. The commit description is misleading, as this commit actually fixes a bug on data corruption for SCSI devices.

In the latest Ubuntu 4.15 kernel, __blk_mq_try_issue_directly() (in "block/blk-mq.c") calls blk_mq_sched_insert_request() if q->mq_ops->queue_rq() returns BLK_STS_RESOURCE, this is not correct and is prone to data corruption.

The c616cbee97ae has the following change:

@@ -1785,7 +1764,7 @@ static blk_status_t __blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx,
        if (bypass_insert)
                return BLK_STS_RESOURCE;

- blk_mq_sched_insert_request(rq, false, run_queue, false);
+ blk_mq_request_bypass_insert(rq, run_queue);
        return BLK_STS_OK;
 }

This change is subtle, now the I/O will not get merged with other pending I/O if a direct issue has failed. This fixed the data corruption for SCSI devices. The bug was not introduced by commit ffe81d45322c.

I recommend them pick up all the relevant patches leading to this commit. If this is difficult we can do a special back ported patch to change the code in __blk_mq_try_issue_directly() to handle requeue.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-azure (Ubuntu):
status: New → Confirmed
Revision history for this message
Edward Vielmetti (ed-packet) wrote :

This data corruption issue was written about extensively here

https://lwn.net/Articles/774440/

as part of the early 4.19 days when a number of folks were seeing ext4 filesystem corruption as a consequence of the blk-mq bug.

Revision history for this message
Edward Vielmetti (ed-packet) wrote :

For reference, this is the extensive upstream kernel discussion

https://bugzilla.kernel.org/show_bug.cgi?id=201685

The filesystem corruption issue is real, not theoretical, and
was the extensive discussion of problems experienced during the early days of 4.19:

Marcelo Cerri (mhcerri)
Changed in linux-azure (Ubuntu Xenial):
status: New → In Progress
assignee: nobody → Marcelo Cerri (mhcerri)
Revision history for this message
Marcelo Cerri (mhcerri) wrote :
Changed in linux-azure (Ubuntu Xenial):
status: In Progress → Fix Committed
Revision history for this message
Eric Desrochers (slashd) wrote :

Would it be worth it to include the changes into Ubuntu generic kernel and derivative Ubuntu kernels as well ?

Revision history for this message
Marcelo Cerri (mhcerri) wrote :
Revision history for this message
Chris Newcomer (cnewcomer) wrote :

We had similar behavior on GCE by running Elasticsearch through the following suite of unit tests:
https://github.com/elastic/rally-eventdata-track

The test suite takes about 5 days to run fully and any corruption can be found by running:
zgrep CorruptIndexException /var/log/elasticsearch/elasticsearch.log

I was able to reproduce the corruption on all 4.15 kernels, but not on any 5.0 kernels.

I was provided a test kernel with the commits mentioned above for 4.15.0-1051-gcp and I ran it through 3 different test iterations without a single report of corruption. As a control, I used the 4.15.0-1051-gcp kernel that is currently available from the proposed pocket (without the commits referenced in this LP) and the testing logged corruption on the 3rd day of tests.

Thanks,
Chris

Marcelo Cerri (mhcerri)
Changed in linux-azure (Ubuntu Bionic):
status: New → Invalid
Changed in linux (Ubuntu Bionic):
status: New → In Progress
Changed in linux (Ubuntu Xenial):
status: New → In Progress
status: In Progress → Invalid
Changed in linux-azure (Ubuntu):
status: Confirmed → Fix Committed
Changed in linux (Ubuntu Bionic):
assignee: nobody → Marcelo Cerri (mhcerri)
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1848739

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Eric Desrochers (slashd)
tags: added: sts
Marcelo Cerri (mhcerri)
summary: - [linux-azure] Patch to prevent possible data corruption
+ [linux] Patch to prevent possible data corruption
Revision history for this message
Marcelo Cerri (mhcerri) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (43.7 KiB)

This bug was fixed in the package linux-azure - 4.15.0-1066.71

---------------
linux-azure (4.15.0-1066.71) xenial; urgency=medium

  * xenial/linux-azure: 4.15.0-1066.71 -proposed tracker (LP: #1856203)

  * [linux-azure] SAUCE patch to "Fix skb protocol value in tpacket_fill_skb()"
    (LP: #1855461)
    - SAUCE: af_packet: Fix skb protocol value in tpacket_fill_skb()

linux-azure (4.15.0-1065.70) xenial; urgency=medium

  * xenial/linux-azure: 4.15.0-1065.70 -proposed tracker (LP: #1854813)

  * [linux-azure] Patch to prevent possible data corruption (LP: #1848739)
    - blk-mq: quiesce queue during switching io sched and updating nr_requests
    - blk-mq: move hctx lock/unlock into a helper
    - blk-mq: factor out a few helpers from __blk_mq_try_issue_directly
    - blk-mq: improve DM's blk-mq IO merging via blk_insert_cloned_request
      feedback
    - dm mpath: fix missing call of path selector type->end_io
    - blk-mq-sched: remove unused 'can_block' arg from blk_mq_sched_insert_request
    - blk-mq: don't dispatch request in blk_mq_request_direct_issue if queue is
      busy
    - blk-mq: introduce BLK_STS_DEV_RESOURCE
    - blk-mq: Rename blk_mq_request_direct_issue() into
      blk_mq_request_issue_directly()
    - blk-mq: don't queue more if we get a busy return
    - blk-mq: dequeue request one by one from sw queue if hctx is busy
    - blk-mq: issue directly if hw queue isn't busy in case of 'none'
    - blk-mq: fix corruption with direct issue
    - blk-mq: fail the request in case issue failure
    - blk-mq: punt failed direct issue to dispatch list

  [ Ubuntu: 4.15.0-73.82 ]

  * bionic/linux: 4.15.0-73.82 -proposed tracker (LP: #1854819)
  * CVE-2019-14901
    - SAUCE: mwifiex: Fix heap overflow in mmwifiex_process_tdls_action_frame()
  * CVE-2019-14896 // CVE-2019-14897
    - SAUCE: libertas: Fix two buffer overflows at parsing bss descriptor
  * CVE-2019-14895
    - SAUCE: mwifiex: fix possible heap overflow in mwifiex_process_country_ie()
  * CVE-2019-18660: patches for Ubuntu (LP: #1853142) // CVE-2019-18660
    - powerpc/64s: support nospectre_v2 cmdline option
    - powerpc/book3s64: Fix link stack flush on context switch
    - KVM: PPC: Book3S HV: Flush link stack on guest exit to host kernel
  * Please add patch fixing RK818 ID detection (LP: #1853192)
    - SAUCE: mfd: rk808: Fix RK818 ID template
  * [SRU][B/OEM-B/OEM-OSP1/D] Enable new Elan touchpads which are not in current
    whitelist (LP: #1853246)
    - HID: quirks: Fix keyboard + touchpad on Lenovo Miix 630
    - Input: elan_i2c - export the device id whitelist
    - HID: quirks: Refactor ELAN 400 and 401 handling
  * Lenovo dock MAC Address pass through doesn't work in Ubuntu (LP: #1827961)
    - r8152: Add macpassthru support for ThinkPad Thunderbolt 3 Dock Gen 2
  * s390/dasd: reduce the default queue depth and nr of hardware queues
    (LP: #1852257)
    - s390/dasd: reduce the default queue depth and nr of hardware queues
  * External microphone can't work on some dell machines with the codec alc256
    or alc236 (LP: #1853791)
    - SAUCE: ALSA: hda/realtek - Move some alc256 pintbls to fallback table
    - SAUCE: ALSA: hda/realtek - Move some al...

Changed in linux-azure (Ubuntu Xenial):
status: Fix Committed → Fix Released
Stefan Bader (smb)
Changed in linux (Ubuntu Bionic):
importance: Undecided → Medium
Changed in linux (Ubuntu Bionic):
status: In Progress → Fix Committed
Marcelo Cerri (mhcerri)
Changed in linux (Ubuntu):
status: Incomplete → Fix Committed
Revision history for this message
Marcelo Cerri (mhcerri) wrote :

A test kernel for bionic linux-generic for amd64 is available at: https://kernel.ubuntu.com/~mhcerri/lp1848739/linux-generic_4.15.0-74.84+lp1848739.tgz

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (79.8 KiB)

This bug was fixed in the package linux - 4.15.0-88.88

---------------
linux (4.15.0-88.88) bionic; urgency=medium

  * bionic/linux: 4.15.0-88.88 -proposed tracker (LP: #1862824)

  * Segmentation fault (kernel oops) with memory-hotplug in
    ubuntu_kernel_selftests on Bionic kernel (LP: #1862312)
    - Revert "mm/memory_hotplug: fix online/offline_pages called w.o.
      mem_hotplug_lock"
    - mm/memory_hotplug: fix online/offline_pages called w.o. mem_hotplug_lock

linux (4.15.0-87.87) bionic; urgency=medium

  * bionic/linux: 4.15.0-87.87 -proposed tracker (LP: #1861165)

  * Bionic update: upstream stable patchset 2020-01-22 (LP: #1860602)
    - scsi: lpfc: Fix discovery failures when target device connectivity bounces
    - scsi: mpt3sas: Fix clear pending bit in ioctl status
    - scsi: lpfc: Fix locking on mailbox command completion
    - Input: atmel_mxt_ts - disable IRQ across suspend
    - iommu/tegra-smmu: Fix page tables in > 4 GiB memory
    - scsi: target: compare full CHAP_A Algorithm strings
    - scsi: lpfc: Fix SLI3 hba in loop mode not discovering devices
    - scsi: csiostor: Don't enable IRQs too early
    - powerpc/pseries: Mark accumulate_stolen_time() as notrace
    - powerpc/pseries: Don't fail hash page table insert for bolted mapping
    - powerpc/tools: Don't quote $objdump in scripts
    - dma-debug: add a schedule point in debug_dma_dump_mappings()
    - clocksource/drivers/asm9260: Add a check for of_clk_get
    - powerpc/security/book3s64: Report L1TF status in sysfs
    - powerpc/book3s64/hash: Add cond_resched to avoid soft lockup warning
    - ext4: update direct I/O read lock pattern for IOCB_NOWAIT
    - jbd2: Fix statistics for the number of logged blocks
    - scsi: tracing: Fix handling of TRANSFER LENGTH == 0 for READ(6) and WRITE(6)
    - scsi: lpfc: Fix duplicate unreg_rpi error in port offline flow
    - f2fs: fix to update dir's i_pino during cross_rename
    - clk: qcom: Allow constant ratio freq tables for rcg
    - irqchip/irq-bcm7038-l1: Enable parent IRQ if necessary
    - irqchip: ingenic: Error out if IRQ domain creation failed
    - fs/quota: handle overflows of sysctl fs.quota.* and report as unsigned long
    - scsi: lpfc: fix: Coverity: lpfc_cmpl_els_rsp(): Null pointer dereferences
    - scsi: ufs: fix potential bug which ends in system hang
    - powerpc/pseries/cmm: Implement release() function for sysfs device
    - powerpc/security: Fix wrong message when RFI Flush is disable
    - scsi: atari_scsi: sun3_scsi: Set sg_tablesize to 1 instead of SG_NONE
    - clk: pxa: fix one of the pxa RTC clocks
    - bcache: at least try to shrink 1 node in bch_mca_scan()
    - HID: logitech-hidpp: Silence intermittent get_battery_capacity errors
    - libnvdimm/btt: fix variable 'rc' set but not used
    - HID: Improve Windows Precision Touchpad detection.
    - scsi: pm80xx: Fix for SATA device discovery
    - scsi: ufs: Fix error handing during hibern8 enter
    - scsi: scsi_debug: num_tgts must be >= 0
    - scsi: NCR5380: Add disconnect_mask module parameter
    - scsi: iscsi: Don't send data to unbound connection
    - scsi: target: iscsi: Wait for all commands to finish before freeing a
...

Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Changed in linux-azure (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.