[Hyper-V] Fiber Channel critical target error

[Gergo Debreczeni]

System details:
Linux version 3.19.0-11-generic (apw@gloin) (gcc version 4.9.2 (Ubuntu 4.9.2-10ubuntu12) ) #11~lp1423343v201503311833 SMP Tue Mar 31 17:40:01 UTC 2015 (Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3)

Ubuntu 3.19.0-11
Ubuntu Vivid 15.04 64bit custom kernel
Platform: WS2012R2

Using a fiber channel disk partition ( dev/sde1 in this example) outputs the following error:
​
[ 1307.744496] blk_update_request: critical target error, dev sde, sector 0
[ 1396.052488] blk_update_request: critical target error, dev sde, sector 0
[ 1456.216570] blk_update_request: critical target error, dev sde, sector 0

Creating a partition works, reading too, but first time writing it hangs for around 20 seconds and after a couple more writes the partition goes into read-only mode:

#echo "test1" > test1.txt
*hangs for 30 seconds*
blk_update_request: critical target error, dev sde, sector 0
# echo "test2" > test2.txt
blk_update_request: critical target error, dev sde, sector 289671192
Aborting journal on device sde1-8.
EXT4-fs error (device sde1): ext4_journal_check_start:56: Detected aborted journal
EXT4-fs (sde1): Remounting filesystem read-only
# echo "test string" > sample.txt
-bash: sample.txt: Read-only file system

Dmesg log:
[ 461.284362] blk_update_request: critical target error, dev sde, sector 0
[ 484.845659] EXT4-fs (sde1): mounted filesystem with ordered data mode. Opts: (null)
[ 548.988377] blk_update_request: critical target error, dev sde, sector 0
[ 609.216340] blk_update_request: critical target error, dev sde, sector 289671192
[ 609.221812] Aborting journal on device sde1-8.
[ 612.237503] EXT4-fs error (device sde1): ext4_journal_check_start:56: Detected aborted journal
[ 612.249529] EXT4-fs (sde1): Remounting filesystem read-only

Additional info:

cat /proc/version_signature
Ubuntu 3.19.0-11.11~lp1423343v201503311833-generic 3.19.3

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1439780

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.1 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc3-vivid/

[Gergo Debreczeni (gdebreczeni)]

Checked with http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc3-unstable/ The issue is still present.

root@ubuntu1504:/mnt# uname -a
Linux ubuntu1504 4.1.0-040100rc3-generic #201505102036 SMP Mon May 11 00:37:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

In dmesg we still see :

[ 57.402313] sdb:
[ 117.432523] blk_update_request: critical target error, dev sdb, sector 0
[ 133.723833] sdb: sdb1
[ 193.752349] blk_update_request: critical target error, dev sdb, sector 0
[ 304.792591] blk_update_request: critical target error, dev sdb, sector 0
[ 364.844409] blk_update_request: critical target error, dev sdb, sector 0

[Joseph Salisbury (jsalisbury)]

Was there a prior kernel version that did not have this issue?

[Joshua R. Poulson (jrp)]

We do not have a test in the 15.04 timeframe that worked, verifying that it worked in previous releases.

[Alex Ng (alexng-v)]

So I tried to repro this on the latest upstream v4.1-fc5 kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc5-unstable/.

It looks like the issue's been fixed there.

[Joseph Salisbury (jsalisbury)]

'd like to perform a reverse bisect to figure out which commit upstream fixes this regression. It would be very helpful to know the last kernel that had this issue and the first kernel that did not.

Can you test the following kernels and report back? We are looking for the earliest kernel version that doesn't have this bug:

v4.1-rc4: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc4-unstable/
v4.1-rc3: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc3-unstable/
v4.1-rc2: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc2-unstable/
v4.1-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc1-unstable/

You don't have to test every kernel. We are just looking for the last kernel that had the bug and the first kernel that did not. Once we know these two versions, we can bisect between them.

[Alex Ng (alexng-v)]

To answer your question, it looks like the issue can be reproduced as recently as v4.1-rc4.

So the last version of the kernel that we see this bug is: v4.1-rc4

And the first version of the kernel we don't see this bug is: v4.1-rc5

[Joseph Salisbury (jsalisbury)]

I started a kernel reverse bisect between v4.1-rc4 and v4.1-rc5. The kernel bisect will require testing of about 7-10 test kernels.

I built the first test kernel, up to the following commit:
1d82b0baf9abd59ae6f53f3102f4e442043763a4

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1439780

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Alex Ng (alexng-v)]

The test kernel still hits this issue.

[Joseph Salisbury (jsalisbury)]

I built the first test kernel, up to the following commit:
0b6280c62026168f79ff4dd1437df131bdfd24f2

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1439780

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Alex Ng (alexng-v)]

Still not working with the kernel up to this commit: 0b6280c62026168f79ff4dd1437df131bdfd24f2

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
b0f155ada4c819f06aa32b4c906e7e76350c7ec1

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1439780

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Alex Ng (alexng-v)]

The issue also shows in this kernel.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
7ce14f6ff26460819345fe8495cf2dd6538b7cdc

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1439780

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Alex Ng (alexng-v)]

Hi Joseph,

Kernel with commits up to 7ce14f6ff26460819345fe8495cf2dd6538b7cdc are also seeing this issue.

I'm can run my own bisect as well so we can save some time going back and forth. Are you basing your bisect off of Linux-stable repository?

Thanks,
Alex

[Joseph Salisbury (jsalisbury)]

Thanks for the update, Alex.

I am performing a "Reverse" bisect between v4.1-rc4 and v4.1-rc5 using the master branch in the mainline repo:
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

I'll continue the bisect unless you prefer to do it. Here is the current bisect log:

# good: [e26081808edadfd257c6c9d81014e3b25e9a6118] Linux 4.1-rc4
git bisect good e26081808edadfd257c6c9d81014e3b25e9a6118
# bad: [ba155e2d21f6bf05de86a78dbe5bfd8757604a65] Linux 4.1-rc5
git bisect bad ba155e2d21f6bf05de86a78dbe5bfd8757604a65
# good: [1d82b0baf9abd59ae6f53f3102f4e442043763a4] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
git bisect good 1d82b0baf9abd59ae6f53f3102f4e442043763a4
# good: [0b6280c62026168f79ff4dd1437df131bdfd24f2] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
git bisect good 0b6280c62026168f79ff4dd1437df131bdfd24f2
# good: [b0f155ada4c819f06aa32b4c906e7e76350c7ec1] drm/exynos: dp: Lower level of EDID read success message
git bisect good b0f155ada4c819f06aa32b4c906e7e76350c7ec1
# good: [7ce14f6ff26460819345fe8495cf2dd6538b7cdc] Merge branch 'for-linus-4.1' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect good 7ce14f6ff26460819345fe8495cf2dd6538b7cdc

[Joseph Salisbury (jsalisbury)]

I'll continue the bisect until I hear back.

I built the next test kernel, up to the following commit:
c5db6a3bdeb72f4238e1faefa4ce4eab7a64baea

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1439780

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Alex Ng (alexng-v)]

Hi,

The issue also shows up in this kernel.

I can take over the bisect. My current bisect log is the same as yours.

I'm currently building the next test kernel up to following commit:
74856fbf441929918c49ff262ace9835048e4e6a

Will let you know when I narrow it down further.

[Joseph Salisbury (jsalisbury)]

Thanks, Alex. I look forward to your results.

[Alex Ng (alexng-v)]

Hi,

I'm finished with the bisect and the following commit will fix this issue:

dc45708ca9988656d706940df5fd102672c5de92 [storvsc: Set the SRB flags correctly when no data transfer is needed]

Looks like we'll want to include this patch.

Here's the rest of my bisect log, in case you're interested:

# bad: [ba155e2d21f6bf05de86a78dbe5bfd8757604a65] Linux 4.1-rc5
# good: [e26081808edadfd257c6c9d81014e3b25e9a6118] Linux 4.1-rc4
git bisect start 'v4.1-rc5' 'v4.1-rc4'
# good: [1d82b0baf9abd59ae6f53f3102f4e442043763a4] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
git bisect good 1d82b0baf9abd59ae6f53f3102f4e442043763a4
# good: [0b6280c62026168f79ff4dd1437df131bdfd24f2] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
git bisect good 0b6280c62026168f79ff4dd1437df131bdfd24f2
# good: [b0f155ada4c819f06aa32b4c906e7e76350c7ec1] drm/exynos: dp: Lower level of EDID read success message
git bisect good b0f155ada4c819f06aa32b4c906e7e76350c7ec1
# good: [7ce14f6ff26460819345fe8495cf2dd6538b7cdc] Merge branch 'for-linus-4.1' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect good 7ce14f6ff26460819345fe8495cf2dd6538b7cdc
# good: [c5db6a3bdeb72f4238e1faefa4ce4eab7a64baea] Merge branch 'timers-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good c5db6a3bdeb72f4238e1faefa4ce4eab7a64baea
# bad: [74856fbf441929918c49ff262ace9835048e4e6a] sd: Disable support for 256 byte/sector disks
git bisect bad 74856fbf441929918c49ff262ace9835048e4e6a
# bad: [8b2564ec7410928639db5c09a34d7d8330f1d759] lpfc: Fix breakage on big endian kernels
git bisect bad 8b2564ec7410928639db5c09a34d7d8330f1d759
# bad: [dc45708ca9988656d706940df5fd102672c5de92] storvsc: Set the SRB flags correctly when no data transfer is needed
git bisect bad dc45708ca9988656d706940df5fd102672c5de92
# first bad commit: [dc45708ca9988656d706940df5fd102672c5de92] storvsc: Set the SRB flags correctly when no data transfer is needed

[Joseph Salisbury (jsalisbury)]

I built a Vivid test kernel with a cherry pick of commit dc45708c. Can you test this kernel and confirm it solves the bug? If it does, we can SRU it to Vivid and Wily and request it be included in the upstream stable kernels.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1439780/

Note, for this test kernel you need to install both the linux-image and linux-image-extra .deb packages.

[Alex Ng (alexng-v)]

This test kernel fixes the issue.

[Joseph Salisbury (jsalisbury)]

Thanks for testing, Alex. I sent and SRU request to have this commit included in Vivid and Wily. The commit was already cc'd to stable as well.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.19.0-22.22

---------------
linux (3.19.0-22.22) vivid; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1465755

  [ Tai Nguyen ]

  * SAUCE: power: reset: Add syscon reboot device node for APM X-Gene
    platform
    - LP: #1463211

  [ Upstream Kernel Changes ]

  * Revert "dm crypt: fix deadlock when async crypto algorithm returns
    -EBUSY"
    - LP: #1465696
  * Bluetooth: ath3k: Add a new ID 0cf3:e006 to ath3k list
    - LP: #1459934
  * cdc-acm: prevent infinite loop when parsing CDC headers.
    - LP: #1460657
  * (upstream) libata: Blacklist queued TRIM on all Samsung 800-series
    - LP: #1338706, #1449005
  * powerpc/powernv: Check image loaded or not before calling flash
    - LP: #1461553
  * ahci: avoton port-disable reset-quirk
    - LP: #1458617
  * Bluetooth: btusb: support public address configuration for ath3012
    - LP: #1459937
  * Bluetooth: btusb: Add setup callback for chip init on USB
    - LP: #1459937
  * Bluetooth: btusb: Add support for QCA ROME chipset family
    - LP: #1459937
  * Bluetooth: btusb: Fix incorrect type in qca_device_info
    - LP: #1459937
  * Bluetooth: btusb: Fix minor whitespace issue in QCA ROME device entries
    - LP: #1459937
  * Bluetooth: btusb: Add support for 0cf3:e007
    - LP: #1459937
  * storvsc: Set the SRB flags correctly when no data transfer is needed
    - LP: #1439780
  * vfs: read file_handle only once in handle_to_path
    - LP: #1416503
    - CVE-2015-1420
  * ozwpan: Use unsigned ints to prevent heap overflow
    - LP: #1463442
    - CVE-2015-4001
  * ozwpan: divide-by-zero leading to panic
    - LP: #1463445
    - CVE-2015-4003
  * ozwpan: Use proper check to prevent heap overflow
    - LP: #1463444
    - CVE-2015-4002
  * ozwpan: unchecked signed subtraction leads to DoS
    - LP: #1463444
    - CVE-2015-4002
  * enclosure: fix WARN_ON removing an adapter in multi-path devices
    - LP: #1415178
  * ASoC: tfa9879: Fix return value check in tfa9879_i2c_probe()
    - LP: #1465696
  * ASoC: samsung: s3c24xx-i2s: Fix return value check in
    s3c24xx_iis_dev_probe()
    - LP: #1465696
  * ASoC: dapm: Enable autodisable on SOC_DAPM_SINGLE_TLV_AUTODISABLE
    - LP: #1465696
  * ASoC: rt5677: add register patch for PLL
    - LP: #1465696
  * btrfs: unlock i_mutex after attempting to delete subvolume during send
    - LP: #1465696
  * ALSA: hda - Fix mute-LED fixed mode
    - LP: #1465696
  * ALSA: hda - Add mute-LED mode control to Thinkpad
    - LP: #1465696
  * arm64: dma-mapping: always clear allocated buffers
    - LP: #1465696
  * ALSA: emu10k1: Fix card shortname string buffer overflow
    - LP: #1465696
  * ALSA: emux: Fix mutex deadlock at unloading
    - LP: #1465696
  * drm/radeon: Use drm_calloc_ab for CS relocs
    - LP: #1465696
  * drm/radeon: adjust pll when audio is not enabled
    - LP: #1465696
  * drm/radeon: add SI DPM quirk for Sapphire R9 270 Dual-X 2G GDDR5
    - LP: #1465696
  * drm/radeon: fix lockup when BOs aren't part of the VM on release
    - LP: #1465696
  * drm/radeon: reset BOs address after clearing it.
    - LP: #1465696
  * drm/radeon: check new address before removing old one
  ...

[Luis Henriques (henrix)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-vivid' to 'verification-done-vivid'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Chris Valean (cvalean)]

Hi Luis,
We've tested now the proposed kernel 3.19.0-22 on Vivid and the virtual fibre channel is working and it's usable.

There is one error message that we see when checking the multipath support, will send an update once that gets clarified on our end.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.19.0-22.22

---------------
linux (3.19.0-22.22) vivid; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1465755

  [ Tai Nguyen ]

  * SAUCE: power: reset: Add syscon reboot device node for APM X-Gene
    platform
    - LP: #1463211

  [ Upstream Kernel Changes ]

  * Revert "dm crypt: fix deadlock when async crypto algorithm returns
    -EBUSY"
    - LP: #1465696
  * Bluetooth: ath3k: Add a new ID 0cf3:e006 to ath3k list
    - LP: #1459934
  * cdc-acm: prevent infinite loop when parsing CDC headers.
    - LP: #1460657
  * (upstream) libata: Blacklist queued TRIM on all Samsung 800-series
    - LP: #1338706, #1449005
  * powerpc/powernv: Check image loaded or not before calling flash
    - LP: #1461553
  * ahci: avoton port-disable reset-quirk
    - LP: #1458617
  * Bluetooth: btusb: support public address configuration for ath3012
    - LP: #1459937
  * Bluetooth: btusb: Add setup callback for chip init on USB
    - LP: #1459937
  * Bluetooth: btusb: Add support for QCA ROME chipset family
    - LP: #1459937
  * Bluetooth: btusb: Fix incorrect type in qca_device_info
    - LP: #1459937
  * Bluetooth: btusb: Fix minor whitespace issue in QCA ROME device entries
    - LP: #1459937
  * Bluetooth: btusb: Add support for 0cf3:e007
    - LP: #1459937
  * storvsc: Set the SRB flags correctly when no data transfer is needed
    - LP: #1439780
  * vfs: read file_handle only once in handle_to_path
    - LP: #1416503
    - CVE-2015-1420
  * ozwpan: Use unsigned ints to prevent heap overflow
    - LP: #1463442
    - CVE-2015-4001
  * ozwpan: divide-by-zero leading to panic
    - LP: #1463445
    - CVE-2015-4003
  * ozwpan: Use proper check to prevent heap overflow
    - LP: #1463444
    - CVE-2015-4002
  * ozwpan: unchecked signed subtraction leads to DoS
    - LP: #1463444
    - CVE-2015-4002
  * enclosure: fix WARN_ON removing an adapter in multi-path devices
    - LP: #1415178
  * ASoC: tfa9879: Fix return value check in tfa9879_i2c_probe()
    - LP: #1465696
  * ASoC: samsung: s3c24xx-i2s: Fix return value check in
    s3c24xx_iis_dev_probe()
    - LP: #1465696
  * ASoC: dapm: Enable autodisable on SOC_DAPM_SINGLE_TLV_AUTODISABLE
    - LP: #1465696
  * ASoC: rt5677: add register patch for PLL
    - LP: #1465696
  * btrfs: unlock i_mutex after attempting to delete subvolume during send
    - LP: #1465696
  * ALSA: hda - Fix mute-LED fixed mode
    - LP: #1465696
  * ALSA: hda - Add mute-LED mode control to Thinkpad
    - LP: #1465696
  * arm64: dma-mapping: always clear allocated buffers
    - LP: #1465696
  * ALSA: emu10k1: Fix card shortname string buffer overflow
    - LP: #1465696
  * ALSA: emux: Fix mutex deadlock at unloading
    - LP: #1465696
  * drm/radeon: Use drm_calloc_ab for CS relocs
    - LP: #1465696
  * drm/radeon: adjust pll when audio is not enabled
    - LP: #1465696
  * drm/radeon: add SI DPM quirk for Sapphire R9 270 Dual-X 2G GDDR5
    - LP: #1465696
  * drm/radeon: fix lockup when BOs aren't part of the VM on release
    - LP: #1465696
  * drm/radeon: reset BOs address after clearing it.
    - LP: #1465696
  * drm/radeon: check new address before removing old one
  ...

[Joshua R. Poulson (jrp)]

We are still seeing some errors.

Here are the results for 15.04 running the upstream 4.1.0-rc8-next built 2015-06-22.
I’m still seeing some error messages, though multipath is at least able to recognize the vFC disks.

root@ubuntu1504:~# multipath -ll
360080e500017c5e2000016bf53d7c0c8 dm-3 IBM,1814 FAStT
size=279G features='1 queue_if_no_path' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=6 status=active
| |- 4:0:1:0 sdd 8:48 active ready running
| `- 4:0:3:0 sdf 8:80 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  |- 4:0:0:0 sdc 8:32 active ghost running
  `- 4:0:2:0 sde 8:64 active ghost running
360022480252826b1808fb8b00780a6be dm-2 Msft,Virtual Disk
size=4.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 3:0:0:0 sdb 8:16 active ready running

Dmesg messages:
[ 140.678778] device-mapper: table: 252:5: multipath: error getting device
[ 140.678888] device-mapper: ioctl: error adding target to table
[ 140.706106] device-mapper: table: 252:5: multipath: error getting device
[ 140.706282] device-mapper: ioctl: error adding target to table

These 2 items might be specific to the FC or can be ignored, still mentioning them for confirmation:
1. The [current] illegal request is still showing up.
2. “failure for create/reload map” messages still appear.

Should we open a new bug or continue with this one?

[Joseph Salisbury (jsalisbury)]

Joshua, it is probably best to open a new bug.

[Joseph Salisbury (jsalisbury)]

The fix for this bug is now in Trusty as commit:

f3599b8 storvsc: Set the SRB flags correctly when no data transfer is needed