iSCSI-target: Deleting a LUN hangs in the kernel

[Pavel Zakharov]

The issue is that deleting a LUN on the target will hang until IO is attempted to this LUN from the initiator.

This issue was introduced when linux-image-generic-hwe-18.04 switched from the 5.0 to the 5.3 kernel. Source package is "linux-meta-hwe". OS is Ubuntu 18.04.4 LTS.

We can see a hung task message in dmesg:
[ 1330.438613] INFO: task iscsicmd:24572 blocked for more than 120 seconds.
[ 1330.439554] Tainted: P OE 5.3.0-26-generic #28~18.04.1-Ubuntu
[ 1330.440594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1330.441666] iscsicmd D 0 24572 8910 0x00000080
[ 1330.441668] Call Trace:
[ 1330.441674] __schedule+0x2a8/0x670
[ 1330.441676] schedule+0x33/0xa0
[ 1330.441678] schedule_timeout+0x1d3/0x2f0
[ 1330.441682] ? __kfifo_to_user_r+0xb0/0xb0
[ 1330.441684] wait_for_completion+0xba/0x140
[ 1330.441688] ? wake_up_q+0x80/0x80
[ 1330.441704] transport_clear_lun_ref+0x27/0x30 [target_core_mod]
[ 1330.441711] core_tpg_remove_lun+0x35/0x100 [target_core_mod]
[ 1330.441716] core_dev_del_lun+0x26/0x70 [target_core_mod]
[ 1330.441721] target_fabric_port_unlink+0x4a/0x50 [target_core_mod]
[ 1330.441724] configfs_unlink+0xea/0x1b0
[ 1330.441727] vfs_unlink+0x111/0x200
[ 1330.441729] do_unlinkat+0x2ad/0x320
[ 1330.441731] __x64_sys_unlink+0x23/0x30
[ 1330.441734] do_syscall_64+0x5a/0x130
[ 1330.441736] entry_SYSCALL_64_after_hwframe+0x44/0xa9
(iscsicmd is a cli tool I created that is similar to targetcli)

You can find more details about this bug and the reproducer on this linux kernel discussion thread: https://lkml.org/lkml/2020/2/7/585

A patch has been already submitted by an iSCSI-target developer, although not yet integrated. (https://marc.info/?l=target-devel&m=158134893208641&w=2)

Would it be possible to get this patch integrated into the linux-image-generic-hwe-18.04 package, and if yes, then what would be the required next steps?

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-image-generic-hwe-18.04 5.3.0.28.96
ProcVersionSignature: Ubuntu 5.3.0-28.30~18.04.1-generic 5.3.13
Uname: Linux 5.3.0-28-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.9
Architecture: amd64
Date: Mon Feb 10 18:41:12 2020
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: linux-meta-hwe
UpgradeStatus: No upgrade log present (probably fresh install)

[Pavel Zakharov (pzakha)]

Added a patch for the upstream fix (upstream commit: c14335ebb92a98646ddbf447e6cacc66de5269ad).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Pavel Zakharov (pzakha)]

BTW, this patch has been accepted into the upstream 5.4-stable branch and is part of the next patch-set: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-5.4.y&id=91aa9e475827c5cd5f0283f3f68c7805882823fe,

Hopefully this means it should be pulled into Ubuntu Focal, right?

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-eoan' to 'verification-done-eoan'. If the problem still exists, change the tag 'verification-needed-eoan' to 'verification-failed-eoan'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Khaled El Mously (kmously)]

@Pavel: Yes, it should be making its way to Focal soon if it hasn't already. Thanks for pointing that out.

[Khaled El Mously (kmously)]

@pavel: Does the current -proposed eoan kernel resolve this issue for you?

[Pavel Zakharov (pzakha)]

Hi Khaled, sorry I missed the notifications. I have not tested the -proposed version, however I've already tested the fix by rebuilding on top of Ubuntu bionic hwe, so as long as the patch is there, we can count this as fixed.

[Khaled El Mously (kmously)]

Hi @pavel. No worries! Thanks for that info - that logic sounds alright to me.

Marking as verified in eoan.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.3.0-46.38

---------------
linux (5.3.0-46.38) eoan; urgency=medium

  * eoan/linux: 5.3.0-43.36 -proposed tracker (LP: #1867301)

  * Fix AMD Stoney Ridge screen flickering under 4K resolution (LP: #1864005)
    - iommu/amd: Disable IOMMU on Stoney Ridge systems

  * Allow BPF tracing under lockdown (LP: #1868626)
    - Revert "UBUNTU: SAUCE: (efi-lockdown) Lock down kprobes"
    - Revert "bpf: Restrict bpf when kernel lockdown is in confidentiality mode"

  * Missing wireless network interface after kernel 5.3.0-43 upgrade with eoan
    (LP: #1868442)
    - iwlwifi: mvm: Do not require PHY_SKU NVM section for 3168 devices

  * Packaging resync (LP: #1786013)
    - [Packaging] resync getabis
    - [Packaging] update helper scripts

  * iSCSI-target: Deleting a LUN hangs in the kernel (LP: #1862682)
    - scsi: Revert "target/core: Inline transport_lun_remove_cmd()"

  * Stop using get_scalar_status command in Dell AIO uart backlight driver
    (LP: #1865402)
    - SAUCE: platform/x86: dell-uart-backlight: add get_display_mode command

  * Eoan update: upstream stable patchset 2020-03-11 (LP: #1867051)
    - Revert "drm/sun4i: dsi: Change the start delay calculation"
    - ovl: fix lseek overflow on 32bit
    - kernel/module: Fix memleak in module_add_modinfo_attrs()
    - media: iguanair: fix endpoint sanity check
    - ocfs2: fix oops when writing cloned file
    - x86/cpu: Update cached HLE state on write to TSX_CTRL_CPUID_CLEAR
    - udf: Allow writing to 'Rewritable' partitions
    - printk: fix exclusive_console replaying
    - iwlwifi: mvm: fix NVM check for 3168 devices
    - sparc32: fix struct ipc64_perm type definition
    - cls_rsvp: fix rsvp_policy
    - gtp: use __GFP_NOWARN to avoid memalloc warning
    - l2tp: Allow duplicate session creation with UDP
    - net: hsr: fix possible NULL deref in hsr_handle_frame()
    - net_sched: fix an OOB access in cls_tcindex
    - net: stmmac: Delete txtimer in suspend()
    - bnxt_en: Fix TC queue mapping.
    - tcp: clear tp->total_retrans in tcp_disconnect()
    - tcp: clear tp->delivered in tcp_disconnect()
    - tcp: clear tp->data_segs{in|out} in tcp_disconnect()
    - tcp: clear tp->segs_{in|out} in tcp_disconnect()
    - rxrpc: Fix use-after-free in rxrpc_put_local()
    - rxrpc: Fix insufficient receive notification generation
    - rxrpc: Fix missing active use pinning of rxrpc_local object
    - rxrpc: Fix NULL pointer deref due to call->conn being cleared on disconnect
    - media: uvcvideo: Avoid cyclic entity chains due to malformed USB descriptors
    - mfd: dln2: More sanity checking for endpoints
    - ipc/msg.c: consolidate all xxxctl_down() functions
    - tracing: Fix sched switch start/stop refcount racy updates
    - rcu: Avoid data-race in rcu_gp_fqs_check_wake()
    - brcmfmac: Fix memory leak in brcmf_usbdev_qinit
    - usb: typec: tcpci: mask event interrupts when remove driver
    - usb: gadget: legacy: set max_speed to super-speed
    - usb: gadget: f_ncm: Use atomic_t to track in-flight request
    - usb: gadget: f_ecm: Use atomic_t to track in-flight request
    - ALSA: usb-audio: Fix endianess in descriptor validatio...