linux-aws: Xen: Issues with detaching volume

[Tim Gardner]

SRU Justification

[Impact]

We are observing issue with the secondary volume stuck in detaching. This is observed with the latest Canonical, Ubuntu EKS Node OS (k8s_1.19), 20.04 LTS, amd64 focal image build on 2022-03-08 and Xen instance type( for eg : m4, t2 instance type )
AMI in eu-west-1 : ami-0f4ffbcba23a6c434
AMI in us-east-1 : ami-021feb4aa3b3c59c3

When terminating an instance with a stuck volume the shutdown process is interrupted by a xen task hanging:
[ 847.895334] INFO: task xenwatch:188 blocked for more than 483 seconds.
[ 847.901573] Not tainted 5.13.0-1017-aws #19~20.04.1-Ubuntu
[ 847.907144] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 847.914462] task:xenwatch state:D stack: 0 pid: 188 ppid: 2 flags:0x00004000
[ 847.914467] Call Trace:
[ 847.914469] <TASK>
[ 847.914472] __schedule+0x2ee/0x900
[ 847.914478] schedule+0x4f/0xc0
[ 847.914479] schedule_preempt_disabled+0xe/0x10
[ 847.914482] __mutex_lock.isra.0+0x183/0x4d0
[ 847.914486] __mutex_lock_slowpath+0x13/0x20
[ 847.914487] mutex_lock+0x32/0x40
[ 847.914489] del_gendisk+0x90/0x200
[ 847.914493] xlvbd_release_gendisk+0x72/0xc0
[ 847.914499] blkback_changed+0x101/0x210
[ 847.914502] xenbus_otherend_changed+0x8f/0x130
[ 847.914507] backend_changed+0x13/0x20
[ 847.914510] xenwatch_thread+0xa6/0x180
[ 847.914513] ? wait_woken+0x80/0x80
[ 847.914517] ? test_reply.isra.0+0x40/0x40
[ 847.914520] kthread+0x12b/0x150
[ 847.914523] ? set_kthread_struct+0x40/0x40
[ 847.914525] ret_from_fork+0x22/0x30
[ 847.914531] </TASK>

this looks like it's waiting on a xen block device to be released.

Following steps used to reproduce the issue:
* Created a m4,t2(xen) instance with the latest ami for latest Canonical, Ubuntu EKS Node OS (k8s_1.19), 20.04 LTS, amd64
* Created a filesystem on volume
* Mounted volume through OS
* Unmounted volume in OS
* Detached volume from AWS console
* Volume gets stuck.

We at the internal team observed this commit upstream: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=05d69d950d9d84218fc9beafd02dea1f6a70e09e

[...] and a del_gendisk from the block device release
method, which will deadlock.

It has a Fixes: tag referring to a commit from 5.13 so this could be the root-cause. While testing, we observed this commit is fixing the issue.

[Test Plan]

Amazon tested

[Where things could go wrong]

Detaching volumes could fail in new and bizarre ways

[Other Info]

SF: #00331175

[Tim Gardner (timg-tpi)]

Patches submitted: https://lists.ubuntu.com/archives/kernel-team/2022-March/129029.html

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws/5.13.0-1022.24 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-impish' to 'verification-done-impish'. If the problem still exists, change the tag 'verification-needed-impish' to 'verification-failed-impish'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Tim Gardner (timg-tpi)]

Amazon tested

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-aws - 5.13.0-1022.24

---------------
linux-aws (5.13.0-1022.24) impish; urgency=medium

  * impish/linux-aws: 5.13.0-1022.24 -proposed tracker (LP: #1967345)

  * linux-aws: Xen: Issues with detaching volume (LP: #1966969)
    - xen-blkfront: sanitize the removal state machine

  * Miscellaneous upstream changes
    - block: add blk_alloc_disk and blk_cleanup_disk APIs

  [ Ubuntu: 5.13.0-40.45 ]

  * impish/linux: 5.13.0-40.45 -proposed tracker (LP: #1966701)
  * CVE-2022-1016
    - netfilter: nf_tables: initialize registers in nft_do_chain()
  * CVE-2022-1015
    - netfilter: nf_tables: validate registers coming from userspace.
  * audit: improve audit queue handling when "audit=1" on cmdline
    (LP: #1965723) // Impish update: upstream stable patchset 2022-03-22
    (LP: #1966021)
    - audit: improve audit queue handling when "audit=1" on cmdline
  * PS/2 Keyboard wakeup from s2idle not functioning on AMD Yellow Carp platform
    (LP: #1961739)
    - PM: s2idle: ACPI: Fix wakeup interrupts handling
  * Low RX performance for 40G Solarflare NICs (LP: #1964512)
    - SAUCE: sfc: The size of the RX recycle ring should be more flexible
  * [UBUNTU 20.04] Fix SIGP processing on KVM/s390 (LP: #1962578)
    - KVM: s390: Simplify SIGP Set Arch handling
    - KVM: s390: Add a routine for setting userspace CPU state
  * Move virtual graphics drivers from linux-modules-extra to linux-modules
    (LP: #1960633)
    - [Packaging] Move VM DRM drivers into modules
  * Impish update: upstream stable patchset 2022-03-09 (LP: #1964422)
    - bnx2x: Utilize firmware 7.13.21.0
    - bnx2x: Invalidate fastpath HSI version for VFs
    - rcu: Tighten rcu_advance_cbs_nowake() checks
    - select: Fix indefinitely sleeping task in poll_schedule_timeout()
    - drm/amdgpu: Use correct VIEWPORT_DIMENSION for DCN2
    - arm64/bpf: Remove 128MB limit for BPF JIT programs
    - Bluetooth: refactor malicious adv data check
    - net: sfp: ignore disabled SFP node
    - net: stmmac: skip only stmmac_ptp_register when resume from suspend
    - s390/hypfs: include z/VM guests with access control group set
    - bpf: Guard against accessing NULL pt_regs in bpf_get_task_stack()
    - scsi: zfcp: Fix failed recovery on gone remote port with non-NPIV FCP
      devices
    - udf: Restore i_lenAlloc when inode expansion fails
    - udf: Fix NULL ptr deref when converting from inline format
    - efi: runtime: avoid EFIv2 runtime services on Apple x86 machines
    - PM: wakeup: simplify the output logic of pm_show_wakelocks()
    - tracing/histogram: Fix a potential memory leak for kstrdup()
    - tracing: Don't inc err_log entry count if entry allocation fails
    - ceph: properly put ceph_string reference after async create attempt
    - ceph: set pool_ns in new inode layout for async creates
    - fsnotify: fix fsnotify hooks in pseudo filesystems
    - Revert "KVM: SVM: avoid infinite loop on NPF from bad address"
    - perf/x86/intel/uncore: Fix CAS_COUNT_WRITE issue for ICX
    - drm/etnaviv: relax submit size limits
    - KVM: x86: Update vCPU's runtime CPUID on write to MSR_IA32_XSS
    - netfilter: nft_payload: do not update lay...