linux-aws: Xen: Issues with detaching volume

Bug #1966969 reported by Tim Gardner
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux-aws (Ubuntu)
Fix Released
Tim Gardner

Bug Description

SRU Justification


We are observing issue with the secondary volume stuck in detaching. This is observed with the latest Canonical, Ubuntu EKS Node OS (k8s_1.19), 20.04 LTS, amd64 focal image build on 2022-03-08 and Xen instance type( for eg : m4, t2 instance type )
AMI in eu-west-1 : ami-0f4ffbcba23a6c434
AMI in us-east-1 : ami-021feb4aa3b3c59c3

When terminating an instance with a stuck volume the shutdown process is interrupted by a xen task hanging:
[ 847.895334] INFO: task xenwatch:188 blocked for more than 483 seconds.
[ 847.901573] Not tainted 5.13.0-1017-aws #19~20.04.1-Ubuntu
[ 847.907144] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 847.914462] task:xenwatch state:D stack: 0 pid: 188 ppid: 2 flags:0x00004000
[ 847.914467] Call Trace:
[ 847.914469] <TASK>
[ 847.914472] __schedule+0x2ee/0x900
[ 847.914478] schedule+0x4f/0xc0
[ 847.914479] schedule_preempt_disabled+0xe/0x10
[ 847.914482] __mutex_lock.isra.0+0x183/0x4d0
[ 847.914486] __mutex_lock_slowpath+0x13/0x20
[ 847.914487] mutex_lock+0x32/0x40
[ 847.914489] del_gendisk+0x90/0x200
[ 847.914493] xlvbd_release_gendisk+0x72/0xc0
[ 847.914499] blkback_changed+0x101/0x210
[ 847.914502] xenbus_otherend_changed+0x8f/0x130
[ 847.914507] backend_changed+0x13/0x20
[ 847.914510] xenwatch_thread+0xa6/0x180
[ 847.914513] ? wait_woken+0x80/0x80
[ 847.914517] ? test_reply.isra.0+0x40/0x40
[ 847.914520] kthread+0x12b/0x150
[ 847.914523] ? set_kthread_struct+0x40/0x40
[ 847.914525] ret_from_fork+0x22/0x30
[ 847.914531] </TASK>

this looks like it's waiting on a xen block device to be released.

Following steps used to reproduce the issue:
* Created a m4,t2(xen) instance with the latest ami for latest Canonical, Ubuntu EKS Node OS (k8s_1.19), 20.04 LTS, amd64
* Created a filesystem on volume
* Mounted volume through OS
* Unmounted volume in OS
* Detached volume from AWS console
* Volume gets stuck.

We at the internal team observed this commit upstream:

[...] and a del_gendisk from the block device release
method, which will deadlock.

It has a Fixes: tag referring to a commit from 5.13 so this could be the root-cause. While testing, we observed this commit is fixing the issue.

[Test Plan]

Amazon tested

[Where things could go wrong]

Detaching volumes could fail in new and bizarre ways

[Other Info]

SF: #00331175

CVE References

Revision history for this message
Tim Gardner (timg-tpi) wrote :
Stefan Bader (smb)
Changed in linux-aws (Ubuntu Impish):
assignee: nobody → Tim Gardner (timg-tpi)
importance: Undecided → Medium
status: New → In Progress
Changed in linux-aws (Ubuntu):
status: New → Invalid
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-aws/5.13.0-1022.24 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-impish' to 'verification-done-impish'. If the problem still exists, change the tag 'verification-needed-impish' to 'verification-failed-impish'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-impish
Revision history for this message
Tim Gardner (timg-tpi) wrote :

Amazon tested

tags: added: verification-done-impish
removed: verification-needed-impish
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (53.4 KiB)

This bug was fixed in the package linux-aws - 5.13.0-1022.24

linux-aws (5.13.0-1022.24) impish; urgency=medium

  * impish/linux-aws: 5.13.0-1022.24 -proposed tracker (LP: #1967345)

  * linux-aws: Xen: Issues with detaching volume (LP: #1966969)
    - xen-blkfront: sanitize the removal state machine

  * Miscellaneous upstream changes
    - block: add blk_alloc_disk and blk_cleanup_disk APIs

  [ Ubuntu: 5.13.0-40.45 ]

  * impish/linux: 5.13.0-40.45 -proposed tracker (LP: #1966701)
  * CVE-2022-1016
    - netfilter: nf_tables: initialize registers in nft_do_chain()
  * CVE-2022-1015
    - netfilter: nf_tables: validate registers coming from userspace.
  * audit: improve audit queue handling when "audit=1" on cmdline
    (LP: #1965723) // Impish update: upstream stable patchset 2022-03-22
    (LP: #1966021)
    - audit: improve audit queue handling when "audit=1" on cmdline
  * PS/2 Keyboard wakeup from s2idle not functioning on AMD Yellow Carp platform
    (LP: #1961739)
    - PM: s2idle: ACPI: Fix wakeup interrupts handling
  * Low RX performance for 40G Solarflare NICs (LP: #1964512)
    - SAUCE: sfc: The size of the RX recycle ring should be more flexible
  * [UBUNTU 20.04] Fix SIGP processing on KVM/s390 (LP: #1962578)
    - KVM: s390: Simplify SIGP Set Arch handling
    - KVM: s390: Add a routine for setting userspace CPU state
  * Move virtual graphics drivers from linux-modules-extra to linux-modules
    (LP: #1960633)
    - [Packaging] Move VM DRM drivers into modules
  * Impish update: upstream stable patchset 2022-03-09 (LP: #1964422)
    - bnx2x: Utilize firmware
    - bnx2x: Invalidate fastpath HSI version for VFs
    - rcu: Tighten rcu_advance_cbs_nowake() checks
    - select: Fix indefinitely sleeping task in poll_schedule_timeout()
    - drm/amdgpu: Use correct VIEWPORT_DIMENSION for DCN2
    - arm64/bpf: Remove 128MB limit for BPF JIT programs
    - Bluetooth: refactor malicious adv data check
    - net: sfp: ignore disabled SFP node
    - net: stmmac: skip only stmmac_ptp_register when resume from suspend
    - s390/hypfs: include z/VM guests with access control group set
    - bpf: Guard against accessing NULL pt_regs in bpf_get_task_stack()
    - scsi: zfcp: Fix failed recovery on gone remote port with non-NPIV FCP
    - udf: Restore i_lenAlloc when inode expansion fails
    - udf: Fix NULL ptr deref when converting from inline format
    - efi: runtime: avoid EFIv2 runtime services on Apple x86 machines
    - PM: wakeup: simplify the output logic of pm_show_wakelocks()
    - tracing/histogram: Fix a potential memory leak for kstrdup()
    - tracing: Don't inc err_log entry count if entry allocation fails
    - ceph: properly put ceph_string reference after async create attempt
    - ceph: set pool_ns in new inode layout for async creates
    - fsnotify: fix fsnotify hooks in pseudo filesystems
    - Revert "KVM: SVM: avoid infinite loop on NPF from bad address"
    - perf/x86/intel/uncore: Fix CAS_COUNT_WRITE issue for ICX
    - drm/etnaviv: relax submit size limits
    - KVM: x86: Update vCPU's runtime CPUID on write to MSR_IA32_XSS
    - netfilter: nft_payload: do not update lay...

Changed in linux-aws (Ubuntu Impish):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers