mdadm, mkfs, other io commands hang, stuck task, bad rip

[Ryan Harper]

1. disco
2. # apt-cache policy linux-image-virtual
linux-image-virtual:
  Installed: 5.0.0.13.14
  Candidate: 5.0.0.13.14
  Version table:
 *** 5.0.0.13.14 500
        500 http://archive.ubuntu.com/ubuntu disco/main amd64 Packages
        100 /var/lib/dpkg/status
3. installation completes sucessfully
4. Running an installation of Disco creating filesystems and probing things will hang (stuck task) every so often. When it happens, the kernel throws an oops and bad rip value.

[ 967.810391] INFO: task mdadm:12213 blocked for more than 120 seconds.
[ 967.811787] Tainted: P O 5.0.0-13-generic #14-Ubuntu
[ 967.813330] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 967.815500] mdadm D 0 12213 12207 0x00000000
[ 967.815503] Call Trace:
[ 967.815509] __schedule+0x2d0/0x840
[ 967.815511] ? __switch_to_asm+0x40/0x70
[ 967.815512] ? __switch_to_asm+0x34/0x70
[ 967.815514] schedule+0x2c/0x70
[ 967.815516] schedule_preempt_disabled+0xe/0x10
[ 967.815518] __mutex_lock.isra.10+0x2e4/0x4c0
[ 967.815522] ? exact_lock+0x11/0x20
[ 967.815524] __mutex_lock_slowpath+0x13/0x20
[ 967.815528] mutex_lock+0x2c/0x30
[ 967.815530] __blkdev_get+0x7b/0x550
[ 967.815532] ? bd_acquire+0xd0/0xd0
[ 967.815533] blkdev_get+0x10c/0x330
[ 967.815535] ? bd_acquire+0xd0/0xd0
[ 967.815537] blkdev_open+0x92/0x100
[ 967.815539] do_dentry_open+0x143/0x3a0
[ 967.815540] vfs_open+0x2d/0x30
[ 967.815542] path_openat+0x2d4/0x16d0
[ 967.815547] ? put_device+0x17/0x20
[ 967.815552] ? scsi_device_put+0x2b/0x30
[ 967.815553] do_filp_open+0x93/0x100
[ 967.815555] ? strncpy_from_user+0x56/0x1b0
[ 967.815558] ? __alloc_fd+0x46/0x140
[ 967.815560] do_sys_open+0x177/0x280
[ 967.815561] ? _cond_resched+0x19/0x30
[ 967.815563] __x64_sys_openat+0x20/0x30
[ 967.815566] do_syscall_64+0x5a/0x110
[ 967.815567] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 967.815568] RIP: 0033:0x7faa36406cce
[ 967.815572] Code: Bad RIP value.
[ 967.815573] RSP: 002b:00007ffedb663f80 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[ 967.815574] RAX: ffffffffffffffda RBX: 0000000000004000 RCX: 00007faa36406cce
[ 967.815575] RDX: 0000000000004000 RSI: 0000564df70257b0 RDI: 00000000ffffff9c
[ 967.815576] RBP: 0000564df70257b0 R08: 0000564df70257b0 R09: 0000000000000000
[ 967.815576] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffedb664240
[ 967.815577] R13: 0000000000000000 R14: 0000000000000001 R15: 0000564df7023e00

[ 967.799227] INFO: task mkfs.btrfs:3767 blocked for more than 120 seconds.
[ 967.803509] Tainted: P O 5.0.0-13-generic #14-Ubuntu
[ 967.807192] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 967.810235] mkfs.btrfs D 0 3767 1866 0x00000000
[ 967.810239] Call Trace:
[ 967.810251] __schedule+0x2d0/0x840
[ 967.810254] ? __switch_to_asm+0x34/0x70
[ 967.810256] ? __switch_to_asm+0x40/0x70
[ 967.810258] schedule+0x2c/0x70
[ 967.810260] schedule_timeout+0x258/0x360
[ 967.810277] wait_for_completion+0xb7/0x140
[ 967.810281] ? wake_up_q+0x80/0x80
[ 967.810293] __floppy_read_block_0+0x138/0x190 [floppy]
[ 967.810298] ? floppy_cmos_show+0x30/0x30 [floppy]
[ 967.810302] floppy_revalidate+0xf8/0x230 [floppy]
[ 967.810307] check_disk_change+0x62/0x70
[ 967.810309] floppy_open+0x2ae/0x380 [floppy]
[ 967.810311] __blkdev_get+0xe5/0x550
[ 967.810313] ? bd_acquire+0xd0/0xd0
[ 967.810315] blkdev_get+0x10c/0x330
[ 967.810316] ? bd_acquire+0xd0/0xd0
[ 967.810318] blkdev_open+0x92/0x100
[ 967.810321] do_dentry_open+0x143/0x3a0
[ 967.810323] vfs_open+0x2d/0x30
[ 967.810326] path_openat+0x2d4/0x16d0
[ 967.810328] ? filename_lookup.part.60+0xe0/0x170
[ 967.810332] ? strncpy_from_user+0x56/0x1b0
[ 967.810333] do_filp_open+0x93/0x100
[ 967.810353] ? strncpy_from_user+0x56/0x1b0
[ 967.810358] ? __alloc_fd+0x46/0x140
[ 967.810360] do_sys_open+0x177/0x280
[ 967.810362] ? _cond_resched+0x19/0x30
[ 967.810364] __x64_sys_openat+0x20/0x30
[ 967.810368] do_syscall_64+0x5a/0x110
[ 967.810369] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 967.810372] RIP: 0033:0x7f46b99374db
[ 967.810377] Code: Bad RIP value.
[ 967.810378] RSP: 002b:00007ffc9e849d70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[ 967.810380] RAX: ffffffffffffffda RBX: 0000556674552080 RCX: 00007f46b99374db
[ 967.810381] RDX: 0000000000080000 RSI: 0000556674552100 RDI: 00000000ffffff9c
[ 967.810382] RBP: 000055667454a270 R08: 0000556674552120 R09: 0000000000004000
[ 967.810382] R10: 0000000000000000 R11: 0000000000000246 R12: 000000005cb8c394
[ 967.810383] R13: 00007f46b99993c4 R14: 00000000dcb8c394 R15: 00007f46b9988425

ProblemType: Bug
DistroRelease: Ubuntu 19.04
Package: linux-image-virtual 5.0.0.13.14
ProcVersionSignature: Ubuntu 5.0.0-13.14-generic 5.0.6
Uname: Linux 5.0.0-13-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 33 Apr 18 18:35 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.10-0ubuntu27
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
Date: Thu Apr 18 18:45:52 2019
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB: 0 bochsdrmfb
ProcKernelCmdLine: root=squash:http://10.245.168.20:34259/root/squashfs ds=nocloud-net;seedfrom=http://10.245.168.20:34259/ console=ttyS0 overlayroot=tmpfs ro systemd.mask=snapd.seeded.service systemd.mask=snapd.service ip=:::::BOOTIF:dhcp BOOTIF=01-52-54-00-12-34-01
RelatedPackageVersions:
 linux-restricted-modules-5.0.0-13-generic N/A
 linux-backports-modules-5.0.0-13-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.12.0-1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-disco
dmi.modalias: dmi:bvnSeaBIOS:bvr1.12.0-1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-disco:cvnQEMU:ct1:cvrpc-i440fx-disco:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-disco
dmi.sys.vendor: QEMU

[Ryan Harper (raharper)]

root@ubuntu:~# lspci -v -nn
00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] [8086:1237] (rev 02)
 Subsystem: Red Hat, Inc. Qemu virtual machine [1af4:1100]
 Flags: fast devsel

00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] [8086:7000]
 Subsystem: Red Hat, Inc. Qemu virtual machine [1af4:1100]
 Flags: medium devsel

00:01.1 IDE interface [0101]: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] [8086:7010] (prog-if 80 [ISA Compatibility mode-only controller, supports bus mastering])
 Subsystem: Red Hat, Inc. Qemu virtual machine [1af4:1100]
 Flags: bus master, medium devsel, latency 0
 [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
 [virtual] Memory at 000003f0 (type 3, non-prefetchable)
 [virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
 [virtual] Memory at 00000370 (type 3, non-prefetchable)
 I/O ports at c080 [size=16]
 Kernel driver in use: ata_piix
 Kernel modules: pata_acpi

00:01.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] (rev 03)
 Subsystem: Red Hat, Inc. Qemu virtual machine [1af4:1100]
 Flags: medium devsel, IRQ 9
 Kernel driver in use: piix4_smbus
 Kernel modules: i2c_piix4

00:02.0 VGA compatible controller [0300]: Device [1234:1111] (rev 02) (prog-if 00 [VGA controller])
 Subsystem: Red Hat, Inc. Device [1af4:1100]
 Flags: fast devsel
 Memory at fd000000 (32-bit, prefetchable) [size=16M]
 Memory at feb90000 (32-bit, non-prefetchable) [size=4K]
 Expansion ROM at 000c0000 [disabled] [size=128K]
 Kernel driver in use: bochs-drm
 Kernel modules: bochs_drm

00:03.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio SCSI [1af4:1004]
 Subsystem: Red Hat, Inc. Virtio SCSI [1af4:0008]
 Physical Slot: 3
 Flags: bus master, fast devsel, latency 0, IRQ 11
 I/O ports at c000 [size=64]
 Memory at feb91000 (32-bit, non-prefetchable) [size=4K]
 Memory at fe000000 (64-bit, prefetchable) [size=16K]
 Capabilities: [98] MSI-X: Enable+ Count=4 Masked-
 Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
 Capabilities: [70] Vendor Specific Information: VirtIO: Notify
 Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
 Capabilities: [50] Vendor Specific Information: VirtIO: ISR
 Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
 Kernel driver in use: virtio-pci

00:04.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG [1af4:1005]
 Subsystem: Red Hat, Inc. Virtio RNG [1af4:0004]
 Physical Slot: 4
 Flags: bus master, fast devsel, latency 0, IRQ 10
 I/O ports at c040 [size=32]
 Memory at fe004000 (64-bit, prefetchable) [size=16K]
 Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
 Capabilities: [70] Vendor Specific Information: VirtIO: Notify
 Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
 Capabilities: [50] Vendor Specific Information: VirtIO: ISR
 Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
 Kernel driver in use: virtio-pci

00:05.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
 Subsystem: Red Hat, Inc. Virtio network device [1af4:0001]
 Physical Slot: 5
 Flags: bus master, fast devsel, la...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Seth Forshee (sforshee)]

Looks like something going wrong with the floppy driver. From the description this sounds like a race. I note that a bug with essentially the same stack trace was fixed in 4.20, de7b75d82f70 "floppy: fix race condition in __floppy_read_block_0()." We have that fix though, so not sure what is causing it here.

[Ryan Harper (raharper)]

Hi Seth,

notice only one of the stack tracks have the floppy, the mdadm one does not. I've also recreated this on a qemu q35 machine type which does not include the floppy device.

[Seth Forshee (sforshee)]

On Mon, Apr 22, 2019 at 08:20:30PM -0000, Ryan Harper wrote:
> Hi Seth,
>
> notice only one of the stack tracks have the floppy, the mdadm one does
> not. I've also recreated this on a qemu q35 machine type which does not
> include the floppy device.

I looked at the madm stack trace, and I was assuming that it occurred
during the same boot as the other one. I suppose I shouldn't have, but
it would make sense. mdadm appears to be blocked on a block device's
bd_mutex, so the cause is probably something else that has hung while
holding that mutex. The mkfs stack trace would be just the kind of thing
to create the mdadm stack trace, if it were trying to open the same
device.

So, were those traces from a single boot or from different boots?

Can you provide stack trace(s) from the q35 machine type for comparison?
Thanks!

[Ryan Harper (raharper)]

Sorry, I missed responding.

This were run in separate VMs, this is under our curtin vmtest integration testing.

Yes, let me get the q35 trace; it doesn't happen as often.

[lilideng (lilideng)]

Same thing happen in HyperV, when run apt-get install -yq linux-azure/disco, it hung in some rate. Like upgrade 3 VMs, 1 VM hung.

VM configuration - 3584 memory, 2 Virtual processors

[ 485.023123] INFO: task mdadm:11824 blocked for more than 120 seconds.
[ 485.029485] Not tainted 5.0.0-15-generic #16-Ubuntu
[ 485.034818] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 485.042754] mdadm D 0 11824 11818 0x00000000
[ 485.042758] Call Trace:
[ 485.042768] __schedule+0x2d0/0x840
[ 485.042770] ? __switch_to_asm+0x34/0x70
[ 485.042771] ? __switch_to_asm+0x40/0x70
[ 485.042774] schedule+0x2c/0x70
[ 485.042776] schedule_timeout+0x258/0x360
[ 485.042780] wait_for_completion+0xb7/0x140
[ 485.042784] ? wake_up_q+0x80/0x80
[ 485.042790] __floppy_read_block_0+0x138/0x190 [floppy]
[ 485.042793] ? floppy_cmos_show+0x30/0x30 [floppy]
[ 485.042797] floppy_revalidate+0xf8/0x230 [floppy]
[ 485.042800] check_disk_change+0x62/0x70
[ 485.042803] floppy_open+0x2ae/0x380 [floppy]
[ 485.042805] __blkdev_get+0xe5/0x550
[ 485.042807] ? bd_acquire+0xd0/0xd0
[ 485.042809] blkdev_get+0x10c/0x330
[ 485.042811] ? bd_acquire+0xd0/0xd0
[ 485.042812] blkdev_open+0x92/0x100
[ 485.042815] do_dentry_open+0x143/0x3a0
[ 485.042818] vfs_open+0x2d/0x30
[ 485.042820] path_openat+0x2d4/0x16d0
[ 485.042825] ? put_device+0x17/0x20
[ 485.042828] ? scsi_device_put+0x2b/0x30
[ 485.042831] do_filp_open+0x93/0x100
[ 485.042835] ? strncpy_from_user+0x56/0x1b0
[ 485.042839] ? __alloc_fd+0xb2/0x140
[ 485.042841] do_sys_open+0x177/0x280
[ 485.042844] __x64_sys_openat+0x20/0x30
[ 485.042847] do_syscall_64+0x5a/0x110
[ 485.042849] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 485.042851] RIP: 0033:0x7f6cda22dcce
[ 485.042856] Code: Bad RIP value.
[ 485.042857] RSP: 002b:00007ffedec2d0e0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[ 485.042859] RAX: ffffffffffffffda RBX: 0000000000004000 RCX: 00007f6cda22dcce
[ 485.042860] RDX: 0000000000004000 RSI: 000055be36cfd870 RDI: 00000000ffffff9c
[ 485.042861] RBP: 000055be36cfd870 R08: 000055be36cfd870 R09: 0000000000000000
[ 485.042862] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffedec2d3a0
[ 485.042863] R13: 0000000000000000 R14: 0000000000000001 R15: 000055be36cfcbb0
[ 495.386112] kauditd_printk_skb: 1 callbacks suppressed
[ 495.386114] audit: type=1400 audit(1565682010.026:62): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.activate" pid=15020 comm="apparmor_parser"
[ 495.386659] audit: type=1400 audit(1565682010.026:63): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.benchmark" pid=15021 comm="apparmor_parser"
[ 495.516297] audit: type=1400 audit(1565682010.158:64): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.buginfo" pid=15022 comm="apparmor_parser"
[ 495.516773] audit: type=1400 audit(1565682010.158:65): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.check-kernel" pid=15023 comm="apparmor_parser"
[ 495.663497] audit: type=1400 audit(1565682010.306:66): apparmor="STATUS" operatio...

[Harry Coin (hcoin)]

I see the same thing related to slowness in a old-style parallel port device passed to a VM. Shortly after there's traffic related to the parallel port the task will hang forever, eventually locking up the VM entirely.
This has happened on two SuperMicro servers, the problem did not exist in Xenial and began with the upgrade to bionic and remains in eoan.
Oct 9 03:43:40 noc2 kernel: [48455.643616] INFO: task CPU 0/KVM:18338 blocked for more than 845 seconds.
Oct 9 03:43:40 noc2 kernel: [48455.643624] Tainted: P O 5.3.0-13-generic #14-Ubuntu
Oct 9 03:43:40 noc2 kernel: [48455.643625] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 9 03:43:40 noc2 kernel: [48455.643627] CPU 0/KVM D 0 18338 1 0x00000100
Oct 9 03:43:40 noc2 kernel: [48455.643630] Call Trace:
Oct 9 03:43:40 noc2 kernel: [48455.643643] __schedule+0x2b9/0x6c0
Oct 9 03:43:40 noc2 kernel: [48455.643655] ? handle_io+0x4c/0xc0 [kvm_intel]
Oct 9 03:43:40 noc2 kernel: [48455.643659] schedule+0x42/0xb0
Oct 9 03:43:40 noc2 kernel: [48455.643661] schedule_preempt_disabled+0xe/0x10
Oct 9 03:43:40 noc2 kernel: [48455.643663] __mutex_lock.isra.0+0x182/0x4f0
Oct 9 03:43:40 noc2 kernel: [48455.643712] ? kvm_arch_vcpu_put+0xe4/0xf0 [kvm]
Oct 9 03:43:40 noc2 kernel: [48455.643731] ? kvm_arch_vcpu_ioctl_run+0x37a/0x590 [kvm]
Oct 9 03:43:40 noc2 kernel: [48455.643733] __mutex_lock_slowpath+0x13/0x20
Oct 9 03:43:40 noc2 kernel: [48455.643735] mutex_lock+0x2e/0x40
Oct 9 03:43:40 noc2 kernel: [48455.643738] pp_ioctl+0x25/0x50 [ppdev]
Oct 9 03:43:40 noc2 kernel: [48455.643741] do_vfs_ioctl+0x407/0x670
Oct 9 03:43:40 noc2 kernel: [48455.643745] ? __secure_computing+0x42/0xe0
Oct 9 03:43:40 noc2 kernel: [48455.643747] ksys_ioctl+0x67/0x90
Oct 9 03:43:40 noc2 kernel: [48455.643748] __x64_sys_ioctl+0x1a/0x20
Oct 9 03:43:40 noc2 kernel: [48455.643752] do_syscall_64+0x5a/0x130
Oct 9 03:43:40 noc2 kernel: [48455.643754] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Oct 9 03:43:40 noc2 kernel: [48455.643756] RIP: 0033:0x7fb9e75b167b
Oct 9 03:43:40 noc2 kernel: [48455.643764] Code: Bad RIP value.
Oct 9 03:43:40 noc2 kernel: [48455.643765] RSP: 002b:00007fb9e4d743a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Oct 9 03:43:40 noc2 kernel: [48455.643767] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fb9e75b167b
Oct 9 03:43:40 noc2 kernel: [48455.643768] RDX: 00007fb9e4d743b7 RSI: 0000000040017086 RDI: 000000000000000a
Oct 9 03:43:40 noc2 kernel: [48455.643769] RBP: 000000000000000a R08: 000055a2756817e8 R09: 000055a275289de0
Oct 9 03:43:40 noc2 kernel: [48455.643770] R10: 0000000000000001 R11: 0000000000000246 R12: 00007fb9e4d743ef
Oct 9 03:43:40 noc2 kernel: [48455.643771] R13: 0000000000000000 R14: 0000000000000001 R15: 000055a276e69010

[eresquigal (eresquigal)]

Same on physical HW (Medion with QuadCore)

This seems to happen during mdadm run, every time a new initrd is being built.

lsblk doesn't hag.
blkid hangs.
mdadm hangs.

In this ps output, mdadm has been hung for almos two hours.

jpc@quad:~$ ps -ft pts/4 ; date
UID PID PPID C STIME TTY TIME CMD
root 7969 28631 0 20:29 pts/4 00:00:00 /bin/sh /var/lib/dpkg/info/linux-image-5.3.0-19-generic.postinst triggered l
root 7970 7969 0 20:29 pts/4 00:00:00 sh /usr/lib/linux/triggers/5.3.0-19-generic
root 7971 7970 0 20:29 pts/4 00:00:00 run-parts --report --exit-on-error --arg=5.3.0-19-generic --arg=/boot/vmlinu
root 8047 7971 0 20:30 pts/4 00:00:00 /bin/sh -e /etc/kernel/postinst.d/initramfs-tools 5.3.0-19-generic /boot/vml
root 8049 8047 0 20:30 pts/4 00:00:00 /bin/sh /usr/sbin/update-initramfs -c -k 5.3.0-19-generic -b /boot
root 8051 8049 0 20:30 pts/4 00:00:00 /bin/sh /usr/sbin/mkinitramfs -o /boot/initrd.img-5.3.0-19-generic.new 5.3.0
root 18519 8051 0 20:30 pts/4 00:00:00 /bin/sh /usr/share/initramfs-tools/hooks/mdadm
root 18642 18519 0 20:30 pts/4 00:00:00 /bin/sh /usr/share/mdadm/mkconf
root 18648 18642 0 20:30 pts/4 00:00:00 /sbin/mdadm --examine --scan --config=partitions
root 28631 17100 0 20:29 pts/4 00:00:00 /usr/bin/dpkg --force-overwrite --status-fd 112 --configure --pending

[1499890.437313] INFO: task mdadm:18648 blocked for more than 120 seconds.
[1499890.437318] Not tainted 5.0.0-31-generic #33-Ubuntu
[1499890.437320] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1499890.437322] mdadm D 0 18648 18642 0x00000080
[1499890.437325] Call Trace:
[1499890.437335] __schedule+0x2d0/0x840
[1499890.437338] ? __switch_to_asm+0x41/0x70
[1499890.437340] ? __switch_to_asm+0x35/0x70
[1499890.437342] schedule+0x2c/0x70
[1499890.437344] schedule_timeout+0x258/0x360
[1499890.437346] wait_for_completion+0xb7/0x140
[1499890.437350] ? wake_up_q+0x80/0x80
[1499890.437358] __floppy_read_block_0+0x138/0x190 [floppy]
[1499890.437362] ? floppy_cmos_show+0x30/0x30 [floppy]
[1499890.437367] floppy_revalidate+0xf8/0x230 [floppy]
[1499890.437370] check_disk_change+0x62/0x70
[1499890.437374] floppy_open+0x2ae/0x380 [floppy]
[1499890.437376] __blkdev_get+0xe5/0x550
[1499890.437378] ? bd_acquire+0xd0/0xd0
[1499890.437380] blkdev_get+0x10c/0x330
[1499890.437382] ? bd_acquire+0xd0/0xd0
[1499890.437384] blkdev_open+0x92/0x100
[1499890.437386] do_dentry_open+0x143/0x3a0
[1499890.437389] vfs_open+0x2d/0x30
[1499890.437391] path_openat+0x2bf/0x1570
[1499890.437394] ? put_device+0x17/0x20
[1499890.437397] ? scsi_device_put+0x2b/0x30
[1499890.437400] do_filp_open+0x93/0x100
[1499890.437403] ? strncpy_from_user+0x57/0x1c0
[1499890.437405] ? __alloc_fd+0x46/0x150
[1499890.437407] do_sys_open+0x177/0x280
[1499890.437409] ? _cond_resched+0x19/0x30
[1499890.437412] __x64_sys_openat+0x20/0x30
[1499890.437415] do_syscall_64+0x5a/0x110
[1499890.437417] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[1499890.437419] RIP: 0033:0x7f84b3cc8fdb
[1499890.437426] Code: Bad RIP value.
[1499890.437427] RSP: 00...

[Matt Zuba (mattzuba)]

I had this problem occur recently and was able to work with one of my System Administrators to narrow it down. I saw the floppy text in the output so I asked my Sys Admin to connect the Floppy Drive in VCenter to the VMs that were having this issue (3 new 18.04 installs) and the issue is now resolved on all of them. blkid, mdadm, nor fdisk hang anymore; they all work as they should. Not sure where the issue exactly lies, but connecting a floppy helped resolve it for now. I have a lot of other 16.04 servers that do not have this issue, for what it's worth.