ubuntu_aufs_smoke_test hangs on E ARM64 node with soft lockup

[Po-Hsu Lin]

Issue found on ARM64 node appleton-kernel.

Kernel - 5.3.0-41

Test output:

16:06:06 DEBUG| [stdout] aufs: rename file2 to dir1: PASSED
16:06:06 DEBUG| [stdout] aufs: move file3 to root and then to dir1: PASSED
16:06:06 DEBUG| [stdout] aufs: truncate: PASSED
16:06:06 DEBUG| [stdout] aufs: stacked: PASSED
16:06:06 DEBUG| [stdout] aufs: modify: PASSED
16:06:06 DEBUG| [stdout] aufs: check hardlink count in root: PASSED
16:06:06 DEBUG| [stdout] aufs: check hardlink count in root after 1 link removed: PASSED
16:06:06 DEBUG| [stdout] aufs: check hardlink count in root after 1 more link removed: PASSED
16:06:06 DEBUG| [stdout] aufs: attempt to create invalid cross link on root: PASSED
16:06:06 DEBUG| [stdout] aufs: remove root view of original file: PASSED
16:06:06 DEBUG| [stdout] aufs: simple stat check: PASSED
16:06:06 DEBUG| [stdout] aufs: simple stat check (touch file): PASSED
16:06:06 DEBUG| [stdout] aufs: simple stat check (chmod u-x): PASSED
16:06:06 DEBUG| [stdout] aufs: simple stat check (chmod u+x): PASSED
16:06:06 DEBUG| [stdout] aufs: simple stat check (chmod g-w): PASSED
16:06:06 DEBUG| [stdout] aufs: simple stat check (chmod g+w): PASSED
16:06:06 DEBUG| [stdout] aufs: simple stat check (chmod o-r): PASSED
16:06:06 DEBUG| [stdout] aufs: simple stat check (chmod o+r): PASSED
16:06:06 DEBUG| [stdout] aufs: simple stat check (chmod 0777): PASSED
16:06:06 DEBUG| [stdout] aufs: simple stat check (access time): PASSED
16:06:06 DEBUG| [stdout] aufs: simple stat check (modify time): PASSED
16:06:06 DEBUG| [stdout] aufs: simple stat check (change size): PASSED
00:00:53 DEBUG| [stdout] aufs: umount: PASSED
(hung here)

syslog output
 snapd[1707]: daemon stop requested to wait for socket activation
 kernel: [ 180.229861] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [swapper/16:0]
 kernel: [ 180.236985] Modules linked in: ipmi_ssif nls_iso8859_1 cp210x joydev input_leds usbserial ipmi_si ipmi_devintf ipmi_msghandler sch_fq_codel ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear mlx5_ib ses enclosure hid_generic usbhid ib_uverbs hid ib_core aes_ce_blk marvell aes_ce_cipher hibmc_drm drm_vram_helper ttm drm_kms_helper crct10dif_ce syscopyarea ghash_ce sysfillrect sha2_ce sysimgblt mlx5_core ixgbe sha256_arm64 fb_sys_fops hisi_sas_v2_hw nvme tls hisi_sas_main sha1_ce xfrm_algo drm mdio megaraid_sas libsas mlxfw nvme_core ehci_platform hns_dsaf scsi_transport_sas hns_enet_drv hns_mdio hnae aes_neon_bs aes_neon_blk crypto_simd cryptd aes_arm64
 kernel: [ 180.237045] CPU: 16 PID: 0 Comm: swapper/16 Not tainted 5.3.0-41-generic #33~18.04.1-Ubuntu
 kernel: [ 180.237046] Hardware name: Hisilicon D05/BC11SPCD, BIOS 1.50 06/01/2018
 kernel: [ 180.237048] pstate: 40400005 (nZcv daif +PAN -UAO)
 kernel: [ 180.237055] pc : __do_softirq+0x98/0x328
 kernel: [ 180.237059] lr : irq_exit+0xc0/0xc8
 kernel: [ 180.237060] sp : ffff000011d0bef0
 kernel: [ 180.237061] x29: ffff000011d0bef0 x28: ffff802fb71a0ec0
 kernel: [ 180.237063] x27: 0000000000000000 x26: ffff000011d0c000
 kernel: [ 180.237065] x25: ffff000011d08000 x24: ffff801fba132400
 kernel: [ 180.237066] x23: ffff00001212bde0 x22: 0000000000000000
 kernel: [ 180.237068] x21: 0000000000000282 x20: 0000000000000002
 kernel: [ 180.237069] x19: ffff000011509000 x18: ffff0000111090f8
 kernel: [ 180.237071] x17: 0000000000000000 x16: 0000000000000000
 kernel: [ 180.237072] x15: 0000000000000001 x14: ffff802fbba021c8
 kernel: [ 180.237074] x13: 0000000000000004 x12: 0000000000000000
 kernel: [ 180.237075] x11: 0000000000000000 x10: 0000000000000040
 kernel: [ 180.237077] x9 : ffff0000119efa18 x8 : ffff0000119efa10
 kernel: [ 180.237079] x7 : ffff801fb9002270 x6 : 000000040a20d777
 kernel: [ 180.237081] x5 : 00000000ffff7079 x4 : 0000802faa50c000
 kernel: [ 180.237083] x3 : ffff000011509580 x2 : 0000802faa50c000
 kernel: [ 180.237085] x1 : 00000000000000e0 x0 : ffff000011509580
 kernel: [ 180.237087] Call trace:
 kernel: [ 180.237090] __do_softirq+0x98/0x328
 kernel: [ 180.237092] irq_exit+0xc0/0xc8
 kernel: [ 180.237094] __handle_domain_irq+0x6c/0xc0
 kernel: [ 180.237096] gic_handle_irq+0xec/0x2d0
 kernel: [ 180.237098] el1_irq+0x104/0x1c0
 kernel: [ 180.237101] rcu_idle_exit+0x4c/0x88
 kernel: [ 180.237104] do_idle+0x15c/0x2b8
 kernel: [ 180.237106] cpu_startup_entry+0x28/0x30
 kernel: [ 180.237109] secondary_start_kernel+0x16c/0x238
 systemd[1]: Created slice User Slice of ubuntu.

[Kleber Sacilotto de Souza (kleber-souza)]

This seems to be the same issue as bug 1953058. I'll duplicate it against that one as Eoan is not supported anymore and 1953058 has more additional info.