jammy beta (220330) live server arm64 iso failed to boot Hisilicon D06 with virtual CD

[Taihsiang Ho]

= Description =
Platforms: Hisilicon D06 (d06-1 and d06-3 in our lab, which are NOT production product)
Image: Jammy Beta (220330)

= Steps to Reproduce =
1. Boot the system with the iso from d06's virtual CD of BMC
2. Launch the kernel via the default grub entry

= Expected Results =
The kernel and image are loaded via the virtual CD, and then subiquity is launched.

= Actual Results =
After the grub stage, when loading the kernel we see (check d06-3.log for the full log details):

[ 1149.949752] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: {^M
[ 1149.949796] rcu: blocking rcu_node structures (internal RCU debug):^M
[ 1180.107036] watchdog: BUG: soft lockup - CPU#6 stuck for 26s! [swapper/6:0]^M
[ 1204.107091] watchdog: BUG: soft lockup - CPU#6 stuck for 48s! [swapper/6:0]^M
[ 1213.237720] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:^M
[ 1213.243639] rcu: 15-....: (2 GPs behind) idle=87d/1/0x4000000000000004 softirq=854/855 fqs=7501 ^M
[ 1213.441723] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: {^M
[ 1213.441764] rcu: blocking rcu_node structures (internal RCU debug):^M
[ 1228.108161] watchdog: BUG: soft lockup - CPU#6 stuck for 70s! [swapper/6:0]^M
[ 1252.106613] watchdog: BUG: soft lockup - CPU#6 stuck for 93s! [swapper/6:0]^M
[ 1276.106317] watchdog: BUG: soft lockup - CPU#6 stuck for 115s! [swapper/6:0]^M
[ 1300.106639] watchdog: BUG: soft lockup - CPU#6 stuck for 137s! [swapper/6:0]^M
[ 1324.114455] watchdog: BUG: soft lockup - CPU#6 stuck for 160s! [swapper/6:0]^M
[ 1330.178015] INFO: task (imesyncd):2464 blocked for more than 120 seconds.^M
[ 1330.184819] Tainted: G L 5.15.0-23-generic #23-Ubuntu^M
[ 1330.184836] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.^M
[ 1330.185017] INFO: task unshare:2502 blocked for more than 120 seconds.^M
[ 1330.185027] Tainted: G L 5.15.0-23-generic #23-Ubuntu^M
[ 1330.185037] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.^M
[ 1330.185325] INFO: task unshare:2507 blocked for more than 120 seconds.^M
** 1238 printk messages dropped **^M
[ 1333.632679] systemd[1]: Failed to start Journal Service.^M

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1967957

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Taihsiang Ho (tai271828)]

Similar log from d06 (d06-3), but enabling kernel parameter `debug` (and remove all `quiet`) at the grub stage.

[Taihsiang Ho (tai271828)]

Redo with d06 (d06-1). This time we have the log of kernel trace. See d06-1-cd-220406-01.log for the full console log.

[ 652.568681] pstate: 004000c9 (nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 652.575630] pc : irq_work_queue_on+0xa4/0xb0
[ 652.579888] lr : rcu_implicit_dynticks_qs+0x1b8/0x350
[ 652.584928] sp : ffff80000b223c50
[ 652.588229] x29: ffff80000b223c50 x28: ffff80000aa30e00 x27: 0000000000000000
[ 652.595354] x26: ffff003f7fb44025 x25: 0000000000000019 x24: ffff80000aa30e00
[ 652.602478] x23: ffff803f759d9000 x22: ffff80000a906000 x21: ffff80000a16b025
[ 652.609602] x20: 0000000000000000 x19: ffff003f7fb43f00 x18: 0000000000000102
[ 652.616727] x17: 3330643830383030 x16: 3030386666666620 x15: 3a20307820303030
[ 652.623850] x14: ffff80000a94b760 x13: ffff80000a94b1e8 x12: 0000000000000002
[ 652.630974] x11: 0000000000000000 x10: 0000000000000b50 x9 : ffff8000082158e4
[ 652.638097] x8 : ffff002084b42b30 x7 : 0000000000000000 x6 : ffff003f7fb44058
[ 652.645221] x5 : 0000000000000000 x4 : 0000000000000000 x3 : ffff003f7fb43140
[ 652.652344] x2 : 0000000000000000 x1 : ffff80000a90da28 x0 : 0000000000000004
[ 652.659469] Call trace:
[ 652.661903] irq_work_queue_on+0xa4/0xb0
[ 652.665813] rcu_implicit_dynticks_qs+0x1b8/0x350
[ 652.670504] force_qs_rnp+0x184/0x340
[ 652.674154] rcu_gp_fqs_loop+0x224/0x394
[ 652.678064] rcu_gp_kthread+0x120/0x164
[ 652.681888] kthread+0x114/0x120
[ 652.685105] ret_from_fork+0x10/0x20
[ 652.688669] ---[ end trace da56ec070b45e740 ]---

[Taihsiang Ho (tai271828)]

Next from my side: there are new daily build. I may try them as well.

[Taihsiang Ho (tai271828)]

Additional info: in our lab, a taishan2280v2 could install the 220330 iso. (tested by ike)

[Taihsiang Ho (tai271828)]

daily iso (220405) tested on d06 (d06-1)

= log snippet =
Loading the CD via virtual media and drop to the prompt asking "Attempt interactive netboot from a URL?"

NOTICE: ERR_ADDRL = 0x0
NOTICE: ERR_ADDRH = 0x0
NOTICE: ERR_MISC0L = 0x0
NOTICE: ERR_MISC0H = 0x0
NOTICE: ERR_MISC1L = 0x0
NOTICE: ERR_MISC1H = 0x2ea6f418
NOTICE: [TotemRasIntOemNodeEri]:[1750L]
NOTICE: [RasEriInterrupt]:[117L]NodeTYP2Status = 0x0
ERROR: [FiqInterruptHandler]:[114L] GicInterrupt1 : 0x1cd!!!
[ 2.400045] cma: cma_alloc: reserved: alloc failed, req-size: 4 pages, ret: -12
[ 2.418203] ACPI: IORT: [Firmware Bug]: [map (____ptrval____)] conflicting mapping for input ID 0x7c00
[ 2.427513] ACPI: IORT: [Firmware Bug]: applying workaround.
[ 2.433174] cma: cma_alloc: reserved: alloc failed, req-size: 4 pages, ret: -12
[ 2.441255] cma: cma_alloc: reserved: alloc failed, req-size: 4 pages, ret: -12
[ 2.457200] ACPI: IORT: [Firmware Bug]: [map (____ptrval____)] conflicting mapping for input ID 0xbc00
[ 2.466504] ACPI: IORT: [Firmware Bug]: applying workaround.
[ 5.854360] hisi_sas_v3_hw 0000:74:02.0: 16 hw queues
[ 5.863968] cma: cma_alloc: reserved: alloc failed, req-size: 4 pages, ret: -12
[ 19.993752] cma: cma_alloc: reserved: alloc failed, req-size: 4 pages, ret: -12
mount: mounting /dev/sr0 on /cdrom failed: No such device or address
Unable to find a medium containing a live file system
Attempt interactive netboot from a URL?
yes no (default yes):
Two methods available for IP configuration:
  * static: for static IP configuration
  * dhcp: for automatic IP configuration
static dhcp (default 'dhcp'):

[Taihsiang Ho (tai271828)]

A production taishan2280v2 works with 220405 daily build[1].

We may not need to worry that much with the failure of d06-1 and d06-3 now.

[1]
ubuntu@scobee-jammy-daily-220405:~$ cat /var/log/installer/media-info ; echo ; uname -a; lsb_release -a; ping -c 4 8.8.8.8; sudo apt update
Ubuntu-Server 22.04 LTS "Jammy Jellyfish" - Daily arm64 (20220405)
Linux scobee-jammy-daily-220405 5.15.0-25-generic #25-Ubuntu SMP Wed Mar 30 15:57:31 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Jammy Jellyfish (development branch)
Release: 22.04
Codename: jammy
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3059ms

[sudo] password for ubuntu:
Hit:1 http://ports.ubuntu.com/ubuntu-ports jammy InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease
Hit:3 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease
Hit:4 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.