Fail to boot VM with console=hvc0 in VM kernel cmdline and qemu cmd -serial stdio

[Chen, Fan]

[summary]
Fail to boot VM with console=hvc0 in VM kernel cmdline and qemu cmd -serial stdio

[Reproduce steps]
1. Boot host with Ubuntu23.10 default kernel
root@emr-111:~# uname -a
Linux emr-111 6.5.0-13-generic #13-Ubuntu SMP PREEMPT_DYNAMIC Fri Nov 3 12:16:05 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

2. Install qemu
apt install qemu-system-common=1:8.0.4+dfsg-1ubuntu3
apt install qemu-system-x86=1:8.0.4+dfsg-1ubuntu3

3. Boot VM with Ubuntu23.10 default kernel, change guest kernel cmdline in it.
img=/home/ubuntu-23.10.qcow2
qemu-system-x86_64 \
    -accel kvm -smp 2 -m 4G -cpu host \
    -drive file=${img},if=none,id=virtio-disk0,format=qcow2 \
    -device virtio-blk-pci,drive=virtio-disk0,bootindex=0 \
    -bios /usr/share/qemu/OVMF.fd \
    -device virtio-net-pci,netdev=nic0 -netdev user,id=nic0,hostfwd=tcp::10023-:22 \
    -serial stdio

Login into guest, modify /boot/grub/grub.cfg, find kernel cmdline of it's default kernel:
linux /vmlinuz-6.5.0-9-generic root=UUID=fc270810-beba-4cec-a2a4-7a9a7f104ec7 ro console=tty1 console=ttyS0 ima_hash=sha384 ima_policy=critical_data
Remove "console=tty1 console=ttyS0", add "console=hvc0 earlyprintk=ttyS0,115200", it should be like:
linux /vmlinuz-6.5.0-9-generic root=UUID=fc270810-beba-4cec-a2a4-7a9a7f104ec7 ro console=hvc0 earlyprintk=ttyS0,115200 ima_hash=sha384 ima_policy=critical_data

Poweroff VM

4. Boot VM again with virtio-serial qemu cmd
img=/home/ubuntu-23.10.qcow2
qemu-system-x86_64 \
    -accel kvm -smp 2 -m 4G -cpu host \
    -drive file=${img},if=none,id=virtio-disk0,format=qcow2 \
    -device virtio-blk-pci,drive=virtio-disk0,bootindex=0 \
    -bios /usr/share/qemu/OVMF.fd \
    -device virtio-net-pci,netdev=nic0 -netdev user,id=nic0,hostfwd=tcp::10023-:22 \
    -chardev stdio,id=mux,mux=on,signal=off -device virtio-serial -device virtconsole,chardev=mux \
    -serial chardev:mux \
VM boot up successfully

Poweroff VM

5. Boot VM again with -serial stdio qemu cmd
img=/home/ubuntu-23.10.qcow2
qemu-system-x86_64 \
    -accel kvm -smp 2 -m 4G -cpu host \
    -drive file=${img},if=none,id=virtio-disk0,format=qcow2 \
    -device virtio-blk-pci,drive=virtio-disk0,bootindex=0 \
    -bios /usr/share/qemu/OVMF.fd \
    -device virtio-net-pci,netdev=nic0 -netdev user,id=nic0,hostfwd=tcp::10023-:22 \
    -serial stdio

Fail to boot VM, guest crash with kernel panic
......
[ 2.305104] Btrfs loaded, zoned=yes, fsverity=yes
[ 32.446015] EXT4-fs (vda1): mounted filesystem fc270810-beba-4cec-a2a4-7a9a7f104ec7 ro with ordered data mode. Quota mode: none.
[ 32.484041] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[ 32.484571] CPU: 0 PID: 1 Comm: init Not tainted 6.5.0-9-generic #9-Ubuntu
[ 32.485060] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 2023.05-2+tdx1.0 11/05/2023
[ 32.485749] Call Trace:
[ 32.485948] <TASK>
[ 32.486114] dump_stack_lvl+0x48/0x70
[ 32.486406] dump_stack+0x10/0x20
[ 32.486664] panic+0x1bb/0x3a0
[ 32.486905] do_exit+0x6a1/0x6f0
[ 32.487160] ? __count_memcg_events+0x64/0xf0
[ 32.487506] do_group_exit+0x35/0x90
[ 32.487786] __x64_sys_exit_group+0x18/0x20
[ 32.488119] do_syscall_64+0x59/0x90
[ 32.488393] ? do_user_addr_fault+0x17a/0x6b0
[ 32.488732] ? exit_to_user_mode_prepare+0x30/0xb0
[ 32.489105] ? irqentry_exit_to_user_mode+0x17/0x20
[ 32.489488] ? irqentry_exit+0x43/0x50
[ 32.489776] ? exc_page_fault+0x94/0x1b0
[ 32.490081] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[ 32.490472] RIP: 0033:0x7f583c09b2ed
[ 32.490747] Code: 02 ba ff ff ff ff 89 d0 c3 0f 1f 40 00 f3 0f 1e fa 48 8b 35 1d 3b 11 00 ba e7 00 00 00 eb 07 66 0f 1f 44 00 00 f4 89 d0 0f 05 <48> 3d 00 f0 ff ff 76 f3 f7 d8 64 89 06 eb ec 0f 1f 40 00 f3 0f 1e
[ 32.492180] RSP: 002b:00007ffe86621b88 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7
[ 32.492761] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f583c09b2ed
[ 32.493306] RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000001
[ 32.493849] RBP: 00007ffe86621f01 R08: 000056221af7e010 R09: 0000000000000007
[ 32.494397] R10: 000056221af7ffd0 R11: 0000000000000202 R12: 0000000000000000
[ 32.494940] R13: 00007ffe86621f40 R14: 0000562219b31598 R15: 00007f583c1f8000
[ 32.495479] </TASK>
[ 32.496940] Kernel Offset: 0x30200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 32.497737] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 ]---

[Expected Result]
VM boot up successfully with qemu cmd -serial stdio, no output from stdio after earlyprint

[Actual Result]
guest crash with kernel panic

<Build: version, kernel>
Host/Guest OS: Ubuntu 23.10
Host kernel: 6.5.0-13-generic
Qemu:1:8.0.4+dfsg-1ubuntu3
Guest kernel:6.5.0-9-generic

<Environment: HW version, SKU#>
CPU: EMR
CPU(s): 224
Vendor ID: GenuineIntel
    CPU family: 6
    Model: 207
    Stepping: 1

[Chen, Fan (fanchen1)]

With the same host kernel and qemu, this issue cannot be reproduced with Centos guest image, it can get expected result: VM boot up successfully
I built the latest linux.git kernel and 6.5.0 release kernel myself on guest, and boot VM with the same config, the issue can be reproduced.
When reproduce, make sure no other "console=" para in guest kernel cmdline, in case "console=hvc0" be overwritten.

[Chen, Fan (fanchen1)]

Can be reproduced with 6.5.0-14-generic

[Philip Cox (philcox)]

After reproducing this on the cloud image with the 6.5.0-9-generic kernel, I have tried and it is no longer reproducible on the newest cloud image. I tested with the cloud image with kernel version 6.5.0-14-generic. Please update and re-test. If it is still reproducible, please re-open the ticket.

[Chen, Fan (fanchen1)]

I can reproduce this issue with the latest image published in 2023-12-20 https://cloud-images.ubuntu.com/releases/mantic/release/ubuntu-23.10-server-cloudimg-arm64.img, re-open it.
Did you change any other setup in host?

[Philip Cox (philcox)]

I can now reproduce it on the latest image, thanks.

I see a few upstream fixes for the hvc driver console that seem that they may fix it. I am checking if they do fix it or not.

[Chen, Fan (fanchen1)]

Can be reproduced with Ubuntu24.04

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.