I installed the current 4.15.0-162-generic kernel from -updates to a host in segmaas, and brought up my Bionic VM I installed for the reproducer. The Bionic VM has linux-crashdump configured, with nr_cpus configured to 8:
ubuntu@bionic-vm:~$ kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-4.15.0-159-generic
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.15.0-159-generic
current state: ready to kdump
$ virsh list
Id Name State
----------------------------------------------------
1 bionic-vm paused
Okay, we can reproduce the issue.
I then enabled -proposed, and installed the 4.15.0-163-generic kernel:
$ uname -rv
4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021
I started the same VM, verified it has a crashkernel with nr_cpus=8:
$ kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-4.15.0-159-generic
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.15.0-159-generic
current state: ready to kdump
$ virsh list
Id Name State
----------------------------------------------------
1 bionic-vm running
Checking the VM, the crashdump completed successfully and restarted correctly.
$ virsh console bionic-vm
...
Copying data : [100.0 %] \ eta: 0s
[ 12.034361] kdump-tools[874]: The dumpfile is saved to /var/crash/202111110246/dump-incomplete.
[ 12.036787] kdump-tools[874]: makedumpfile Completed.
[ 12.038752] kdump-tools[874]: * kdump-tools: saved vmcore in /var/crash/202111110246
[ 12.058489] kdump-tools[874]: * running makedumpfile --dump-dmesg /proc/vmcore /var/crash/202111110246/dmesg.202111110246
[ 12.069700] kdump-tools[874]: The dmesg log is saved to /var/crash/202111110246/dmesg.202111110246.
[ 12.072574] kdump-tools[874]: makedumpfile Completed.
[ 12.074245] kdump-tools[874]: * kdump-tools: saved dmesg content in /var/crash/202111110246
[ 12.159319] kdump-tools[874]: Thu, 11 Nov 2021 02:46:54 +0000
...
I then did this a further 4 times, each time the crashdump completed succesfully.
I then destroyed and started the VM fresh, and kept crashdumping. It worked every time.
The 4.15.0-163-generic kernel in -proposed fixes the issue. Happy to mark as verified.
Performing verification for Bionic.
I installed the current 4.15.0-162-generic kernel from -updates to a host in segmaas, and brought up my Bionic VM I installed for the reproducer. The Bionic VM has linux-crashdump configured, with nr_cpus configured to 8:
ubuntu@bionic-vm:~$ kdump-config show panic_on_ oops=1 lib/kdump/ vmlinuz: symbolic link to /boot/vmlinuz- 4.15.0- 159-generic lib/kdump/ initrd. img: symbolic link to /var/lib/ kdump/initrd. img-4.15. 0-159-generic
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x
/var/
kdump initrd:
/var/
current state: ready to kdump
kexec command: line="BOOT_ IMAGE=/ boot/vmlinuz- 4.15.0- 159-generic root=UUID= 5c23b432- 4d00-4a01- 8ecd-964803c8e1 0a ro console=tty1 console=ttyS0 reset_devices systemd. unit=kdump- tools-dump. service nr_cpus=8 irqpoll nousb ata_piix. prefer_ ms_hyperv= 0 ignore_loglevel nmi_debug= state,regs apic_extnmi=none" --initrd= /var/lib/ kdump/initrd. img /var/lib/ kdump/vmlinuz
/sbin/kexec -p --command-
I then triggered the crashdump in the VM:
$ sudo -s
# echo c > /proc/sysrq-trigger
On the host, we look at the tail of:
$ sudo tail -f /var/log/ libvirt/ qemu/bionic- vm.log 00000 DR1=00000000000 00000 DR2=00000000000 00000 DR3=00000000000 00000 f0ff0 DR7=00000000000 00400 000000
...
KVM internal error. Suberror: 1
emulation failure
EAX=0000de8f EBX=00000000 ECX=0000008f EDX=00000600
ESI=00000000 EDI=00000000 EBP=00000000 ESP=0000f90c
EIP=0000cdb1 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 000f0000 0000ffff 00009b00
SS =de00 000de000 0000ffff 00009300
DS =de00 000de000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 00000000 0000ffff
IDT= 00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=32b2a000 CR4=00000000
DR0=00000000000
DR6=00000000fff
EFER=0000000000
Code=66 83 c4 28 66 5b 66 c3 66 56 66 53 66 52 b1 8f 88 c8 e6 70 <e4> 71 66 0f b6 f0 66 89 f2 67 88 54 24 03 88 c8 e6 70 66 31 db 88 d8 e6 71 66 56 66 68 1a
$ virsh list ------- ------- ------- ------- ------- ------- ---
Id Name State
-------
1 bionic-vm paused
Okay, we can reproduce the issue.
I then enabled -proposed, and installed the 4.15.0-163-generic kernel:
$ uname -rv
4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021
I started the same VM, verified it has a crashkernel with nr_cpus=8:
$ kdump-config show panic_on_ oops=1 lib/kdump/ vmlinuz: symbolic link to /boot/vmlinuz- 4.15.0- 159-generic lib/kdump/ initrd. img: symbolic link to /var/lib/ kdump/initrd. img-4.15. 0-159-generic
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x
/var/
kdump initrd:
/var/
current state: ready to kdump
kexec command: line="BOOT_ IMAGE=/ boot/vmlinuz- 4.15.0- 159-generic root=UUID= 5c23b432- 4d00-4a01- 8ecd-964803c8e1 0a ro console=tty1 console=ttyS0 reset_devices systemd. unit=kdump- tools-dump. service nr_cpus=8 irqpoll nousb ata_piix. prefer_ ms_hyperv= 0 ignore_loglevel nmi_debug= state,regs apic_extnmi=none" --initrd= /var/lib/ kdump/initrd. img /var/lib/ kdump/vmlinuz
/sbin/kexec -p --command-
I then triggered the crashdump:
$ sudo -s
# echo c > /proc/sysrq-trigger
On the host:
$ sudo tail -f /var/log/ libvirt/ qemu/bionic- vm.log 11T02:41: 45.970264Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H :ECX.svm [bit 2] 11T02:41: 45.971185Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H :ECX.svm [bit 2]
...
2021-11-
2021-11-
$ virsh list ------- ------- ------- ------- ------- ------- ---
Id Name State
-------
1 bionic-vm running
Checking the VM, the crashdump completed successfully and restarted correctly.
$ virsh console bionic-vm 202111110246/ dump-incomplete . 202111110246 202111110246/ dmesg.202111110 246 202111110246/ dmesg.202111110 246. 202111110246
...
Copying data : [100.0 %] \ eta: 0s
[ 12.034361] kdump-tools[874]: The dumpfile is saved to /var/crash/
[ 12.036787] kdump-tools[874]: makedumpfile Completed.
[ 12.038752] kdump-tools[874]: * kdump-tools: saved vmcore in /var/crash/
[ 12.058489] kdump-tools[874]: * running makedumpfile --dump-dmesg /proc/vmcore /var/crash/
[ 12.069700] kdump-tools[874]: The dmesg log is saved to /var/crash/
[ 12.072574] kdump-tools[874]: makedumpfile Completed.
[ 12.074245] kdump-tools[874]: * kdump-tools: saved dmesg content in /var/crash/
[ 12.159319] kdump-tools[874]: Thu, 11 Nov 2021 02:46:54 +0000
...
I then did this a further 4 times, each time the crashdump completed succesfully.
I then destroyed and started the VM fresh, and kept crashdumping. It worked every time.
The 4.15.0-163-generic kernel in -proposed fixes the issue. Happy to mark as verified.