kdump fails on focal + linux-image-generic-hwe-20.04 kernel

[Ryan Harper]

1)
$ lsb_release -rd
Description: Ubuntu 20.04.5 LTS
Release: 20.04
2) ubuntu@ubuntu:~$ apt-cache policy makedumpfile
makedumpfile:
  Installed: 1:1.6.7-1ubuntu2.4
  Candidate: 1:1.6.7-1ubuntu2.4
  Version table:
 *** 1:1.6.7-1ubuntu2.4 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1:1.6.7-1ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
ubuntu@ubuntu:~$ apt-cache policy kexec-tools
kexec-tools:
  Installed: 1:2.0.18-1ubuntu1
  Candidate: 1:2.0.18-1ubuntu1
  Version table:
 *** 1:2.0.18-1ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status
$ apt-cache policy linux-image-generic-hwe-20.04
linux-image-generic-hwe-20.04:
  Installed: 5.15.0.56.62~20.04.22
  Candidate: 5.15.0.56.62~20.04.22
  Version table:
 *** 5.15.0.56.62~20.04.22 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
        100 /var/lib/dpkg/status
     5.4.0.26.32 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

3) crash dump of linux-image-generic-hwe-20.04 kernel to complete successfully
4) the kexec'ed kernel panics and fails to capture the kernel crash dump

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: kdump-tools 1:1.6.7-1ubuntu2.4
ProcVersionSignature: User Name 5.15.0-56.62~20.04.1-generic 5.15.64
Uname: Linux 5.15.0-56-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.25
Architecture: amd64
CasperMD5CheckResult: skip
Date: Wed Dec 21 15:02:20 2022
SourcePackage: makedumpfile
UpgradeStatus: No upgrade log present (probably fresh install)

[Ryan Harper (raharper)]

Serial console output of VM kexec panic after triggering crashdump on 5.15 hwe kernel.

[Ryan Harper (raharper)]

Testing with jammy-server image (which uses 5.15 kernel) it crashes fine.

The general steps are:

1) boot VM with server image
2) apt install kdump-tools (enable kexec, enable kdump)
3) reboot, check crashdump param is set
4) kdump-config show (says ok to dump)
5) echo c | sudo tee /proc/sysrq-trigger

For Focal, upgrade to linux-image-generic-hwe-20.04 kernel and repeat steps 4 and 5.

[dann frazier (dannf)]

From the serial log, it looks to me like you don't have enough memory reserved. Try bumping up crashkernel=? You're not getting to the point of running makedumpfile so I don't see how it would be impacted.

Its hard to say why jammy works for you and focal does not. I don't think we have a larger crashkernel= value by default. It is likely something more subtle like the initramfs in jammy consuming less memory.

Note that once the memory issue is resolved, I suspect you'll hit bug 1970672.

[Ryan Harper (raharper)]

Hi Dann,

Thanks for the comments.

VM has 4G, bumping the craskkernel size 256 still fails, but moving up to 512M allows this to work, but the equivalent kernel in jammy works with 192M...

Thoughts?

And you're quite right, once the dump works, I see bug 1970672

[ 3.684815] kdump-tools[487]: Starting kdump-tools:
[ 3.685258] kdump-tools[493]: * running makedumpfile -c -d 31 /proc/vmcore /var/crash/202212211756/dump-incomplete
Copying data : [100.0 %] \ eta: 0s
[ 5.850607] kdump-tools[510]: The kernel version is not supported.
[ 5.851463] kdump-tools[510]: The makedumpfile operation may be incomplete.
[ 5.852609] kdump-tools[510]: The dumpfile is saved to /var/crash/202212211756/dump-incomplete.
[ 5.853977] kdump-tools[510]: makedumpfile Completed.
[ 5.857098] kdump-tools[493]: * kdump-tools: saved vmcore in /var/crash/202212211756
[ 5.887243] kdump-tools[493]: * running makedumpfile --dump-dmesg /proc/vmcore /var/crash/202212211756/dmesg.202212211756
[ 5.888335] kdump-tools[516]: The kernel version is not supported.
[ 5.888940] kdump-tools[516]: The makedumpfile operation may be incomplete.
[ 5.889589] kdump-tools[516]: The dmesg log is saved to /var/crash/202212211756/dmesg.202212211756.
[ 5.890346] kdump-tools[516]: makedumpfile Completed.
[ 5.890998] kdump-tools[493]: * kdump-tools: saved dmesg content in /var/crash/202212211756
[ 5.894985] kdump-tools[519]: Wed, 21 Dec 2022 17:56:45 +0000
[ 5.898013] kdump-tools[520]: Rebooting.
[ 5.938356] reboot: Restarting system