Improve sysctl handling on kdump-tools

[Dan Streetman]

[impact]

Documentation, and past behavior, for kdump-tools was that the KDUMP_SYSCTL variable in the /etc/default/kdump-tools file would be applied to the system kernel params at kdump 'load'. However this is no longer true, and those params are no longer applied to the system's kernel param settings.

[test case]

install linux-crashdump (and kdump-tools).

Edit the /etc/default/kdump-tools file to set the KDUMP_SYSCTL param to something other than default, e.g.:

KDUMP_SYSCTL="kernel.panic_on_oops=1 kernel.panic_on_warn=1"

reboot, or unload/reload kdump, to pick up the changes to the file.

Check if the panic_on_warn param is set:

$ cat /proc/sys/kernel/panic_on_warn
0

the problem does not seem to be with sysctl, as manually calling it does work:

$ KDUMP_SYSCTL="kernel.panic_on_oops=1 kernel.panic_on_warn=1"
$ cat /proc/sys/kernel/panic_on_warn
0
$ sudo sysctl -w $KDUMP_SYSCTL
kernel.panic_on_oops = 1
kernel.panic_on_warn = 1
$ cat /proc/sys/kernel/panic_on_warn
1

[regression potential]

TBD

[Dan Streetman (ddstreet)]

Also to clarify, manually unloading/reloading kdump doesn't set the params properly:

ubuntu@makedumpfile-e:~$ grep . /proc/sys/kernel/panic_on_warn
0
ubuntu@makedumpfile-e:~$ sudo kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1 kernel.panic_on_warn=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0xaa000000
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.3.0-10-generic
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.3.0-10-generic
current state: ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.3.0-10-generic root=PARTUUID=7975ae09-1bd6-4c5f-b460-5f5218cc2327 ro console=tty1 console=ttyS0 systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
ubuntu@makedumpfile-e:~$ sudo kdump-config unload
 * unloaded kdump kernel
ubuntu@makedumpfile-e:~$ sudo kdump-config load
 * Creating symlink /var/lib/kdump/vmlinuz
 * Creating symlink /var/lib/kdump/initrd.img
 * loaded kdump kernel
ubuntu@makedumpfile-e:~$ grep . /proc/sys/kernel/panic_on_warn
0

[Thadeu Lima de Souza Cascardo (cascardo)]

This was somehow removed between kdump-tools 1.5.5 and 1.5.9. I'll see if I can pinpoint the exact version in history and try to find some justification. But this is an old regression.

[Thadeu Lima de Souza Cascardo (cascardo)]

commit 9a77ef188b24742b595f69891b97d48dc997f5e2
Author: Louis Bouchard <email address hidden>
Date: Thu Jan 29 16:03:04 2015 +0100

    [debian] Fix panic_on_oops faulty handling

    Signed-off-by: Louis Bouchard <email address hidden>

diff --git a/debian/kdump-config b/debian/kdump-config
index 7ea26cab1c95..64489a74757f 100755
--- a/debian/kdump-config
+++ b/debian/kdump-config
@@ -310,9 +310,10 @@ function fadump_register()
        logger -t $NAME "fadump registered successfully"

        # Last step: make sure panic_on_oops is enabled
- if [ -x /sbin/sysctl -a "$KDUMP_SYSCTL" != " " ] ; then
- sysctl -w $KDUMP_SYSCTL >/dev/null
- fi
+ PANIC_ON_OOPS=`/sbin/sysctl kernel.panic_on_oops | cut -d" " -f3`
+ if [ $PANIC_ON_OOPS != "1" ] ; then
+ sysctl -w kernel.panic_on_oops=1 >/dev/null
+ fi
 }

 # Returns: none. prints warnings or exit
@@ -393,9 +394,11 @@ function kdump_load()
        fi

        # Last step: make sure panic_on_oops is enabled
- if [ -x /sbin/sysctl -a "$KDUMP_SYSCTL" != " " ] ; then
- sysctl -w $KDUMP_SYSCTL >/dev/null
- fi
+ PANIC_ON_OOPS=`/sbin/sysctl kernel.panic_on_oops | cut -d" " -f3`
+ if [ $PANIC_ON_OOPS != "1" ] ; then
+ sysctl -w kernel.panic_on_oops=1 >/dev/null
+ fi
+
 }

[Guilherme G. Piccoli (gpiccoli)]

Thanks Dan for the report and Cascardo for finding the offender commit that introduced this regression!

I think we have room for improvements here, my considerations are:
a) KDUMP_SYSCTL is a misleading name; it seems to be related with sysctls set _when_ dumping, which is also an important thing to control. Despite this, the option meaning was related to the kdump sysctl _triggers_.

b) Even by re-enabling this option we don't have a fine tuning configuration of sysctls during the dump.

So, my idea to fix/improve the situation is: let's have a KDUMP_PANIC_TRIGGERS variable (that defaults to panic_on_oops) and a KDUMP_SYSCTL_ON_DUMP variable that overrides the desired sysctls set on normal kernel when the kdump kernel boots (and which should defaults to have hugepages disabled, since it's a known problem when kdumping to have hugepages "consuming" the system memory).

Cheers,

Guilherme

[Guilherme G. Piccoli (gpiccoli)]

I've reported a Debian bug with the proposed fixes; the merge report has the information about the approach used to deal with sysctl in kdump: https://salsa.debian.org/debian/makedumpfile/-/merge_requests/2

Cheers,

Guilherme

[Brian Murray (brian-murray)]

The Eoan Ermine has reached end of life, so this bug will not be fixed for that release

[dann frazier (dannf)]

Guillherme's MR was merged in 1.6.8-1, so marking Fix Released.

[dann frazier (dannf)]

Correction: looks like it was 1:1.6.7-4. git-debrebase confused me.