Setting a kernel commandline parameter can crash snapd

[Tom Cook]

I tried using the new snap way of setting a kernel commandline parameter:

sudo snap set system system.kernel.cmdline-append="initcall_blacklist=simpledrm_platform_driver_init"

The snapd daemon now segfaults every time it tries to start. It doesn't leave a crash file in /var/crash, but it leaves the log below in syslog.

I guess that the parameter I've set fails the allowlist processing on `cmdline-append' but crashing snapd seems an extreme response to it.

As well as reporting the bug, I'd appreciate any suggestions on how to recover the system from here. I expect that this leaves me unable to anything with snap - snaps won't run and I'm unable to apply any updates that require a snap step (eg kernel upgrades!)

2024-06-10T09:46:55.087549+01:00 VSYS-LT28 snap-failure[7147]: logger.go:93: DEBUG: will consider standby after: 5s
2024-06-10T09:46:55.087583+01:00 VSYS-LT28 snap-failure[7147]: logger.go:93: DEBUG: activation done in 2.546s
2024-06-10T09:46:55.088913+01:00 VSYS-LT28 snap-failure[7147]: logger.go:93: DEBUG: Next refresh scheduled for 2024-06-10T16:42:19+01:00.
2024-06-10T09:46:55.151959+01:00 VSYS-LT28 snap-failure[7147]: logger.go:93: DEBUG: ignoring line 141 ("X-GNOME-FullName=Inkscape Vector Graphics Editor") in source of desktop file "inkscape_inkscape.desktop"
2024-06-10T09:46:55.151979+01:00 VSYS-LT28 snap-failure[7147]: logger.go:93: DEBUG: ignoring line 262 ("TryExec=inkscape") in source of desktop file "inkscape_inkscape.desktop"
2024-06-10T09:46:55.152100+01:00 VSYS-LT28 snap-failure[7147]: logger.go:93: DEBUG: ignoring line 154 ("X-MultipleArgs=false") in source of desktop file "firefox_firefox.desktop"
2024-06-10T09:46:55.152340+01:00 VSYS-LT28 snap-failure[7147]: logger.go:93: DEBUG: ignoring line 253 ("TryExec=gimp-2.10") in source of desktop file "gimp_gimp.desktop"
2024-06-10T09:46:55.152356+01:00 VSYS-LT28 snap-failure[7147]: logger.go:93: DEBUG: ignoring line 1 ("Encoding=UTF-8") in source of desktop file "thunderbird_thunderbird.desktop"
2024-06-10T09:46:55.152389+01:00 VSYS-LT28 snap-failure[7147]: logger.go:93: DEBUG: ignoring line 108 ("X-MultipleArgs=false") in source of desktop file "thunderbird_thunderbird.desktop"
2024-06-10T09:46:55.305924+01:00 VSYS-LT28 snap-failure[7147]: logger.go:93: DEBUG: Running task 1566 on Doing: Run configure hook of "core" snap
2024-06-10T09:46:55.333013+01:00 VSYS-LT28 snap-failure[7147]: logger.go:93: DEBUG: kernel option: validating system.kernel.cmdline-append="initcall_blacklist=simpledrm_platform_driver_init"
2024-06-10T09:46:55.335192+01:00 VSYS-LT28 snap-failure[7147]: panic: runtime error: invalid memory address or nil pointer dereference
2024-06-10T09:46:55.335241+01:00 VSYS-LT28 snap-failure[7147]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x64aaf31a27c5]
2024-06-10T09:46:55.335250+01:00 VSYS-LT28 snap-failure[7147]: goroutine 242 [running]:
2024-06-10T09:46:55.335256+01:00 VSYS-LT28 snap-failure[7147]: github.com/snapcore/snapd/overlord/configstate/configcore.validateCmdlineParamsAreAllowed(0x64aaf32c4b36?, {0x64aaf38990f8?, 0xc00041d800?}, {0xc0006c0cc0, 0x31})
2024-06-10T09:46:55.335263+01:00 VSYS-LT28 snap-failure[7147]: #011/build/snapd/parts/snapd-deb/build/overlord/configstate/configcore/kernel.go:67 +0x45
2024-06-10T09:46:55.335272+01:00 VSYS-LT28 snap-failure[7147]: github.com/snapcore/snapd/overlord/configstate/configcore.validateCmdlineAppend({0x64aaf3898648?, 0xc00017fc80})
2024-06-10T09:46:55.335278+01:00 VSYS-LT28 snap-failure[7147]: #011/build/snapd/parts/snapd-deb/build/overlord/configstate/configcore/kernel.go:99 +0x352
2024-06-10T09:46:55.335284+01:00 VSYS-LT28 snap-failure[7147]: github.com/snapcore/snapd/overlord/configstate/configcore.(*withStateHandler).validate(0xc00043da20, {0x7ca9919a41c0?, 0xc00017fc80})
2024-06-10T09:46:55.335290+01:00 VSYS-LT28 snap-failure[7147]: #011/build/snapd/parts/snapd-deb/build/overlord/configstate/configcore/runwithstate.go:110 +0x46
2024-06-10T09:46:55.335296+01:00 VSYS-LT28 snap-failure[7147]: github.com/snapcore/snapd/overlord/configstate/configcore.applyHandlers({0x7ca9919a5960, 0xc000312dc8}, {0x64aaf3898648, 0xc00017fc80}, {0xc0000da800, 0x1c, 0xc0008d3a58?})
2024-06-10T09:46:55.335303+01:00 VSYS-LT28 snap-failure[7147]: #011/build/snapd/parts/snapd-deb/build/overlord/configstate/configcore/runwithstate.go:169 +0xf4
2024-06-10T09:46:55.335309+01:00 VSYS-LT28 snap-failure[7147]: github.com/snapcore/snapd/overlord/configstate/configcore.Run({0x7ca9919a5960?, 0xc000312dc8?}, {0x64aaf3898648?, 0xc00017fc80?})
2024-06-10T09:46:55.335315+01:00 VSYS-LT28 snap-failure[7147]: #011/build/snapd/parts/snapd-deb/build/overlord/configstate/configcore/runwithstate.go:148 +0x45
2024-06-10T09:46:55.335320+01:00 VSYS-LT28 snap-failure[7147]: github.com/snapcore/snapd/overlord/configstate.Init.func1(0x64aaf3775d40?)
2024-06-10T09:46:55.335326+01:00 VSYS-LT28 snap-failure[7147]: #011/build/snapd/parts/snapd-deb/build/overlord/configstate/configmgr.go:82 +0x3a
2024-06-10T09:46:55.335332+01:00 VSYS-LT28 snap-failure[7147]: github.com/snapcore/snapd/overlord/hookstate.(*HookManager).runHook(0xc0006ee0f0, 0xc0002c7340, 0xc000371680?, 0xc000371680, 0x0?)
2024-06-10T09:46:55.335347+01:00 VSYS-LT28 snap-failure[7147]: #011/build/snapd/parts/snapd-deb/build/overlord/hookstate/hookmgr.go:399 +0x5f1
2024-06-10T09:46:55.335353+01:00 VSYS-LT28 snap-failure[7147]: github.com/snapcore/snapd/overlord/hookstate.(*HookManager).runHookForTask(0xc0006ee0f0, 0x64aaf32a3304?, 0xa?, 0xc0003ae568?, 0x64aaf2974a1d?)
2024-06-10T09:46:55.335362+01:00 VSYS-LT28 snap-failure[7147]: #011/build/snapd/parts/snapd-deb/build/overlord/hookstate/hookmgr.go:313 +0x76
2024-06-10T09:46:55.335367+01:00 VSYS-LT28 snap-failure[7147]: github.com/snapcore/snapd/overlord/hookstate.(*HookManager).doRunHook(0x0?, 0xc000287e00, 0x64aaf29d3537?)
2024-06-10T09:46:55.335391+01:00 VSYS-LT28 snap-failure[7147]: #011/build/snapd/parts/snapd-deb/build/overlord/hookstate/hookmgr.go:265 +0x118
2024-06-10T09:46:55.335398+01:00 VSYS-LT28 snap-failure[7147]: github.com/snapcore/snapd/overlord/state.(*TaskRunner).run.func1()
2024-06-10T09:46:55.335404+01:00 VSYS-LT28 snap-failure[7147]: #011/build/snapd/parts/snapd-deb/build/overlord/state/taskrunner.go:220 +0xb4
2024-06-10T09:46:55.335410+01:00 VSYS-LT28 snap-failure[7147]: gopkg.in/tomb%2ev2.(*Tomb).run(0xc00057eb40, 0x68742f6572616873?)
2024-06-10T09:46:55.335418+01:00 VSYS-LT28 snap-failure[7147]: #011/build/snapd/parts/snapd-deb/build/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x36
2024-06-10T09:46:55.335424+01:00 VSYS-LT28 snap-failure[7147]: created by gopkg.in/tomb%2ev2.(*Tomb).Go
2024-06-10T09:46:55.335430+01:00 VSYS-LT28 snap-failure[7147]: #011/build/snapd/parts/snapd-deb/build/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xee
2024-06-10T09:46:55.336930+01:00 VSYS-LT28 snap-failure[7141]: error: snapd failed: exit status 2
2024-06-10T09:46:55.337493+01:00 VSYS-LT28 systemd[1]: snapd.failure.service: Main process exited, code=exited, status=1/FAILURE
2024-06-10T09:46:55.337529+01:00 VSYS-LT28 systemd[1]: snapd.failure.service: Failed with result 'exit-code'.
2024-06-10T09:46:55.337574+01:00 VSYS-LT28 systemd[1]: Failed to start snapd.failure.service - Failure handling of the snapd snap.
2024-06-10T09:46:55.337640+01:00 VSYS-LT28 systemd[1]: snapd.failure.service: Consumed 10.916s CPU time.

[Tom Cook (tom-k-cook)]

I've spent a little while trying to debug this but someone familiar with the code will have a better idea. I've verified that a build from the current master branch shows the same problem.

The immediate problem is that devicestate.CurrentGadgetData() can return nil,nil (at handlers_gadget.go:60) in the case that there is not yet any data associated with the gadget. Adding a check for this to configcore.validateCmdlineParamsAreAllowed() prevents the crash and appears to be enough to run `snap unset system system.kernel.cmdline-append`; even though this fails because it also runs validateCmdlineParamsAreAllowed(), it fails after it has written the updated config.

I've tried stepping into this in a debugger but it's beyond me for the moment. In infoForDeviceSnap(), deviceCtx.Model() is returning a model which has gadgetSnap == nil and this in turn results in CurrentGadgetData returning nil,nil - but I can't say why the model is in that state.

[Philip Meulengracht (the-meulengracht)]

Hey!

Thank you for reporting this - in essence what you are experiencing was an oversight in the validation of the kernel parameters.

Kernel parameters are only supported on systems where a model assertion is available, and has both a kernel and gadget snap. On regular, classic systems, like yours, this is not supported.

A PR has just been merged to add additional safeguards around this. This fix is scheduled for snapd 2.64.

[Tom Cook (tom-k-cook)]

Thanks for following this up and I'm glad a fix has resulted. In terms of my system, does this mean I'm not using snap to manage the kernel and initramfs? Possibly as a result of upgrading from 23.10 rather than installing from scratch.

Should I instead be editing grub configuration by hand to add kernel commandline params?