Regression: /etc/modules checked against blacklist or it's really hard to load blacklisted watchdog modules when one really wants one

[Kees Cook]

Impossible / hard to force the system to load a watchdog module because it is blacklisted by the kernel auto-generated list of "watchdog" modules.

/etc/modules used to "just work" before.

e.g. bcm2835_wdt module on arm64

===

Before systemd-modules-load, /etc/init.d/kmod would load modules directly with "modprobe" (and _not_ "modprobe -b"):

load_module() {
  local module args
  module="$1"
  args="$2"

  if [ "$VERBOSE" != no ]; then
    log_action_msg "Loading kernel module $module"
    modprobe $module $args || true
  else
    modprobe $module $args > /dev/null 2>&1 || true
  fi
}

However, under 18.04, systemd-modules-load will _ignore_ modules that are manually listed in /etc/modules and process them with the blacklist (the same as "modprobe -b" would). This means that it is not possible to manually load modules that are blacklisted (like watchdog modules):

systemd-238/src/modules-load/modules-load.c:

static int load_module(struct kmod_ctx *ctx, const char *m) {
        const int probe_flags = KMOD_PROBE_APPLY_BLACKLIST;
...
                default:
                        err = kmod_module_probe_insert_module(mod, probe_flags,
                                                              NULL, NULL, NULL, NULL);

                        if (err == 0)
                                log_info("Inserted module '%s'", kmod_module_get_name(mod));
                        else if (err == KMOD_PROBE_APPLY_BLACKLIST)
                                log_info("Module '%s' is blacklisted", kmod_module_get_name(mod));

Blacklists should _not_ be applied by systemd-modules-load.

[Kees Cook (kees)]

https://github.com/systemd/systemd/pull/8830

[Dimitri John Ledkov (xnox)]

Looking at where the blacklists come from, it looks like kernel autogenerates "blacklist_linux_4.15-0-15-generic.conf" files which have a combination of OSS and watchdog blacklists.

this indeed is not very nice/easy to override.

I wonder if the "watchdog" like modules should be shipped in separate files elsewhere on disk, and collected into a "watchdog.conf" file which matches blacklists for all installed kernels? That way it would be easier for admin to override it, by dropping in an empty /etc/modules-load.d/watchdog.conf?

Not sure how to make these modules 'available' yet not autoloaded. Split them into a separate -watchdog-modules binary package? not installed = not autoloaded? without any blacklists at all?

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1767172

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Dimitri John Ledkov (xnox)]

Please note automatic blacklisting of watchdog modules was done to prevent "accidental" watchdog misuse, resulting in "random" shutdowns of production machines. See https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1432837 for more details.

I'm not sure how to reconcile "blacklist as a precaution" with "I know what I am doing, I really want watchdog".

[Mathieu Trudel-Lapierre (cyphermox)]

Kees,

I the fix in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1766052 sufficient to address the situation? Do we still need to do more to systemd (note, the GH issue was closed upstream following Dimitri's input)?

[Kees Cook (kees)]

I think it's fine. It sounds like there will just be no way to override package-installed blacklists any more. That's unfortunate, but it's a very rare situation.

[Jan Graichen (jgraichen)]

> I think it's fine. It sounds like there will just be no way to override package-installed blacklists any more. That's unfortunate, but it's a very rare situation.

The i6300esb watchdog driver is required for every KVM/qemu virtual machine with an emulated watchdog (https://wiki.openstack.org/wiki/LibvirtWatchdog).

Having no easy option to enable that module for virtual machines is a blocker for having them reset when stuck.

See the following issues to:

* https://bugs.launchpad.net/ubuntu/+source/linux/+bug/642930
* https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1948040
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1009350
* https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1432837