systemd ignoring /etc/modules due to blacklist

[Paul Crawford]

I tried the daily build of 16.04 32-bit to test out the watchdog daemon code. Usually (Ubuntu 10.04-14.04) I add the watchdog module in /etc/modules so it is loaded at boot-time, as watchdog timer modules are not normally auto-loaded due to the risk of an unexpected reboot.

However I now find that systemd is choosing to ignore my command to load the module in /etc/modules since it appears in the watchdog blacklist. Typical syslog entries look like this:

Jan 19 16:46:14 ubuntu systemd-modules-load[337]: Module 'softdog' is blacklisted
Jan 19 17:53:23 ubuntu systemd-modules-load[342]: Module 'softdog' is blacklisted

This is just dumb! I have explicitly told the system to load the module, an action that works perfectly well using modprobe or by adding it to the start script for the watchdog, and yet systemd chooses to override that because of the blacklist for auto-loaded modules (in this case in /etc/modprobe.d/blacklist-watchdog.conf).

$ lsb_release -rd
Description: Ubuntu Xenial Xerus (development branch)
Release: 16.04

$ apt-cache policy systemd
systemd:
  Installed: 228-4ubuntu1
  Candidate: 228-4ubuntu1
  Version table:
 *** 228-4ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/main i386 Packages
        100 /var/lib/dpkg/status

What I expect to happen is modules added to /etc/modules are loaded at boot time, and not subject to the blacklist for hardware detect / automatic loading.

[Olivier Louvignes (olouvignes)]

I'm having the exact same issue, can't load watchdog TCO-related modules because they are blacklisted.

Did you find any workaround?

I don't want to mess with the blacklist list.

[Paul Crawford (psc-sat)]

Generally me work-around so far has been to avoid systemd!
However, where that is not an option you might be able to edit /etc/default/watchdog so the module is loaded on watchdog start-up. It is not ideal as you might have some other reasons for wanting the /dev/watchdog virtual file to be testable without the watchdog having started though (e.g. automated install script that probes harder support, etc).

[Olivier Louvignes (olouvignes)]

Thanks for the tip, will try that. What is extremely frustrating is that to unblacklist a module you have to edit 4 different, some dynamic files. I don't really get how such poor design could ever be validated. Maybe we're missing something (systemd config?).

```
# Blacklists that must be edited on 16.04:
root@toto:/lib/modprobe.d# grep -R iTCO_wdt /*/modprobe.d
/etc/modprobe.d/blacklist-watchdog.conf:# blacklist iTCO_wdt
/lib/modprobe.d/blacklist_linux_4.4.0-64-generic.conf:blacklist iTCO_wdt
/lib/modprobe.d/blacklist_linux-hwe-edge_4.8.0-34-generic.conf:blacklist iTCO_wdt
/lib/modprobe.d/blacklist_linux_4.4.0-63-generic.conf:blacklist iTCO_wdt
```

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andrejs Bondarevs (abondarevs)]

I'm having the same issue. In fact, removing iTCO_wdt from all blacklists doesn't solve the problem. At the boot I still get:

systemd-modules-load[365]: Module 'iTCO_wdt' is blacklisted

Adding the following to /etc/modprobe.d/watchdog.conf doesn't help either:

install /sbin/modprobe iTCO_wdt

[Ingo Fischer (apollon77)]

I also have this problem and it is stupid to have watchdog started manually using systemd after each (re)boot to work.

I have removed all blacklist-entries for iTCO_wdt and also rebuild initramfs ... but it still says at boot that module is blacklisted ... :-(

[starox (starox)]

Same problem here, once you have removed the modules from every blacklist files, you must update initramfs:

$ sudo update-initramfs -u

Then, it works ...

[Ingo Fischer (apollon77)]

I tried that but I had no success.

I now have the /etc/init.d/watchdog start in /etc/rc.local (systemctl did not worked that way) and so start using that

[David (arggggggggggggggggggggggggggggggggggggggggggh)]

This problem still exists.

The problem is that these modules are listed in the blacklist files provided with the kernel(/lib/modprobe.d/blacklist_linux-*.conf). Given that these files aren't located in /etc they apparently are not configuration files. Also very new kernel package you get a new file with a different name.

Why are they blacklisted in this way? Why not only in /etc/modprobe.d/blacklist-watchdog.conf? Is there a way to overwrite this blacklisting through configuration files?

[AW01545 (ricknickle)]

Issue still exists in 18.04.

[AW01545 (ricknickle)]

In my case I'm trying to get softdog to load for SBD instead of the watchdog daemon.

I can see the point of marking it as opinion, but as Olivier points out, you have to eliminate the blacklist entry from all of the blacklists spammed into /lib/modprobe.d, systemd-modules-load does not discriminate for particular kernel versions and reads every .conf file.

[Ishwor Gurung (ishworgurung)]

To get this working on 18.04, delete all the blacklisted module name from /lib/modprobe.d/*.conf

Then, add the module name to `/etc/modules-load.d/myfoo-module.conf`.

After that three things are needed:

$ sudo update-initramfs -u
$ sudo depmod -a
$ sudo systemctl reboot

[Filippo Argiolas (fargiolas)]

Any news about this? it seems all watchdog modules are blacklisted unless the watchdog daemon is installed and one module is enabled in /etc/default/watchdog. With basic watchdog functionality already enabled in systemd shouldn't this be changed to support it?

[Nathan Rennie-Waldock (nathan-renniewaldock)]

This issue makes it annoying to load watchdog drivers, as systemd now supports hardware watchdogs they'd have to be added to rc.local or a custom service. The pre-systemd behaviour only applied the blacklist with udev, not when processing /etc/modules(-load.d) so that they could be prevented from loading automatically unless explicitly listed by the user. So this is more of a regression due to behaviour change than opinion.

[Jan-Jonas Sämann (janjonas)]

What @fargiolas wrote, solved this issue for me on 20.04.

Configure watchdog module in /etc/defaults/watchdog

The watchdog service loads the module itself on demand. You don't have to load the module yourself then.

Seems that the issue can be closed now.

[Federico Reghenzani (federeghe)]

In some systems, like embedded devices, it is needed to have the control of the (hardware) watchdog in the application itself, not managed by systemd. In this case, the solution of /etc/default/watchdog doesn't work. Then, if all modules are blacklisted, I have to create a script and manually load them, this is not the best thing, especially for a watchdog!

So, this issue still exists for watchdog not managed by the watchdog daemon.

The question should be: why ubuntu needs to blacklist all these modules?

[James Cuzella (trinitronx)]

Confirmed this still exists in 22.04 (Jammy Jellyfish), with symptoms being SystemD booting always into "degraded" state because wdmd.service (Watchdog Multiplexing Daemon) will always fail due to missing watchdog kernel module(s).

Check this is the case with:

    sudo systemctl status
    sudo systemctl list-units --failed

The solution suggested by @ishworgurung works to allow iTCO_wdt (and other watchdogs) to be loaded by the kernel during boot appropriately. Then, wdmd.service starts up successfully and SystemD enters successful "running" state (assuming all other SystemD services are working properly).
However, every kernel package update will place another blacklist file in /lib/modprobe.d/blacklist_linux_$(uname -r).conf. So this is only a temporary workaround until these modules are not blacklisted by default.

The blacklist files are part of linux-modules-$(uname -r) package(s). Therefore, this bug should be listed as also affecting Linux kernel source packages for Ubuntu.

    $ dpkg -S /lib/modprobe.d/blacklist_linux_*.conf
    linux-modules-5.15.0-74-generic: /lib/modprobe.d/blacklist_linux_5.15.0-74-generic.conf
    linux-modules-5.4.0-149-generic: /lib/modprobe.d/blacklist_linux_5.4.0-149-generic.conf
    # ^^ These packages own /lib/modprobe.d/blacklist_linux*.conf files

    $ apt-cache show linux-modules-$(uname -r) | grep -C5 Source: | sed -e 's/^/ /'
    Package: linux-modules-5.15.0-74-generic
    Architecture: amd64
    Version: 5.15.0-74.81
    Priority: optional
    Section: kernel
    Source: linux
    Origin: Ubuntu
    Maintainer: Ubuntu Kernel Team <email address hidden>
    Bugs: https://bugs.launchpad.net/ubuntu/+filebug
    Installed-Size: 123864
    Depends: linux-image-5.15.0-74-generic | linux-image-unsigned-5.15.0-74-generic
    ## ^^ Part of Linux APT package(s) in Ubuntu

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1535840

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.