[bionic] dlm_controld won't start due to missing device files

[Rafael David Tinoco]

Right after installing dlm and dlm_controld, when trying to start the service, you will face:

rafaeldtinoco@clubionic01:~$ systemctl status dlm.service
● dlm.service - dlm control daemon
   Loaded: loaded (/lib/systemd/system/dlm.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2020-03-06 19:13:32 UTC; 35s ago
     Docs: man:dlm_controld
           man:dlm.conf
           man:dlm_stonith
  Process: 31644 ExecStart=/usr/sbin/dlm_controld --foreground $DLM_CONTROLD_OPTS (code=exited, status=1/FAILURE)
  Process: 31643 ExecStartPre=/sbin/modprobe dlm (code=exited, status=0/SUCCESS)
 Main PID: 31644 (code=exited, status=1/FAILURE)

Mar 06 19:13:21 clubionic01 systemd[1]: Starting dlm control daemon...
Mar 06 19:13:21 clubionic01 dlm_controld[31644]: 8746 dlm_controld 4.0.7 started
Mar 06 19:13:32 clubionic01 dlm_controld[31644]: 8756 cannot find device /dev/misc/dlm-control with minor 56
Mar 06 19:13:32 clubionic01 systemd[1]: dlm.service: Main process exited, code=exited, status=1/FAILURE
Mar 06 19:13:32 clubionic01 systemd[1]: dlm.service: Failed with result 'exit-code'.
Mar 06 19:13:32 clubionic01 systemd[1]: Failed to start dlm control daemon.

This happens even after the module "dlm" was loaded (as you can see in the ExecStartPre= line).

(k)rafaeldtinoco@clubionic01:~$ ls -lah /dev/dlm*
crw------- 1 root root 10, 56 Mar 6 19:08 /dev/dlm-control
crw------- 1 root root 10, 55 Mar 6 19:08 /dev/dlm-monitor
crw------- 1 root root 10, 54 Mar 6 19:08 /dev/dlm_plock

And we can see hardcoded device filename paths:

dlm_controld/action.c: rv = find_udev_device("/dev/misc/dlm-control", control_minor);
dlm_controld/action.c: rv = find_udev_device("/dev/misc/dlm-monitor", monitor_minor);
dlm_controld/action.c: rv = find_udev_device("/dev/misc/dlm_plock", plock_minor);

And the way rules.d files are generated:

rafaeldtinoco@workstation:~/.../dlm$ grep -r UDEVDIR *
libdlm/Makefile:UDEVDIR=/usr/lib/udev/rules.d
libdlm/Makefile: $(INSTALL) -d $(DESTDIR)/$(UDEVDIR)
libdlm/Makefile: $(INSTALL) -m 644 $(UDEV_TARGET) $(DESTDIR)/$(UDEVDIR)

And that the package contains those:

(k)rafaeldtinoco@clubionic01:~$ cat /lib/udev/rules.d/51-dlm.rules
KERNEL=="dlm-control", MODE="0666", SYMLINK+="misc/dlm-control"
KERNEL=="dlm-monitor", MODE="0666", SYMLINK+="misc/dlm-monitor"
KERNEL=="dlm_plock", MODE="0666", SYMLINK+="misc/dlm_plock"
KERNEL=="dlm_*", MODE="0660", SYMLINK+="misc/%k"

[Rafael David Tinoco (rafaeldtinoco)]

I have monitored udev when modprobing dlm module and those are the udev events being triggered:

(k)rafaeldtinoco@clubionic01:~$ sudo udevadm monitor --kernel
monitor will print the received events for:
KERNEL - the kernel uevent

KERNEL[10516.364250] add /module/dlm (module)
KERNEL[10516.364425] add /kernel/slab/:0000496 (slab)
KERNEL[10516.364579] add /kernel/slab/:0000312 (slab)
KERNEL[10516.364730] add /devices/virtual/misc/dlm-control (misc)
KERNEL[10516.364849] add /devices/virtual/misc/dlm-monitor (misc)
KERNEL[10516.364980] add /devices/virtual/misc/dlm_plock (misc)

and when being removed:

KERNEL[10713.488367] remove /devices/virtual/misc/dlm_plock (misc)
KERNEL[10713.488465] remove /devices/virtual/misc/dlm-control (misc)
KERNEL[10713.488539] remove /devices/virtual/misc/dlm-monitor (misc)
KERNEL[10713.488635] remove /kernel/slab/:0000496 (slab)
KERNEL[10713.488692] remove /kernel/slab/:0000312 (slab)
KERNEL[10713.488825] remove /module/dlm (module)

[Rafael David Tinoco (rafaeldtinoco)]

It only happens during package installation. Right after the package is already installed, you can "modprobe -r dlm" and "modprobe dlm" again and then the devices are properly translated from existing - from the pkg - udev rules.

(k)rafaeldtinoco@clubionic03:~$ systemctl status dlm
● dlm.service - dlm control daemon
   Loaded: loaded (/lib/systemd/system/dlm.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-03-06 20:25:24 UTC; 1min 24s ago
     Docs: man:dlm_controld
           man:dlm.conf
           man:dlm_stonith
 Main PID: 1365 (dlm_controld)
    Tasks: 2 (limit: 2338)
   CGroup: /system.slice/dlm.service
           └─1365 /usr/sbin/dlm_controld --foreground

Mar 06 20:25:23 clubionic03 systemd[1]: Starting dlm control daemon...
Mar 06 20:25:23 clubionic03 dlm_controld[1365]: 67 dlm_controld 4.0.7 started
Mar 06 20:25:24 clubionic03 systemd[1]: Started dlm control daemon.

(k)rafaeldtinoco@clubionic03:~$ ls -1lah /dev/misc/*
lrwxrwxrwx 1 root root 14 Mar 6 20:24 /dev/misc/dlm-control -> ../dlm-control
lrwxrwxrwx 1 root root 14 Mar 6 20:24 /dev/misc/dlm-monitor -> ../dlm-monitor
lrwxrwxrwx 1 root root 12 Mar 6 20:24 /dev/misc/dlm_plock -> ../dlm_plock