watchdog daemon going in to failed state on reboot

[Paul Crawford]

I was testing the watchdog daemon code on the development version of 16.04 today and found the associated systemd service files for this has some bugs.

The first of these was a typo in /lib/systemd/system/watchdog.service where there was a missing ['] character. This lead to an error message "Unbalanced quoting, ignoring" which was easy to fix. The update is now in the source-forge repository for the project as commit http://sourceforge.net/p/watchdog/code/ci/38e6430f80907a84741c760ef48df69a679b294c/

However, I have not found the reason for the second problem where the daemon has some fault at reboot time and goes in to "failed state" and then it will not restart with the machine booting. The typical related syslog entries are:

Jan 19 16:46:08 ubuntu watchdog[2066]: stopping daemon (5.14)
Jan 19 16:46:08 ubuntu systemd[1]: Stopping watchdog daemon...
Jan 19 16:46:09 ubuntu systemd[1]: watchdog.service: Control process exited, code=exited status=1
Jan 19 16:46:09 ubuntu systemd[1]: Stopped watchdog daemon.
Jan 19 16:46:09 ubuntu systemd[1]: watchdog.service: Unit entered failed state.
Jan 19 16:46:09 ubuntu systemd[1]: watchdog.service: Triggering OnFailure= dependencies.
Jan 19 16:46:09 ubuntu systemd[1]: watchdog.service: Failed to enqueue OnFailure= job: Resource deadlock avoided
Jan 19 16:46:09 ubuntu systemd[1]: watchdog.service: Failed with result 'exit-code'.

I am guessing this is related to the shut-down approach for the watchdog daemon where normally the wd_keepalive daemon is started afterwards (to prevent a reboot if the hardware module is configured for "no way out" so the timer cannot be stopped).

$ lsb_release -rd
Description: Ubuntu Xenial Xerus (development branch)
Release: 16.04

$ apt-cache policy watchdog
watchdog:
  Installed: 5.14-3
  Candidate: 5.14-3
  Version table:
 *** 5.14-3 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages
        100 /var/lib/dpkg/status

I have contacted the watchdog project maintainer with a view to working out a solution to this. This bug report is more of a marker to let folks know that there is an issue here if you plan on having high-availability servers based on Ubuntu 16.04 (well, any systemd based system really..) where watchdog-based fault recovery is an expectation.

[Paul Crawford (psc-sat)]

Just to add this is only a problem on shut down / reboot, you can manually stop and start the daemon without problems using:

service watchdog start
service watchdog stop

[Paul Crawford (psc-sat)]

This may be a duplicate of bug #1535854

[Paul Crawford (psc-sat)]

Doh! I meant bug #1448924 (above is this bug)

[Paul Crawford (psc-sat)]

I just tested the proposed fix from https://bugs.launchpad.net/ubuntu/+source/watchdog/+bug/1448924/comments/7 and while it includes the fix for the "unbalanced quoting" it does not fix the system going in to a failed state.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[skywriter (xxxiter)]

May 2017. Watchdog still doesn't start after reboot. No relevant records in syslog by 'grep watchdog'.

[skywriter (xxxiter)]

I obtained that the file '/lib/systemd/system/watchdog.service' does not have [Install] section. Fixed by copying this file into '/etc/systemd/system' and modifying it.

[Darrell Enns (darrellenns)]

Here is the upstream fix in debian: https://sources.debian.net/src/watchdog/5.15-2/debian/watchdog.service/

Basically, just add "WantedBy=default.target" to the install section.

For some reason Ubuntu 16.04 is still on Watchdog 5.14-3, even though 5.15-2 (with the fix) is already in Debian sid.

[Lenin (gagarin)]

Fixed with 18.04 LTS