watchdogd doesn't start on boot

[Arne Bockholdt]

[Impact]

 * The watchdog service in Xenial does not start automatically even after
   being configured. This makes the service not function reliably.

   This affects the 5.14 release in Xenial only and is already fixed in
   5.15 included in the developement release.

[Test Case]

 * On a Xenial 16.04 system
    - lxc launch ubuntu-daily:xenial x1
    - lxc exec x1 /bin/bash

    # inside the container x1
    - apt-get update && apt-get install watchdog

    # display status after installing (returns inactive)
    - systemctl is-active watchdog

    # reboot (this exits the container)
    - reboot

    - lxc exec x1 /bin/bash
    # inside the container x1, check the status of the service
    - systemctl is-active watchdog (returns active)

    # With current version of watchdog (5.14-3ubuntu0.16.04.1)
    # the service does not start automatically;

    # With the updated version (5.14-3ubuntu0.16.04.2)
    # the service will be started automatically after a reboot

[Regression Potential]

 * Users of watchdog service which previously did not start automatically
   may encounter watchdog triggering actions at unexpected times.

[Original Description]

Description: Ubuntu 15.04
Release: 15.04

watchdog:
  Installed: 5.14-3
  Candidate: 5.14-3
  Version table:
 *** 5.14-3 0
        500 http://de.archive.ubuntu.com/ubuntu/ vivid/universe amd64 Packages
        100 /var/lib/dpkg/status

Expected:

Watchdog daemon starts on system boot.

Current:

After updating from Utopic to Vivid the watchdog daemon doesn't start. A manual service watchdog start after boot up starts the daemon normally.
There seems to be a problem with the transition of the init scripts from Upstart to SystemD.

[Christian Decker (decker-christian)]

Hello,

what are your hardware settings?

Greets

[Arne Bockholdt (u-launchpad-net-bockholdt-com)]

The problem happens on several KVM machines with (virtual) Intel 6300ESB WatchDog. Here's the driver load log from dmesg :

[37539.207098] i6300esb: Intel 6300ESB WatchDog Timer Driver v0.05
[37539.210190] i6300esb: initialized (0xffffc9000007c000). heartbeat=30 sec (nowayout=0)

I've attached the watchdog.conf that worked since the initial install of the VMs.

[Dave Gilbert (ubuntu-treblig)]

Medium: Problem with non-essential hardware component
(With a workaround of restarting the service).

Arne: I suggest trying a systemctl enable watchdogd to see if it starts at the next boot and remembers it.

[Arne Bockholdt (u-launchpad-net-bockholdt-com)]

Tried that before but I get the following messages :

Synchronizing state for watchdog.service with sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d watchdog defaults
Executing /usr/sbin/update-rc.d watchdog enable
The unit files have no [Install] section. They are not meant to be enabled
using systemctl.
Possible reasons for having this kind of units are:
1) A unit may be statically enabled by being symlinked from another unit's
   .wants/ or .requires/ directory.
2) A unit's purpose may be to act as a helper for some other unit which has
   a requirement dependency on it.
3) A unit may be started when needed via activation (socket, path, timer,
   D-Bus, udev, scripted systemctl call, ...).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andre Tomt (andre-tomt)]

On some systems, watchdog not starting at boot can lead to infinite system reset loops.

Better behaved hardware watchdogs starts the counter on first open/write to /dev/watchdog and thus not starting on boot is not fatal, but will cause system lockups to go undetected.

[Arne Bockholdt (u-launchpad-net-bockholdt-com)]

Just tested it with the current Wily version and the bug still persists. So I had a look into the systemd service script and I was able to fix it for myself, simply copy this to file /lib/systemd/system/watchdog.service :

<snip>

[Unit]
Description=watchdog daemon
Conflicts=wd_keepalive.service
After=multi-user.target
OnFailure=wd_keepalive.service

[Service]
Type=forking
EnvironmentFile=/etc/default/watchdog
ExecStartPre=/bin/sh -c '[ -z "${watchdog_module}" ] || [ "${watchdog_module}" = "none" ] || /sbin/modprobe $watchdog_module'
ExecStart=/bin/sh -c '[ $run_watchdog != 1 ] || exec /usr/sbin/watchdog $watchdog_options'
ExecStopPost=/bin/sh -c '[ $run_wd_keepalive != 1 ] || false'

[Install]
WantedBy=multi-user.target

</snip>

How do I submit patches/fixes to the Canonical repos so that fix get into the distro ?

[Paul Crawford (psc-sat)]

This is probably the same problem as bug #1535854 thought there could be a number of related factors, for example bug #1535840 for module loading.

[Paul Crawford (psc-sat)]

I tested the proposed fix from comment #7 on bug #1535854 but it did not work for that system (16.04) :(

[Ryan Harper (raharper)]

Pull in two fixes from upstream debian 5.15 release which resolves the service starting issue.

Note, users upgrading will need to run:

systemctl enable watchdog

after upgrading to ensure watchdog service is enabled (new installers of the package will have the service enabled by default).

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "fix-watchdog-service.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Ryan Harper (raharper)]

I've uploaded a version of watchdog with this debdiff attached to my ppa:

ppa:raharper/bugfixes

Please give a go and see if that resolves the starting on boot issue.

I've verified this fix myself under QEMU with a Xenial cloud image:

# create some user-data for easy login
% cat user-data
#cloud-config
password: passw0rd
chpasswd: { expire: False }
ssh_pwauth: True

# create a local-datasource
% cloud-localds -m local seed.img user-data

# launch the cloud-image under snapshot with the seed and watchdog device
qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 \
    -drive file=xenial-server-cloudimg-amd64-disk1.img,snapshot=on,format=qcow2,id=boot \
    -watchdog i6300esb -watchdog-action reset \
    -net user -net nic,model=virtio -nographic \
    -cdrom seed.img \
    -redir tcp:2222::22 \
    -serial stdio

# ssh into the guest after boot
1. ssh -o UserKnownHostsFile=/dev/null -p 2222 ubuntu@localhost

# install watchdog kernel module (not in cloud image by default)
2. sudo apt-get update && sudo apt-get install linux-image-generic

# load watchdog module
3. sudo modprobe i6300esb

# install watchdog service
4. sudo apt-get install watchdog

3. sudo systemctl status watchdog

# observe error from current package
Apr 18 20:32:01 ubuntu systemd[1]: [/lib/systemd/system/watchdog.service:10] Unbalanced quoting, ignoring:

# install new package
4. sudo dpkg --install watchdog_5.14-3ubuntu1_amd64.deb

# tell systemd to install watchdog service
5. systemctl enable watchdog
Synchronizing state of watchdog.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable watchdog

# start it up
# systemctl start watchdog
root@ubuntu:~# echo $?
0
root@ubuntu:~# systemctl status watchdog
● watchdog.service - watchdog daemon
   Loaded: loaded (/lib/systemd/system/watchdog.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2016-04-18 20:33:14 UTC; 6s ago
  Process: 10518 ExecStart=/bin/sh -c [ $run_watchdog != 1 ] || exec /usr/sbin/watchdog $watchdog_options
  Process: 10515 ExecStartPre=/bin/sh -c [ -z "${watchdog_module}" ] || [ "${watchdog_module}" = "none" ]
 Main PID: 10521 (watchdog)
    Tasks: 1 (limit: 512)
   Memory: 356.0K
      CPU: 5ms
   CGroup: /system.slice/watchdog.service
           └─10521 /usr/sbin/watchdog

Apr 18 20:33:14 ubuntu systemd[1]: Starting watchdog daemon...
Apr 18 20:33:14 ubuntu watchdog[10521]: starting daemon (5.14):
Apr 18 20:33:14 ubuntu watchdog[10521]: int=1s realtime=yes sync=no soft=no mla=0 mem=0
Apr 18 20:33:14 ubuntu watchdog[10521]: ping: no machine to check
Apr 18 20:33:14 ubuntu watchdog[10521]: file: no file to check
Apr 18 20:33:14 ubuntu watchdog[10521]: pidfile: no server process to check
Apr 18 20:33:14 ubuntu watchdog[10521]: interface: no interface to check
Apr 18 20:33:14 ubuntu watchdog[10521]: temperature: no sensors to check
Apr 18 20:33:14 ubuntu watchdog[10521]: test=none(0) repair=none(0) alive=none heartbeat=none to=root no_a
Apr 18 20:33:14 ubuntu systemd[1]: Started watchdog daemon.

[Paul Crawford (psc-sat)]

I will try to look tomorrow or so at the proposed fix.
Have you checked if it is still loaded OK following a reboot?

[Ryan Harper (raharper)]

On Mon, Apr 18, 2016 at 4:05 PM, Paul Crawford <email address hidden>
wrote:

> I will try to look tomorrow or so at the proposed fix.
> Have you checked if it is still loaded OK following a reboot?
>

Yes, it continues to work after reboot.

It's best if you configure /etc/default/watchdog and specify the module of
your watchdog device;
This ensures that the watchdog service will modprobe the correct module for
your watchdog device
in the case that it doesn't autoload.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1448924
>
> Title:
> watchdogd doesn't start on boot
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/watchdog/+bug/1448924/+subscriptions
>

[Michael Terry (mterry)]

This bug was fixed in the package watchdog - 5.15-1

---------------
watchdog (5.15-1) unstable; urgency=medium

  * New upstream version.
  * Updated Dutch translation. (Closes: #772760)
  * Updated Russian translation. (Closes: #776125)
  * Updated Japanese translation. (Closes: #815857)
  * Added missing quote to systemd service file. (Closes: #783166, #798294)
  * Use dh_builddeb to build the package. (Closes: #784211, #808692)
  * Re-added Install stanza to service file. (Closes: #793309)

 -- Michael Meskes <email address hidden> Thu, 21 Jan 2016 22:54:48 +0100

[David Kowis (dkowis)]

Forgive my ignorance, but is this going to make it into a standard xenial updates repo?

It's very broke as it stands right now.

[David Kowis (dkowis)]

Also, I've grabbed and installed the 5.15-1 .dpkg. systemctl reports that it's enabled, but it never starts on boot. Checking right after boot yields this:

```
root@hylden:~# systemctl status watchdog
● watchdog.service - watchdog daemon
   Loaded: loaded (/lib/systemd/system/watchdog.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
root@hylden:~# ps -eaf | grep watchdog
root 10 2 0 20:01 ? 00:00:00 [watchdog/0]
root 11 2 0 20:01 ? 00:00:00 [watchdog/1]
root 978 959 0 20:02 pts/0 00:00:00 grep --color=auto watchdog
```

```
root@hylden:/etc/default# systemctl is-enabled watchdog
enabled
```

I've disabled and re-enabled the init script via `systemctl disable watchdog` and `systemctl enable watchdog` but I never get a watchdog started on boot. I always have to manually start it.

What am I missing?

[David Kowis (dkowis)]

Sorry for the noise on this bug! It turns out I'm simply not waiting long enough for it to come up. It is being started on boot, and everything is working like I expect, just isn't showing up.

I do still have the question about when it'll make it into a Xenial-updates repo, though.

Thanks, and again sorry for the bug noise.

[Alexander E. Patrakov (patrakov-gmail)]

David, how long did you have to wait? For me, it's still not started, even though the uptime is 7 minutes. And I don't see how it is supposed to be started, given that nothing pulls it in (i.e. it is not WantedBy anything).

[Ryan Harper (raharper)]

On Mon, Jun 13, 2016 at 8:10 PM, David Kowis <email address hidden> wrote:

> Also, I've grabbed and installed the 5.15-1 .dpkg. systemctl reports
> that it's enabled, but it never starts on boot. Checking right after
> boot yields this:
>
> ```
> root@hylden:~# systemctl status watchdog
> ● watchdog.service - watchdog daemon
> Loaded: loaded (/lib/systemd/system/watchdog.service; enabled; vendor
> preset: enabled)
> Active: inactive (dead)
> root@hylden:~# ps -eaf | grep watchdog
> root 10 2 0 20:01 ? 00:00:00 [watchdog/0]
> root 11 2 0 20:01 ? 00:00:00 [watchdog/1]
> root 978 959 0 20:02 pts/0 00:00:00 grep --color=auto watchdog
> ```
>
>
> ```
> root@hylden:/etc/default# systemctl is-enabled watchdog
> enabled
> ```
>
> I've disabled and re-enabled the init script via `systemctl disable
> watchdog` and `systemctl enable watchdog` but I never get a watchdog
> started on boot. I always have to manually start it.
>
> What am I missing?
>

Here's my path

1. apt-get install watchdog
2. systemctl status watchdog

> systemctl status -l watchdog
> ● watchdog.service - watchdog daemon
> Loaded: loaded (/lib/systemd/system/watchdog.service; static; vendor
> preset: enabled)
> Active: inactive (dead)
>

it's enabled, but not started since we've not booted since the installation
of the package.

3. systemctl start watchdog
4. systemctl status -l watchdog
● watchdog.service - watchdog daemon
   Loaded: loaded (/lib/systemd/system/watchdog.service; static; vendor
preset: enabled)
   Active: active (running) since Tue 2016-06-28 14:07:28 UTC; 1min 37s ago
  Process: 1851 ExecStart=/bin/sh -c [ $run_watchdog != 1 ] || exec
/usr/sbin/watchdog $watchdog_options (
  Process: 1850 ExecStartPre=/bin/sh -c [ -z "${watchdog_module}" ] || [
"${watchdog_module}" = "none" ] |
 Main PID: 1853 (watchdog)
    Tasks: 1
   Memory: 184.0K
      CPU: 23ms
   CGroup: /system.slice/watchdog.service
           └─1853 /usr/sbin/watchdog

Jun 28 14:07:28 x1 systemd[1]: Started watchdog daemon.
Jun 28 14:07:28 x1 watchdog[1853]: starting daemon (5.14):
Jun 28 14:07:28 x1 watchdog[1853]: int=1s realtime=yes sync=no soft=no
mla=0 mem=0
Jun 28 14:07:28 x1 watchdog[1853]: ping: no machine to check
Jun 28 14:07:28 x1 watchdog[1853]: file: no file to check
Jun 28 14:07:28 x1 watchdog[1853]: pidfile: no server process to check
Jun 28 14:07:28 x1 watchdog[1853]: interface: no interface to check
Jun 28 14:07:28 x1 watchdog[1853]: temperature: no sensors to check

[Ryan Harper (raharper)]

On Mon, Jun 13, 2016 at 8:18 PM, David Kowis <email address hidden> wrote:

> Sorry for the noise on this bug! It turns out I'm simply not waiting
> long enough for it to come up. It is being started on boot, and
> everything is working like I expect, just isn't showing up.
>
>
> I do still have the question about when it'll make it into a
> Xenial-updates repo, though.
>

It won't go into Xenial automatically. We'd need to fix a specific bug. I
think that for this bug in particular,
the missing [Install] section in the systemd service file (and the missing
quote) are the two changes that
we'd like backported to 5.14.

For now, in Xenial, if you apply this diff to your
/lib/systemd/system/watchdog.service file

% diff -u watchdog.service-xenial watchdog.service-yakkety
--- watchdog.service-xenial 2016-06-28 09:15:26.794096890 -0500
+++ watchdog.service-yakkety 2016-06-28 09:15:42.938657193 -0500
@@ -12,3 +12,5 @@
 ExecStopPost=/bin/sh -c '[ $run_wd_keepalive != 1 ] || false'

 [Install]
+WantedBy=default.target
+

And then:

systemctl daemon-reload
systemctl enable watchdog

And reboot

You should find watchdog service started at boot time.

If that's sufficient for you, I can start the SRU process for bringing in
the change to the service file in Xenial.

[Alexander E. Patrakov (patrakov-gmail)]

I will test it later today, but - why are you using default.target instead of multi-user.target?

[Alexander E. Patrakov (patrakov-gmail)]

The proposed modification of the service file works. It works both with default.target and with multi-user.target.

[Ryan Harper (raharper)]

On Tue, Jun 28, 2016 at 1:01 PM, Alexander E. Patrakov <
<email address hidden>> wrote:

> The proposed modification of the service file works. It works both with
> default.target and with multi-user.target.
>

Thanks for testing.

The target selection was based on what's upstream in the debian packaging.

I'll pull together the paperwork for SRU'ing that fix into Xenial.

> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1448924
>
> Title:
> watchdogd doesn't start on boot
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/watchdog/+bug/1448924/+subscriptions
>

[Olivier Louvignes (olouvignes)]

Encountered this today on an up-to-date xenial.

Quite scary to see such a critical service not being functional on a up-to-date LTS distro.

[Алексей (f-mail-r)]

+1 is there any chances to get backport for xenial?

[Алексей (f-mail-r)]

Ryan, but I couldnt find this version in repos:
# cat /etc/apt/sources.list
deb http://mirror.yandex.ru/ubuntu xenial main restricted universe multiverse
deb http://mirror.yandex.ru/ubuntu xenial-security main restricted universe multiverse
deb http://mirror.yandex.ru/ubuntu xenial-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-backports main restricted universe multiverse

# apt-get -qq update

# apt-cache policy watchdog
watchdog:
  Installed: 5.14-3ubuntu0.16.04.1
  Candidate: 5.14-3ubuntu0.16.04.1
  Version table:
 *** 5.14-3ubuntu0.16.04.1 500
        500 http://mirror.yandex.ru/ubuntu xenial-updates/universe amd64 Packages
        100 /var/lib/dpkg/status
     5.14-3 500
        500 http://mirror.yandex.ru/ubuntu xenial/universe amd64 Packages

where is it?
Thank you

[Ryan Harper (raharper)]

It's not been released. I've just started the SRU process for getting it
into xenial-updates.

If you want to test, you can try out the package from the PPA:

ppa:raharper/bugfixes

On Thu, May 4, 2017 at 4:27 AM, Алексей <email address hidden> wrote:

> Ryan, but I couldnt find this version in repos:
> # cat /etc/apt/sources.list
> deb http://mirror.yandex.ru/ubuntu xenial main restricted universe
> multiverse
> deb http://mirror.yandex.ru/ubuntu xenial-security main restricted
> universe multiverse
> deb http://mirror.yandex.ru/ubuntu xenial-updates main restricted
> universe multiverse
> deb http://archive.ubuntu.com/ubuntu xenial-backports main restricted
> universe multiverse
>
> # apt-get -qq update
>
> # apt-cache policy watchdog
> watchdog:
> Installed: 5.14-3ubuntu0.16.04.1
> Candidate: 5.14-3ubuntu0.16.04.1
> Version table:
> *** 5.14-3ubuntu0.16.04.1 500
> 500 http://mirror.yandex.ru/ubuntu xenial-updates/universe amd64
> Packages
> 100 /var/lib/dpkg/status
> 5.14-3 500
> 500 http://mirror.yandex.ru/ubuntu xenial/universe amd64 Packages
>
> where is it?
> Thank you
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1448924
>
> Title:
> watchdogd doesn't start on boot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/watchdog/+bug/
> 1448924/+subscriptions
>

[Алексей (f-mail-r)]

okay, is there any ETA for SRU process to be finished?
We can fix this by hands while fix not released, but we dont like this way

[Ryan Harper (raharper)]

Please review the SRU process here.

https://wiki.ubuntu.com/StableReleaseUpdates

When it lands in -proposed, your help in the verification process is most
welcomed.

On Thu, May 4, 2017 at 10:46 AM, Алексей <email address hidden> wrote:

> okay, is there any ETA for SRU process to be finished?
> We can fix this by hands while fix not released, but we dont like this way
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1448924
>
> Title:
> watchdogd doesn't start on boot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/watchdog/+bug/
> 1448924/+subscriptions
>

[lepidas blades rompolos (lepidas)]

I followed post #21 change and solved