[2.3, service-tracking] MAAS service tracking never notices tracked daemons crash (ntp is an example) or viceversa

[Jason Hobbs]

If ntp crashes on a region controller, for example, via SIGSEGV, MAAS never seems to notice. The service indicator in the UI stays green for ntp on that controller, MAAS continues to tell nodes to use the VIP for ntp, even though it may not work.

To reproduce:

1) kill -SIGSEGV <pidof ntpd> on a MAAS region controller.
2) verify that the controller's status never changes and ntp continues to be up.
3) if using HA, verify that crm status shows all units up for the MAAS vip.

This is with 2.3.0-6434-gd354690-0ubuntu1~16.04.1.

[Andres Rodriguez (andreserl)]

How long did you wait to make the determination that the service crash wasn't noticed by MAAS?

Also, did you confirm that system *never* tried to start the service again ?

[Blake Rouse (blake-rouse)]

I believe there is a 30 second delay.

[Jason Hobbs (jason-hobbs)]

Ok, I waited 19 minutes, which isn't never but it's never-ish.

http://paste.ubuntu.com/26525461/

maas.service_monitor: does recognize it's not running, but it doesn't restart it, and it doesn't report any thing to MAAS.

[Blake Rouse (blake-rouse)]

19 minutes seems long enough ;-)

[Andres Rodriguez (andreserl)]

I can confirm this is also the other way around.

e.g. a had a broken bind, i fixed it, restart it manually, systemd reported it was running fine, but MAAS never updated that in the UI.

[Blake Rouse (blake-rouse)]

I have been trying to reproduce this with latest master that includes https://code.launchpad.net/~blake-rouse/maas/+git/maas/+merge/342811 which I believe was the actual cause of this issue.

Also while reproducing this issue we noticed that the 1 minute delay is to long to wait to restart services so we are lowing them to 1.