dnsmasq is down after too often restarts by cobbler

[Roman Podoliaka]

master-pkg-mos-systest-ubuntu-xenial (e.g. https://packaging-ci.infra.mirantis.net/view/MOS%20master/job/master-pkg-mos-systest-ubuntu-xenial/356/console) is currently failing with the following error:

AssertionError: Task 'deploy' has incorrect status. error != ready, 'All nodes are finished. Failed tasks: Task[setup_repositories/1], Task[setup_repositories/3], Task[setup_repositories/2], Task[setup_repositories/5], Task[setup_repositories/4], Task[setup_repositories/6] Stopping the deployment process!'

This is due to the fact that dnsmasq on the master node is down:

[root@nailgun ~]# systemctl status dnsmasq.service
● dnsmasq.service - DNS caching server.
   Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/dnsmasq.service.d
           └─restart.conf
   Active: failed (Result: start-limit) since Fri 2017-03-03 13:40:22 UTC; 40min ago
  Process: 4348 ExecStart=/usr/sbin/dnsmasq -k (code=exited, status=0/SUCCESS)
 Main PID: 4348 (code=exited, status=0/SUCCESS)

Mar 03 13:40:25 nailgun.test.domain.local systemd[1]: dnsmasq.service failed.
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: start request repeated too quickly for dnsmasq.service
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: Failed to start DNS caching server..
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: dnsmasq.service failed.
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: start request repeated too quickly for dnsmasq.service
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: Failed to start DNS caching server..
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: dnsmasq.service failed.
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: start request repeated too quickly for dnsmasq.service
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: Failed to start DNS caching server..
Mar 03 13:40:26 nailgun.test.domain.local systemd[1]: dnsmasq.service failed.

systemd gives up restarting the dnsmasq after subsequent terminations:

[root@nailgun ~]# grep SIGTERM /var/log/dnsmasq.log
...
Mar 3 13:39:17 dnsmasq[3517]: exiting on receipt of SIGTERM
Mar 3 13:39:17 dnsmasq[3525]: exiting on receipt of SIGTERM
Mar 3 13:39:17 dnsmasq[3545]: exiting on receipt of SIGTERM
Mar 3 13:39:19 dnsmasq[3565]: exiting on receipt of SIGTERM
Mar 3 13:39:20 dnsmasq[3655]: exiting on receipt of SIGTERM
Mar 3 13:40:21 dnsmasq[4001]: exiting on receipt of SIGTERM
Mar 3 13:40:21 dnsmasq[4182]: exiting on receipt of SIGTERM
Mar 3 13:40:21 dnsmasq[4240]: exiting on receipt of SIGTERM
Mar 3 13:40:22 dnsmasq[4281]: exiting on receipt of SIGTERM
Mar 3 13:40:22 dnsmasq[4307]: exiting on receipt of SIGTERM
Mar 3 13:40:22 dnsmasq[4348]: exiting on receipt of SIGTERM

It's not clear why cobbler restarts dnsmasq so often:

[root@nailgun ~]# grep "running: service dnsmasq restart" /var/log/cobbler/cobbler.log
Fri Mar 3 13:39:16 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:16 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:16 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:16 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:17 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:17 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:19 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:20 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:20 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:20 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:20 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:20 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:24 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:26 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:39:38 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:40:21 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:40:21 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:40:21 2017 - INFO | running: service dnsmasq restart
Fri Mar 3 13:40:22 2017 - INFO | running: service dnsmasq restart