Kayobe reports "no host ntp daemon is running" incorrectly

[Martin Ananda Boeker]

Env: 2023.1, Ubuntu 22.04

About 1/3 times when I run `kayobe overcloud service deploy` on my group of 42 systems, one or two of them fail with this error:

fatal: [OMC3-RG4-03-SRV-05]: FAILED! => changed=false
  msg: 'No host NTP daemon is running. Please install and configure a host NTP daemon. Alternatively, set ''prechecks_enable_host_ntp_checks'' to ''false'' to disable this check if not using one of the following NTP daemons: chrony, ntpd, systemd-timesyncd.'

However, whenever I see this message I always check the server(s) where it is reported, and they are always fine:

(kayobe) ubuntu@kayobevm:~$ ssh OMC3-RG4-03-SRV-05 "chronyc sources"
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^+ OMC3-RG4-01-CPL-01 6 6 377 12 +49us[ +49us] +/- 28ms
^* OMC3-RG4-02-CPL-01 6 6 377 20 +106us[ +161us] +/- 28ms
^+ OMC3-RG4-03-CPL-01 6 6 377 11 +71us[ +71us] +/- 28ms

The blocks that test this are here:

    - name: Check for a running host NTP daemon # noqa command-instead-of-module
      vars:
        prechecks_host_ntp_daemons:
          - chrony
          - chronyd
          - ntp
          - ntpd
          - systemd-timesyncd
      become: true
      command:
        cmd: "systemctl is-active {{ prechecks_host_ntp_daemons | join(' ') }}"
      register: systemctl_is_active
      changed_when: false
      failed_when: false
      check_mode: false

    - name: Fail if a host NTP daemon is not running
      fail:
        msg: >-
          No host NTP daemon is running.
          Please install and configure a host NTP daemon.
          Alternatively, set 'prechecks_enable_host_ntp_checks' to 'false' to
          disable this check if not using one of the following NTP daemons:
          chrony, ntpd, systemd-timesyncd.
      when:
        - systemctl_is_active.rc != 0

I don't know what exactly makes the output unhappy, but it's very sporadic.

[Martin Ananda Boeker (mboeker)]

It's not any particular host repeatedly, the last two times I saw this error were each about different systems, but all of them were synchronized down to < ~100us.

[Martin Ananda Boeker (mboeker)]

I just tried the command twice in a row, and got different results:

(kayobe) ubuntu@kayobevm:~$ ssh OMC3-RG4-03-SRV-05 "sudo systemctl is-active chrony chronyd ntp ntpd sytemd-timesyncd; echo $?"
active
active
inactive
inactive
inactive
3
(kayobe) ubuntu@kayobevm:~$ ssh OMC3-RG4-03-SRV-05 "sudo systemctl is-active chrony chronyd ntp ntpd sytemd-timesyncd; echo $?"
active
active
inactive
inactive
inactive
0

Perhaps the check should be something more definitive?