Patch is updated and test result with https://review.opendev.org/c/starlingx/monitoring/+/787588/4 [IPv4] When ntp server is not selected: controller-1:~$ ntpq -np remote refid st t when poll reach delay offset jitter ============================================================================== 192.168.204.2 .INIT. 16 u - 64 0 0.000 0.000 0.000 206.75.147.25 192.168.10.254 2 u 46 64 1 250.884 17.371 0.000 149.56.121.17 213.251.128.249 2 u 45 64 1 269.301 20.717 0.000 ==> No NTP server is selected. [sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list +----------+------------------------------------------------------------------------+-------------------------------+----------+---------------------+ | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp | +----------+------------------------------------------------------------------------+-------------------------------+----------+---------------------+ | 100.114 | NTP configuration does not contain any valid or reachable NTP servers. | host=controller-1.ntp | major | 2021-04-28T12:30:31 | | | | | | .574720 | | | | | | | | 100.114 | NTP address 149.56.121.17 is not a valid or a reachable NTP server. | host=controller-1.ntp=149.56. | minor | 2021-04-28T12:30:31 | | | | 121.17 | | .531136 | | | | | | | | 100.114 | NTP address 206.75.147.25 is not a valid or a reachable NTP server. | host=controller-1.ntp=206.75. | minor | 2021-04-28T12:30:31 | | | | 147.25 | | .521752 | | | | | | | +----------+------------------------------------------------------------------------+-------------------------------+----------+---------------------+ ==> 1 major alarm and 2 minor alarms as expected 2021-04-28T12:30:31.515 controller-1 collectd[119298]: info NTP query plugin server list: ['0.pool.ntp.org', '1.pool.ntp.org'] 2021-04-28T12:30:31.521 controller-1 collectd[119298]: info NTPQ: 192.168.204.2 .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-28T12:30:31.521 controller-1 collectd[119298]: info NTPQ: 206.75.147.25 192.168.10.254 2 u 24 64 1 250.884 17.371 0.000 2021-04-28T12:30:31.531 controller-1 collectd[119298]: info NTP query plugin raised alarm 100.114:host=controller-1.ntp=206.75.147.25 2021-04-28T12:30:31.531 controller-1 collectd[119298]: info NTP query plugin added '206.75.147.25' to unreachable servers list: ['206.75.147.25'] 2021-04-28T12:30:31.531 controller-1 collectd[119298]: info NTPQ: 149.56.121.17 213.251.128.249 2 u 23 64 1 269.301 20.717 0.000 2021-04-28T12:30:31.574 controller-1 collectd[119298]: info NTP query plugin raised alarm 100.114:host=controller-1.ntp=149.56.121.17 2021-04-28T12:30:31.574 controller-1 collectd[119298]: info NTP query plugin added '149.56.121.17' to unreachable servers list: ['206.75.147.25', '149.56.121.17'] 2021-04-28T12:30:31.574 controller-1 collectd[119298]: info NTP query plugin removing missing '158.69.35.226' server from reachable server list 2021-04-28T12:30:31.574 controller-1 collectd[119298]: info NTP query plugin removing missing '95.216.138.141' server from reachable server list 2021-04-28T12:30:31.574 controller-1 collectd[119298]: info NTP query plugin peer is selected 2021-04-28T12:30:31.618 controller-1 collectd[119298]: info NTP query plugin raised alarm 100.114:host=controller-1.ntp ==> ntpq output is formatted into one line ("info NTPQ:" line) and raised 3 alarm (1 major and 2 minor) as expected After a while npt selected peer as NTP server: controller-1:~$ ntpq -np remote refid st t when poll reach delay offset jitter ============================================================================== *192.168.204.2 205.206.70.42 3 u 27 64 76 0.041 0.585 0.535 +206.75.147.25 192.168.10.254 2 u 12 64 77 213.128 -1.912 15.215 +149.56.121.17 213.251.128.249 2 u 11 64 77 212.247 -5.216 12.694 ==> controller-0 is selected("*" is front in line). Other 2 NTP server are reachable ("+" is front in line) [sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list ==> No alarm found. Which means previous alarm is suppressed. 2021-04-28T12:40:31.513 controller-1 collectd[119298]: info NTP query plugin server list: ['0.pool.ntp.org', '1.pool.ntp.org'] 2021-04-28T12:40:31.531 controller-1 collectd[119298]: info NTPQ: *192.168.204.2 205.206.70.42 3 u 51 64 376 0.041 0.585 1.334 2021-04-28T12:40:31.581 controller-1 collectd[119298]: info NTP query plugin 100.114:host=controller-1.ntp alarm cleared 2021-04-28T12:40:31.581 controller-1 collectd[119298]: info NTP query plugin selected server is peer 2021-04-28T12:40:31.581 controller-1 collectd[119298]: info NTP query plugin peer controller has a reliable source: 205.206.70.42 2021-04-28T12:40:31.581 controller-1 collectd[119298]: info NTPQ: +206.75.147.25 192.168.10.254 2 u 37 64 377 213.128 -1.912 18.800 2021-04-28T12:40:31.669 controller-1 collectd[119298]: info NTP query plugin 100.114:host=controller-1.ntp=206.75.147.25 alarm cleared 2021-04-28T12:40:31.669 controller-1 collectd[119298]: info NTPQ: +149.56.121.17 213.251.128.249 2 u 35 64 377 200.731 -13.335 10.589 2021-04-28T12:40:31.756 controller-1 collectd[119298]: info NTP query plugin 100.114:host=controller-1.ntp=149.56.121.17 alarm cleared 2021-04-28T12:40:31.756 controller-1 collectd[119298]: info NTP query plugin reachable servers: ['206.75.147.25', '149.56.121.17'] ==> Peer is selected ("selected server is peer" above) with reliable source ("peer controller has a reliable source: 205.206.70.42") [IPv6] When ntp server is not selected: controller-1:~$ ntpq -np remote refid st t when poll reach delay offset jitter ============================================================================== face::2 153.24.162.44 2 u 10 64 3 0.214 -4.615 0.256 64:ff9b::45a4:c6c0 128.105.39.11 3 u 17 64 3 59.100 -6.975 0.437 64:ff9b::cc0b:c90a 216.218.254.202 2 u 16 64 3 82.504 -6.921 0.502 64:ff9b::45a4:cbe7 132.163.96.1 2 u 16 64 3 59.092 -6.491 0.166 ==> No NTP server is selected. [sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list +----------+--------------------------------------------------------------------------+------------------------------+----------+---------------------+ | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp | +----------+--------------------------------------------------------------------------+------------------------------+----------+---------------------+ | 100.114 | NTP configuration does not contain any valid or reachable NTP servers. | host=controller-1 | major | 2021-04-27T20:04:53 | | | | | | .534331 | | | | | | | | 100.114 | NTP address 64:ff9b::45a4:cbe7 is not a valid or a reachable NTP server. | host=controller-1=64:ff9b:: | minor | 2021-04-27T20:04:53 | | | | 45a4:cbe7 | | .490646 | | | | | | | | 100.114 | NTP address 64:ff9b::cc0b:c90a is not a valid or a reachable NTP server. | host=controller-1=64:ff9b:: | minor | 2021-04-27T20:04:53 | | | | cc0b:c90a | | .445772 | | | | | | | | 100.114 | NTP address 64:ff9b::45a4:c6c0 is not a valid or a reachable NTP server. | host=controller-1=64:ff9b:: | minor | 2021-04-27T20:04:53 | | | | 45a4:c6c0 | | .400994 | | | | | | | +----------+--------------------------------------------------------------------------+------------------------------+----------+---------------------+ ==> 1 major alarm and 3 minor alarms as expected # /var/log/daemon.log with filter "NTP" 2021-04-27T20:04:53.379 controller-1 collectd[113893]: info NTP query plugin server list: ['0.pool.ntp.org', '1.pool.ntp.org', '3.pool.ntp.org'] 2021-04-27T20:04:53.400 controller-1 collectd[113893]: info NTPQ: face::2 153.24.162.44 2 u 48 64 7 0.189 -5.175 0.450 2021-04-27T20:04:53.400 controller-1 collectd[113893]: info NTPQ: 64:ff9b::45a4:c6c0 128.105.39.11 3 u 52 64 7 59.100 -6.975 0.463 2021-04-27T20:04:53.444 controller-1 collectd[113893]: info NTP query plugin raised alarm 100.114:host=controller-1=64:ff9b::45a4:c6c0 2021-04-27T20:04:53.445 controller-1 collectd[113893]: info NTP query plugin added '64:ff9b::45a4:c6c0' to unreachable servers list: ['64:ff9b::45a4:c6c0'] 2021-04-27T20:04:53.445 controller-1 collectd[113893]: info NTPQ: 64:ff9b::cc0b:c90a 216.218.254.202 2 u 53 64 7 82.238 -7.619 0.513 2021-04-27T20:04:53.489 controller-1 collectd[113893]: info NTP query plugin raised alarm 100.114:host=controller-1=64:ff9b::cc0b:c90a 2021-04-27T20:04:53.489 controller-1 collectd[113893]: info NTP query plugin added '64:ff9b::cc0b:c90a' to unreachable servers list: ['64:ff9b::45a4:c6c0', '64:ff9b::cc0b:c90a'] 2021-04-27T20:04:53.489 controller-1 collectd[113893]: info NTPQ: 64:ff9b::45a4:cbe7 132.163.96.1 2 u 53 64 7 56.238 5.121 11.530 2021-04-27T20:04:53.533 controller-1 collectd[113893]: info NTP query plugin raised alarm 100.114:host=controller-1=64:ff9b::45a4:cbe7 2021-04-27T20:04:53.533 controller-1 collectd[113893]: info NTP query plugin added '64:ff9b::45a4:cbe7' to unreachable servers list: ['64:ff9b::45a4:c6c0', '64:ff9b::cc0b:c90a', '64:ff9b::45a4:cbe7'] 2021-04-27T20:04:53.534 controller-1 collectd[113893]: info NTP query plugin removing missing '64:ff9b::b869:b610' server from reachable server list 2021-04-27T20:04:53.534 controller-1 collectd[113893]: info NTP query plugin removing missing '64:ff9b::c64a:337d' server from reachable server list 2021-04-27T20:04:53.534 controller-1 collectd[113893]: info NTP query plugin no selected server 2021-04-27T20:04:53.576 controller-1 collectd[113893]: info NTP query plugin raised alarm 100.114:host=controller-1 ==> ntpq output is formatted into one line ("info NTPQ:" line) and raised 4 alarm (1 major and 3 minor) as expected After a while npt selected peer as NTP server: controller-1:~$ ntpq -np remote refid st t when poll reach delay offset jitter ============================================================================== *face::2 153.24.162.44 2 u 34 64 17 0.005 -5.383 0.546 +64:ff9b::45a4:c6c0 128.105.39.11 3 u 39 64 17 45.657 -1.006 6.281 +64:ff9b::cc0b:c90a 216.218.254.202 2 u 42 64 17 81.933 -7.501 0.345 +64:ff9b::45a4:cbe7 132.163.96.1 2 u 40 64 17 46.530 -0.567 5.791 ==> controller-0 is selected("*" is front in line). Other 3 NTP server are reachable ("+" is front in line) [sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list ==> No alarm found. Which means previous alarm is suppressed. # /var/log/daemon.log with filter "NTP" 2021-04-27T20:09:53.444 controller-1 collectd[113893]: info NTP query plugin server list: ['0.pool.ntp.org', '1.pool.ntp.org', '3.pool.ntp.org'] 2021-04-27T20:09:53.466 controller-1 collectd[113893]: info NTPQ: *face::2 153.24.162.44 2 u 23 64 377 0.005 -5.383 2.040 2021-04-27T20:09:53.509 controller-1 collectd[113893]: info NTP query plugin 100.114:host=controller-1 alarm cleared 2021-04-27T20:09:53.510 controller-1 collectd[113893]: info NTP query plugin selected server is peer 2021-04-27T20:09:53.510 controller-1 collectd[113893]: info NTP query plugin peer controller has a reliable source: 153.24.162.44 2021-04-27T20:09:53.510 controller-1 collectd[113893]: info NTPQ: +64:ff9b::45a4:c6c0 128.105.39.11 3 u 22 64 377 45.657 -1.006 5.115 2021-04-27T20:09:53.595 controller-1 collectd[113893]: info NTP query plugin 100.114:host=controller-1=64:ff9b::45a4:c6c0 alarm cleared 2021-04-27T20:09:53.595 controller-1 collectd[113893]: info NTPQ: +64:ff9b::cc0b:c90a 216.218.254.202 2 u 26 64 377 68.495 0.507 6.065 2021-04-27T20:09:53.679 controller-1 collectd[113893]: info NTP query plugin 100.114:host=controller-1=64:ff9b::cc0b:c90a alarm cleared 2021-04-27T20:09:53.679 controller-1 collectd[113893]: info NTPQ: +64:ff9b::45a4:cbe7 132.163.96.1 2 u 22 64 377 46.141 2.021 5.959 2021-04-27T20:09:53.764 controller-1 collectd[113893]: info NTP query plugin 100.114:host=controller-1=64:ff9b::45a4:cbe7 alarm cleared 2021-04-27T20:09:53.764 controller-1 collectd[113893]: info NTP query plugin removing missing '64:ff9b::74cb:3741' server from reachable server list 2021-04-27T20:09:53.764 controller-1 collectd[113893]: info NTP query plugin selected server changed from '64:ff9b::c64a:337d' to 'face::2' 2021-04-27T20:09:53.764 controller-1 collectd[113893]: info NTP query plugin reachable servers: ['64:ff9b::45a4:c6c0', '64:ff9b::cc0b:c90a', '64:ff9b::45a4:cbe7'] ==> Peer is selected ("selected server is peer" above) with reliable source ("peer controller has a reliable source: 153.24.162.44")