Distributed Cloud - NTP config does not propagate properly

[Yang Liu]

Brief Description
-----------------
In Distributed Cloud, NTP configuration propogates before it even applies to system controllers, and the NTP state will not be synced to subclouds if we toggle the config on system controller.
If was also noticed if NTP state was changed, all hosts will be config out-of-date instead of just controllers. I'm unsure about the expected behavior for this one, but for comparison, if only the NTP servers were changed, then only controllers will be config out-of-date.

Severity
--------
Major

Steps to Reproduce
------------------
1. Configure DC to enable NTP with valid NTP servers
2. Central controller: system ntp-modify enabled=False
3. Wait for subcloud sync audit to happen in /var/log/dcmanager/dcmanager.log
4. Central controller: Lock/unlock hosts to apply the change
5. Central controller: system ntp-modify enabled=True
6. Wait for subcloud sync audit to happen in /var/log/dcmanager/dcmanager.log

Expected Behavior
------------------
3. Change does not propagate to subclouds before step4
4. Managed subclouds: only controllers are config out-of-date and needed lock/unlock to clear alarms
6. Managed subclouds: NTP shows enabled in 'system ntp-show'

Actual Behavior
----------------
3. Changes propagated to managed subclouds before NTP config is applied in central region. i.e., on subcloud, config out-of-date alarms appeared and ntp shows disabled in 'system ntp-show',
4. Managed subcloud: ALL hosts config out-of-date and required lock/unlock to clear alarms. (Note that if NTP servers are changed instead of state(enable/disable), then only controllers will be config out-of-date as expected)
6. Managed subclouds: NTP stayed as disabled in 'system ntp-show'. Workaround: system ntp-modify in all subclouds to re-enabled NTP.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Distributed Cloud

Branch/Pull Time/Commit
-----------------------
master as of 2018-10-24_20-18-00

Timestamp/Logs
--------------
System controllers
2018-11-09T14:01:13.000 controller-0 -sh: info HISTORY: PID=147014 UID=1875 system ntp-modify enabled=False
2018-11-09T14:26:56.000 controller-1 -sh: info HISTORY: PID=27275 UID=1875 system ntp-modify enabled=True

[Ghada Khalil (gkhalil)]

Requesting input from Bart

[Bart Wensley (bartwensley)]

The following "actual behaviors" are expected:

3. Changes propagated to managed subclouds before NTP config is applied in central region. i.e., on subcloud, config out-of-date alarms appeared and ntp shows disabled in 'system ntp-show',
Explanation: Changes to the shared config are propagated to subclouds immediately - we do not wait for hosts to be locked/unlocked in the system controller.

4. Managed subcloud: ALL hosts config out-of-date and required lock/unlock to clear alarms. (Note that if NTP servers are changed instead of state(enable/disable), then only controllers will be config out-of-date as expected)
Explanation: When the NTP state is changed from enabled to disabled, all hosts need to be locked/unlocked to safely update their ntp configuration.

The following "actual behavior" is not expected.

6. Managed subclouds: NTP stayed as disabled in 'system ntp-show'. Workaround: system ntp-modify in all subclouds to re-enabled NTP.

If NTP had been re-enabled in the system controller, then this should have been propagated to the subclouds. The NTP should hvae been enabled in the subclouds and all the hosts in the subclouds should have been marked as config out-of-date.

[Ghada Khalil (gkhalil)]

Targeting stx.2019.03 - the ntp state should have propagated to the subclouds

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/625128

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/625128
Committed: https://git.openstack.org/cgit/openstack/stx-distcloud/commit/?id=7fe134264bcb57b8f58588276107a440c7dec63d
Submitter: Zuul
Branch: master

commit 7fe134264bcb57b8f58588276107a440c7dec63d
Author: Alex Kozyrev <email address hidden>
Date: Thu Dec 13 16:09:31 2018 -0500

    Convert NTP/PTP enabled flag to string implicitly in DC sync.

    NTP/PTP enabled flag is not propagated properly to subclouds.
    The root cause is the wrong boolean/string comparison of the enabled
    flag. Need to to convert it to string to simplify the logic.

    Change-Id: Ie9d67c567732caf5edba751fd1310b94d2c084ca
    Closes-Bug: 1802530
    Signed-off-by: Alex Kozyrev <email address hidden>