Comment 2 for bug 1665426

Ok so I think I figured out who is constantly changing /etc/ntp.conf.
It is os-apply-config. It keeps pushing the configuration at /usr/libexec/os-apply-config/templates/etc/ntp.conf, then puppet will run and will notice a different ntp.conf and will do its usual config regeneration + restart.

So we see this in the journal for os-collect-config:
Feb 15 16:51:51 overcloud-controller-0 os-collect-config[1939]: [2017/02/15 04:51:51 PM] [INFO] writing /etc/ntp.conf
Feb 15 16:51:51 overcloud-controller-0 os-collect-config[1939]: dib-run-parts Wed Feb 15 16:51:51 UTC 2017 Running /usr/libexec/os-refresh-config/configure.d/50-ntp
Feb 15 16:51:52 overcloud-controller-0 os-collect-config[1939]: dib-run-parts Wed Feb 15 16:51:52 UTC 2017 50-ntp completed
Feb 15 16:51:55 overcloud-controller-0 os-collect-config[1939]: dib-run-parts Wed Feb 15 16:51:55 UTC 2017 50-ntp 0.315