ntpd on undercloud is not configured

[Sergii Golovatiuk]

Description
===========
ntpd is not configured on undercloud

ntpd.conf on undercloud looks like
---%<---
[stack@undercloud ~]$ cat /etc/ntp.conf
# ntp.conf: Managed by puppet.
#
# Enable next tinker options:
# panic - keep ntpd from panicking in the event of a large clock skew
# when a VM guest is suspended and resumed;
# stepout - allow ntpd change offset faster
tinker panic 0

disable monitor

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1

# Set up servers for ntpd with next options:
# server - IP address or DNS name of upstream NTP server
# iburst - allow send sync packages faster if upstream unavailable
# prefer - select preferrable server
# minpoll - set minimal update frequency
# maxpoll - set maximal update frequency

# Driftfile.
driftfile /var/lib/ntp/drift

---%<---

ntpd is not working.

[root@undercloud ~]# ntpq
ntpq> peers
ntpq: read: Connection refused
ntpq>

Steps to reproduce
==================
Install Undercloud, check ntpd

Environment
===========
Just undercloud

[Emilien Macchi (emilienm)]

The NTP configuration on the containerized undercloud and non containerized undercloud is the same:

http://logs.openstack.org/07/571307/1/gate/tripleo-ci-centos-7-undercloud-oooq/adc122a/logs/undercloud/etc/ntp.conf.txt.gz
http://logs.openstack.org/12/575712/2/check/tripleo-ci-centos-7-undercloud-containers/031191f/logs/undercloud/etc/ntp.conf.txt.gz

So if there is a problem, it's on both containerized and non-containerized undercloud.

[Emilien Macchi (emilienm)]

I just deployed a fresh containerized undercloud and NTP works fine:

https://asciinema.org/a/k3AkDOTneOhz5uhr6qwksLJFH

[Emilien Macchi (emilienm)]

Closing the bug as I couldn't reproduce. I'm happy to re-open if you provide a way to reproduce.

[Sergii Golovatiuk (sgolovatiuk)]

That's not correct. I've just installed fresh master by quickstart where I have

extra_args: >-
  --libvirt-type qemu
  --ntp-server pool.ntp.org

I looked at ntp.conf and it was created as shown in #1. However, I found that undercloud was not containerized. I am trying to reproduce the same behaviour on containerized undercloud.

[Sergii Golovatiuk (sgolovatiuk)]

Here is a /etc/ntp.conf. when we install containerized undercloud

---%<---
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

#broadcast 192.168.1.255 autokey # broadcast server
#broadcastclient # broadcast client
#broadcast 224.0.1.1 autokey # multicast server
#multicastclient 224.0.1.1 # multicast client
#manycastserver 239.255.254.254 # manycast server
#manycastclient 239.255.254.254 autokey # manycast client

# Enable public key cryptography.
#crypto

includefile /etc/ntp/crypto/pw

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8

# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats

# Disable the monitoring facility to prevent amplification attacks using ntpdc
# monlist command when default restrict does not include the noquery flag. See
# CVE-2013-5211 for more details.
# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitor
---%<---

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/576181

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/576450

[OpenStack Infra (hudson-openstack)]

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.openstack.org/576181
Reason: failed in the gate, clearing

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/576450

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/576181
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=d49442152a3318e5afcc86bb8af6e58a74b23878
Submitter: Zuul
Branch: master

commit d49442152a3318e5afcc86bb8af6e58a74b23878
Author: Alex Schultz <email address hidden>
Date: Mon Jun 18 09:40:50 2018 -0600

    Fall back to puppet-ntp defaults

    If the ntp server is not configured in the undercloud.conf, we should
    fall back to the defaults provided by puppet-ntp otherwise we end up
    with an invalid ntp configuration.

    Change-Id: I0000e1cf736b513dbc58c0d39f7e1c0137b660dd
    Closes-BUg: #1777140

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/576450
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=cc05b49e5e471f2a1c8647507ecfd71e12c24ad2
Submitter: Zuul
Branch: stable/queens

commit cc05b49e5e471f2a1c8647507ecfd71e12c24ad2
Author: Alex Schultz <email address hidden>
Date: Mon Jun 18 09:40:50 2018 -0600

    Fall back to puppet-ntp defaults

    If the ntp server is not configured in the undercloud.conf, we should
    fall back to the defaults provided by puppet-ntp otherwise we end up
    with an invalid ntp configuration.

    Change-Id: I0000e1cf736b513dbc58c0d39f7e1c0137b660dd
    Closes-BUg: #1777140
    (cherry picked from commit d49442152a3318e5afcc86bb8af6e58a74b23878)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/577254

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/577254
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=cdd28f8c2ed60beb8780d97b3fb38a89836d27ba
Submitter: Zuul
Branch: stable/pike

commit cdd28f8c2ed60beb8780d97b3fb38a89836d27ba
Author: Alex Schultz <email address hidden>
Date: Mon Jun 18 09:40:50 2018 -0600

    Fall back to puppet-ntp defaults

    If the ntp server is not configured in the undercloud.conf, we should
    fall back to the defaults provided by puppet-ntp otherwise we end up
    with an invalid ntp configuration.

    Change-Id: I0000e1cf736b513dbc58c0d39f7e1c0137b660dd
    Closes-BUg: #1777140
    (cherry picked from commit d49442152a3318e5afcc86bb8af6e58a74b23878)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/instack-undercloud 8.4.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/instack-undercloud 7.4.13 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/instack-undercloud 9.2.0 release.

[David Hill (david-hill-ubisoft)]

If we manually configured a ntpd server for the overcloud nodes on the undercloud, this revert the manual changes . I've spent some time trying to figureout what was happening in my undercloud because I use it as a ntp server.

[Sudeep Batra (sudeep.batra)]

I tried installing the Rocky release ,followed the below steps :
    1 export VIRTHOST=127.0.0.2
    2 git clone https://github.com/openstack/tripleo-quickstart.git
    3 cd tripleo-quickstart/
    4 bash quickstart.sh --install-deps
    5 ssh-keygen
    6 ssh-copy-id root@$VIRTHOST # otherwise installation fails #
    7 bash quickstart.sh -R rocky --no-clone --tags all --nodes config/nodes/1ctlr_1comp_1ceph.yml -I $VIRTHOST
Error still for NTP in the undercloud_install.log
TASK [Ensure system is NTP time synced] ****************************************
fatal: [undercloud]: FAILED! ntpdate[9153]: no server suitable for synchronization found",etc..

so the manual steps are still needed.

[Alex Schultz (alex-schultz)]

Sudeep, that means the ntp servers provided (they can be configured in undercloud.conf) are not reachable. You must have available NTP servers int order to install. By default it'll use pool.ntp.org (or related hosts). If NTP is blocked on your network, you need to provide an internal time source.

[Sudeep Batra (sudeep.batra)]

Thanks, so when I run ntpdate on undercloud it works.
[stack@undercloud ~]$ sudo ntpdate USMIIWEGAD00001.ericsson.se
 9 Jan 23:09:09 ntpdate[9560]: step time server 147.117.251.28 offset -92.814406 sec

But the install undercloud still fails.. log attached.

[Sudeep Batra (sudeep.batra)]

Thanks, so when I run ntpdate on undercloud it works.
[stack@undercloud ~]$ sudo ntpdate 147.117.251.28
 9 Jan 23:17:57 ntpdate[9584]: adjust time server 147.117.251.28 offset 0.030118 sec

But the install undercloud still fails.. log attached.

[Alex Schultz (alex-schultz)]

Yes, the undercloud_ntp_servers is still not set correctly. From the logs: ["ntpdate", "-u", "pool.ntp.org"]

[Sudeep Batra (sudeep.batra)]

Thanks Alex,
used the below steps to deploy TripleO using Rocky >
export RELEASE=rocky
export VIRTHOST=127.0.0.2
time bash quickstart.sh \
      --teardown all \
      --release $RELEASE \
      --nodes /home/user/tripleo-quickstart/nodes.yaml \
      --config /home/user/tripleo-quickstart/config.yaml \
      -e undercloud_undercloud_ntp_servers=147.117.251.28 \
      --clean \
      $VIRTHOST

But it does not install the overcloud as well.
How to install the overcloud also automated using quickstart.

[Bogdan Dobrelya (bogdando)]

AFAIK, quickstart can only install an undercloud for you.
The overcloud installation requires some manual preparations, and there is also a flavors matching issue https://bugs.launchpad.net/tripleo/+bug/1788875 ... Or you can try installing a standalone all in one cloud once you have your undercloud operational, see https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/standalone.html.