deployment fails on Ensure system is NTP time synced

Bug #1806521 reported by Alex Schultz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Alex Schultz

Bug Description

With the disabling of the NtpPool configuration (https://review.openstack.org/#/c/620981/), we are only configuring a single ntp source. This can lead to NTP time sync failures if the one server that is picked is not currently available. We need to specify multiple default NtpServer entries in order to prevent the time sync from failing if we are not going to use the NtpPool configuration.

http://logs.openstack.org/41/621341/1/gate/tripleo-ci-centos-7-standalone/0157d94/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2018-12-03_23_21_18

2018-12-03 23:21:18 | TASK [Ensure system is NTP time synced] ****************************************
2018-12-03 23:24:28 | fatal: [centos-7-rax-dfw-0000887853]: FAILED! => {"changed": true, "cmd": ["chronyc", "waitsync", "20"], "delta": "0:03:10.160256", "end": "2018-12-03 23:24:28.396431", "msg": "non-zero return code", "rc": 1, "start": "2018-12-03 23:21:18.236175", "stderr": "", "stderr_lines": [], "stdout": "try: 1, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 2, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 3, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 4, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 5, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 6, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 7, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 8, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 9, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 10, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 11, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 12, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 13, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 14, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 15, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 16, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 17, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 18, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 19, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 20, refid: 00000000, correction: 0.000000000, skew: 0.000", "stdout_lines": ["try: 1, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 2, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 3, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 4, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 5, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 6, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 7, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 8, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 9, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 10, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 11, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 12, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 13, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 14, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 15, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 16, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 17, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 18, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 19, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 20, refid: 00000000, correction: 0.000000000, skew: 0.000"]}

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/621930

Changed in tripleo:
assignee: nobody → Alex Schultz (alex-schultz)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/621930
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=90d3723175deca490e53019dfb10c066863fa7cb
Submitter: Zuul
Branch: master

commit 90d3723175deca490e53019dfb10c066863fa7cb
Author: Alex Schultz <email address hidden>
Date: Mon Dec 3 20:20:49 2018 -0700

    Specify multiple NtpServers by default

    The NtpServer default set now includes multiple pool.ntp.org hosts to
    ensure that the time can be properly synced during the deployment.
    Having only a single timesource can lead to deployment failures if the
    time source is unavailable during the deployment. It is recommended
    that you either set multiple NtpServers or use the NtpPool
    configuration to ensure that enough time sources are available for the
    hosts. Note that the NtpPool configuration is only available when using
    chrony.

    Change-Id: I5b82d77cbf0f2e8c2a59645a72aa533d7d2c86b8
    Closes-Bug: #1806521

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/624189

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (master)

Reviewed: https://review.openstack.org/624189
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=a57fc1f9af5b2f1358487874bb1969ca72486d83
Submitter: Zuul
Branch: master

commit a57fc1f9af5b2f1358487874bb1969ca72486d83
Author: Alex Schultz <email address hidden>
Date: Mon Dec 10 13:26:12 2018 -0700

    Specify multiple NtpServers by default

    Chrony needs to have multiple servers for the initial sync as it won't
    retry a dns lookup when we run waitsync.

    Change-Id: Id0418f5c17402b5c4fdfad350204907b3f39034c
    Closes-Bug: #1806521

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/625347

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (master)

Reviewed: https://review.openstack.org/625347
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=e9c907a9229f6b79a4063facaea9411e30a054f6
Submitter: Zuul
Branch: master

commit e9c907a9229f6b79a4063facaea9411e30a054f6
Author: Alex Schultz <email address hidden>
Date: Fri Dec 14 13:55:32 2018 -0700

    Lift the limit on ntp servers

    We accept multiple servers from the config but were only using the first
    one which can lead to ntp failures. We should be using the complete list
    provided by the end user.

    Change-Id: If2fd70253b8ad949825f2636bce80d5851851b1f
    Closes-Bug: #1806521

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/625614

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/rocky)

Reviewed: https://review.openstack.org/625614
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=e8efd140b15d2552558e976ebf3c6a872af1aafd
Submitter: Zuul
Branch: stable/rocky

commit e8efd140b15d2552558e976ebf3c6a872af1aafd
Author: Alex Schultz <email address hidden>
Date: Fri Dec 14 13:55:32 2018 -0700

    Lift the limit on ntp servers

    We accept multiple servers from the config but were only using the first
    one which can lead to ntp failures. We should be using the complete list
    provided by the end user.

    Change-Id: If2fd70253b8ad949825f2636bce80d5851851b1f
    Closes-Bug: #1806521
    (cherry picked from commit e9c907a9229f6b79a4063facaea9411e30a054f6)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient 11.2.0

This issue was fixed in the openstack/python-tripleoclient 11.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.3.0

This issue was fixed in the openstack/tripleo-heat-templates 10.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient 10.6.1

This issue was fixed in the openstack/python-tripleoclient 10.6.1 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers