tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 failed with NTP error

Bug #1869842 reported by Sandeep Yadav
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Incomplete
Medium
yatin

Bug Description

Job tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 failed with NTP error

https://logserver.rdoproject.org/98/714598/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/08be6c9/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

~~~
2020-03-31 03:32:37 | TASK [AllNodesValidationConfig] ************************************************
2020-03-31 03:32:37 | Tuesday 31 March 2020 03:32:37 +0000 (0:00:00.631) 0:01:37.106 *********
2020-03-31 03:32:37 | [WARNING]: sftp transfer mechanism failed on [192.168.24.8]. Use
2020-03-31 03:32:37 | ANSIBLE_DEBUG=1 to see detailed information
2020-03-31 03:32:37 | [WARNING]: sftp transfer mechanism failed on [192.168.24.27]. Use
2020-03-31 03:32:37 | ANSIBLE_DEBUG=1 to see detailed information
2020-03-31 03:32:37 | [WARNING]: sftp transfer mechanism failed on [192.168.24.16]. Use
2020-03-31 03:32:37 | ANSIBLE_DEBUG=1 to see detailed information
2020-03-31 03:32:37 | [WARNING]: sftp transfer mechanism failed on [192.168.24.14]. Use
2020-03-31 03:32:37 | ANSIBLE_DEBUG=1 to see detailed information
2020-03-31 03:32:38 | fatal: [overcloud-controller-0]: FAILED! => changed=true
2020-03-31 03:32:38 | msg: non-zero return code
2020-03-31 03:32:38 | rc: 1
2020-03-31 03:32:38 | stderr: |-
2020-03-31 03:32:38 | Shared connection to 192.168.24.8 closed.
2020-03-31 03:32:38 | stderr_lines: <omitted>
2020-03-31 03:32:38 | stdout: |-
2020-03-31 03:32:38 | Trying to ping default gateway 10.0.0.1...Ping to 10.0.0.1 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Trying to ping 192.168.24.8 for local network 192.168.24.0/24.
2020-03-31 03:32:38 | Ping to 192.168.24.8 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Failed to start Hiera: RuntimeError: Config file /etc/puppetlabs/puppet/hiera.yaml not found
2020-03-31 03:32:38 | Testing NTP...FAILURE
2020-03-31 03:32:38 | stdout_lines: <omitted>
2020-03-31 03:32:38 | fatal: [overcloud-controller-2]: FAILED! => changed=true
2020-03-31 03:32:38 | msg: non-zero return code
2020-03-31 03:32:38 | rc: 1
2020-03-31 03:32:38 | stderr: |-
2020-03-31 03:32:38 | Shared connection to 192.168.24.16 closed.
2020-03-31 03:32:38 | stderr_lines: <omitted>
2020-03-31 03:32:38 | stdout: |-
2020-03-31 03:32:38 | Trying to ping default gateway 10.0.0.1...Ping to 10.0.0.1 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Trying to ping 192.168.24.8 for local network 192.168.24.0/24.
2020-03-31 03:32:38 | Ping to 192.168.24.8 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Failed to start Hiera: RuntimeError: Config file /etc/puppetlabs/puppet/hiera.yaml not found
2020-03-31 03:32:38 | Testing NTP...FAILURE
2020-03-31 03:32:38 | stdout_lines: <omitted>
2020-03-31 03:32:38 | fatal: [overcloud-novacompute-0]: FAILED! => changed=true
2020-03-31 03:32:38 | msg: non-zero return code
2020-03-31 03:32:38 | rc: 1
2020-03-31 03:32:38 | stderr: |-
2020-03-31 03:32:38 | Shared connection to 192.168.24.14 closed.
2020-03-31 03:32:38 | stderr_lines: <omitted>
2020-03-31 03:32:38 | stdout: |-
2020-03-31 03:32:38 | Trying to ping default gateway 192.168.24.1...Ping to 192.168.24.1 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Trying to ping 192.168.24.14 for local network 192.168.24.0/24.
2020-03-31 03:32:38 | Ping to 192.168.24.14 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Failed to start Hiera: RuntimeError: Config file /etc/puppetlabs/puppet/hiera.yaml not found
2020-03-31 03:32:38 | Testing NTP...FAILURE
2020-03-31 03:32:38 | stdout_lines: <omitted>
2020-03-31 03:32:38 | fatal: [overcloud-controller-1]: FAILED! => changed=true
2020-03-31 03:32:38 | msg: non-zero return code
2020-03-31 03:32:38 | rc: 1
2020-03-31 03:32:38 | stderr: |-
2020-03-31 03:32:38 | Shared connection to 192.168.24.27 closed.
2020-03-31 03:32:38 | stderr_lines: <omitted>
2020-03-31 03:32:38 | stdout: |-
2020-03-31 03:32:38 | Trying to ping default gateway 10.0.0.1...Ping to 10.0.0.1 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Trying to ping 192.168.24.8 for local network 192.168.24.0/24.
2020-03-31 03:32:38 | Ping to 192.168.24.8 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Failed to start Hiera: RuntimeError: Config file /etc/puppetlabs/puppet/hiera.yaml not found
2020-03-31 03:32:38 | Testing NTP...FAILURE
2020-03-31 03:32:38 | stdout_lines: <omitted>
~~~

https://logserver.rdoproject.org/98/714598/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/08be6c9/logs/overcloud-controller-0/var/log/dnf.log.txt.gz

~~~
2020-03-31T02:18:06Z INFO Downloading Packages:
2020-03-31T02:18:06Z DEBUG error: Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=centos (https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=centos).
2020-03-31T02:18:06Z DDEBUG Cleaning up.
2020-03-31T02:18:06Z INFO The downloaded packages were saved in cache until the next successful transaction.
2020-03-31T02:18:06Z INFO You can remove cached packages by executing 'dnf clean packages'.
2020-03-31T02:18:06Z SUBDEBUG
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 224, in do_transaction
    self.download_packages(install_pkgs, self.output.progress, total_cb)
  File "/usr/lib/python3.6/site-packages/dnf/base.py", line 1159, in download_packages
    self._download_remote_payloads(payloads, drpm, progress, callback_total)
  File "/usr/lib/python3.6/site-packages/dnf/base.py", line 1089, in _download_remote_payloads
    raise dnf.exceptions.DownloadError(errors._irrecoverable)
dnf.exceptions.DownloadError: Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=centos

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 65, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 98, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 122, in cli_run
    ret = resolving(cli, base)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 166, in resolving
    base.do_transaction(display=displays)
  File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 231, in do_transaction
    raise dnf.exceptions.Error(errstr)
dnf.exceptions.Error: Error downloading packages:
  Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=centos
2020-03-31T02:18:06Z CRITICAL Error: Error downloading packages:
  Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=centos
.
.
2020-03-31T03:34:22Z SUBDEBUG
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 65, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 94, in _main
    cli.configure(list(map(ucd, args)), option_parser())
  File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 964, in configure
    self._configure_repos(opts)
  File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 769, in _configure_repos
    raise dnf.exceptions.RepoError(msg % repo)
dnf.exceptions.RepoError: Unknown repo: 'delorean-*-deps-a'
2020-03-31T03:34:22Z CRITICAL Error: Unknown repo: 'delorean-*-deps-a'
~~~

yatin (yatinkarel)
Changed in tripleo:
milestone: none → ussuri-3
summary: - r_1comp-featureset001 failed with NTP error
+ tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 failed with NTP error
Revision history for this message
Sandeep Yadav (sandeepyadav93) wrote :

The above ovb job failure seems related to mirror issue.

- Issue faced during overcloud image modify[0].
- Repo setup script was executed but some task failed[1] during repo setup.
- Task[2] (Install mock package) didn't finished
- Future tasks for disabling epel and removing epel didn't ran.
- Later puppet hiera from epel got installed and caused that NTP Error[3], because block[4] was not executed due to puppet error so it reported ntp failure.

Expectations/ Possible Improvements:-

* It was expected that repo_setup.sh should fail if it failed to install some packages and all-nodes.sh[3] shouldn't have been executed.

* epel repo removal should not be skipped.

* Using infra mirrors instead of fedora mirror should improve the situation.

[0] https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/modify-image/tasks/manual.yml#L58-L59

[1]

https://logserver.rdoproject.org/98/714598/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/08be6c9/logs/undercloud/home/zuul/repo_setup.sh.1585621037.log.txt.gz

 ~~~
 2020-03-31 02:18:06 | Error: Error downloading packages:
 2020-03-31 02:18:06 | Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=cento
 ~~~

More details here:-

* https://logserver.rdoproject.org/98/714598/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/08be6c9/logs/overcloud-controller-0/var/log/dnf.log.txt.gz

[2] https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-8/master.yml#L39

[3] https://logserver.rdoproject.org/98/714598/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/08be6c9/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

~~~
2020-03-31 03:32:38 | fatal: [overcloud-controller-0]: FAILED! => changed=true
2020-03-31 03:32:38 | msg: non-zero return code
2020-03-31 03:32:38 | rc: 1
2020-03-31 03:32:38 | stderr: |-
2020-03-31 03:32:38 | Shared connection to 192.168.24.8 closed.
2020-03-31 03:32:38 | stderr_lines: <omitted>
2020-03-31 03:32:38 | stdout: |-
2020-03-31 03:32:38 | Trying to ping default gateway 10.0.0.1...Ping to 10.0.0.1 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Trying to ping 192.168.24.8 for local network 192.168.24.0/24.
2020-03-31 03:32:38 | Ping to 192.168.24.8 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Failed to start Hiera: RuntimeError: Config file /etc/puppetlabs/puppet/hiera.yaml not found
~~~

[4] https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/validation-scripts/all-nodes.sh#L107

Revision history for this message
Alex Schultz (alex-schultz) wrote :

2020-03-31 03:32:38 | Failed to start Hiera: RuntimeError: Config file /etc/puppetlabs/puppet/hiera.yaml not found

This indicates a bad version of puppet was likely installed (e.g from epel)

Revision history for this message
Alex Schultz (alex-schultz) wrote :

Also epel is in the logs, please remove epel prior to installing tripleo.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-quickstart (master)

Fix proposed to branch: master
Review: https://review.opendev.org/716373

Changed in tripleo:
assignee: nobody → wes hayutin (weshayutin)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-quickstart (master)

Reviewed: https://review.opendev.org/716373
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=4fd8b1efb4015fcfef65041f145a3cc0050be616
Submitter: Zuul
Branch: master

commit 4fd8b1efb4015fcfef65041f145a3cc0050be616
Author: Wes Hayutin <email address hidden>
Date: Tue Mar 31 11:41:22 2020 -0600

    epel is NOT allowed in tripleo, hack for mock

    working to get mock and deps avail in
    dlrn-deps. Until then NUKE epel after
    it's purpose has been served.

    Closes-Bug: #1869842
    Change-Id: I5621fdc9e6353ec94f41a37cbc626f08fe9af372

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
Alfredo Moralejo (amoralej) wrote :

Created trello card in RDO https://trello.com/c/OmeqSzC8/734-use-mock-from-non-epel-repo to follow-up.

Changed in tripleo:
assignee: wes hayutin (weshayutin) → Sandeep Yadav (sandeepyadav93)
status: Fix Released → Triaged
Revision history for this message
Sandeep Yadav (sandeepyadav93) wrote :

Hi,

In this case, actual issue was transient and hit due to deployment moved forward even if repo setup failed in image build.

I will try to work further on one of the pending possible improvements:-

* It was expected all-nodes.sh[0] shouldn't have been executed if some package install fails in repo_setup.sh step. Like it did in eg: [1].

[0] https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/validation-scripts/all-nodes.sh#L107

[1]

https://logserver.rdoproject.org/98/714598/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/08be6c9/logs/undercloud/home/zuul/repo_setup.sh.1585621037.log.txt.gz

 ~~~
 2020-03-31 02:18:06 | Error: Error downloading packages:
 2020-03-31 02:18:06 | Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=cento
 ~~~

wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-3 → ussuri-rc3
Changed in tripleo:
assignee: Sandeep Yadav (sandeepyadav93) → yatin (yatinkarel)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/719542
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=7c5adf712259b4c4e9eb72f12034d4b32875dfd0
Submitter: Zuul
Branch: master

commit 7c5adf712259b4c4e9eb72f12034d4b32875dfd0
Author: yatinkarel <email address hidden>
Date: Mon Apr 13 20:20:14 2020 +0530

    Do not install mock from epel repo

    mock package from epel is added in RDO CentOS8 ussuri build-deps
    repo with [1], so let's use that instead of epel to avoid
    epel related issues.

    [1] https://review.rdoproject.org/r/#/q/topic:rdo-centos8-remove-epel

    Partial-Bug: #1869842
    Change-Id: Ifcb1c5e64ed0005f760aef8435a980d5bdefb193

wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc3 → victoria-1
Changed in tripleo:
milestone: victoria-1 → victoria-3
Changed in tripleo:
milestone: victoria-3 → wallaby-1
Changed in tripleo:
milestone: wallaby-1 → wallaby-2
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Changed in tripleo:
milestone: wallaby-3 → wallaby-rc1
Changed in tripleo:
milestone: wallaby-rc1 → xena-1
Revision history for this message
Marios Andreou (marios-b) wrote :

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

Changed in tripleo:
milestone: xena-1 → none
status: In Progress → Incomplete
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.