Job tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 failed with NTP error
https://logserver.rdoproject.org/98/714598/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/08be6c9/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
~~~
2020-03-31 03:32:37 | TASK [AllNodesValidationConfig] ************************************************
2020-03-31 03:32:37 | Tuesday 31 March 2020 03:32:37 +0000 (0:00:00.631) 0:01:37.106 *********
2020-03-31 03:32:37 | [WARNING]: sftp transfer mechanism failed on [192.168.24.8]. Use
2020-03-31 03:32:37 | ANSIBLE_DEBUG=1 to see detailed information
2020-03-31 03:32:37 | [WARNING]: sftp transfer mechanism failed on [192.168.24.27]. Use
2020-03-31 03:32:37 | ANSIBLE_DEBUG=1 to see detailed information
2020-03-31 03:32:37 | [WARNING]: sftp transfer mechanism failed on [192.168.24.16]. Use
2020-03-31 03:32:37 | ANSIBLE_DEBUG=1 to see detailed information
2020-03-31 03:32:37 | [WARNING]: sftp transfer mechanism failed on [192.168.24.14]. Use
2020-03-31 03:32:37 | ANSIBLE_DEBUG=1 to see detailed information
2020-03-31 03:32:38 | fatal: [overcloud-controller-0]: FAILED! => changed=true
2020-03-31 03:32:38 | msg: non-zero return code
2020-03-31 03:32:38 | rc: 1
2020-03-31 03:32:38 | stderr: |-
2020-03-31 03:32:38 | Shared connection to 192.168.24.8 closed.
2020-03-31 03:32:38 | stderr_lines: <omitted>
2020-03-31 03:32:38 | stdout: |-
2020-03-31 03:32:38 | Trying to ping default gateway 10.0.0.1...Ping to 10.0.0.1 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Trying to ping 192.168.24.8 for local network 192.168.24.0/24.
2020-03-31 03:32:38 | Ping to 192.168.24.8 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Failed to start Hiera: RuntimeError: Config file /etc/puppetlabs/puppet/hiera.yaml not found
2020-03-31 03:32:38 | Testing NTP...FAILURE
2020-03-31 03:32:38 | stdout_lines: <omitted>
2020-03-31 03:32:38 | fatal: [overcloud-controller-2]: FAILED! => changed=true
2020-03-31 03:32:38 | msg: non-zero return code
2020-03-31 03:32:38 | rc: 1
2020-03-31 03:32:38 | stderr: |-
2020-03-31 03:32:38 | Shared connection to 192.168.24.16 closed.
2020-03-31 03:32:38 | stderr_lines: <omitted>
2020-03-31 03:32:38 | stdout: |-
2020-03-31 03:32:38 | Trying to ping default gateway 10.0.0.1...Ping to 10.0.0.1 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Trying to ping 192.168.24.8 for local network 192.168.24.0/24.
2020-03-31 03:32:38 | Ping to 192.168.24.8 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Failed to start Hiera: RuntimeError: Config file /etc/puppetlabs/puppet/hiera.yaml not found
2020-03-31 03:32:38 | Testing NTP...FAILURE
2020-03-31 03:32:38 | stdout_lines: <omitted>
2020-03-31 03:32:38 | fatal: [overcloud-novacompute-0]: FAILED! => changed=true
2020-03-31 03:32:38 | msg: non-zero return code
2020-03-31 03:32:38 | rc: 1
2020-03-31 03:32:38 | stderr: |-
2020-03-31 03:32:38 | Shared connection to 192.168.24.14 closed.
2020-03-31 03:32:38 | stderr_lines: <omitted>
2020-03-31 03:32:38 | stdout: |-
2020-03-31 03:32:38 | Trying to ping default gateway 192.168.24.1...Ping to 192.168.24.1 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Trying to ping 192.168.24.14 for local network 192.168.24.0/24.
2020-03-31 03:32:38 | Ping to 192.168.24.14 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Failed to start Hiera: RuntimeError: Config file /etc/puppetlabs/puppet/hiera.yaml not found
2020-03-31 03:32:38 | Testing NTP...FAILURE
2020-03-31 03:32:38 | stdout_lines: <omitted>
2020-03-31 03:32:38 | fatal: [overcloud-controller-1]: FAILED! => changed=true
2020-03-31 03:32:38 | msg: non-zero return code
2020-03-31 03:32:38 | rc: 1
2020-03-31 03:32:38 | stderr: |-
2020-03-31 03:32:38 | Shared connection to 192.168.24.27 closed.
2020-03-31 03:32:38 | stderr_lines: <omitted>
2020-03-31 03:32:38 | stdout: |-
2020-03-31 03:32:38 | Trying to ping default gateway 10.0.0.1...Ping to 10.0.0.1 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Trying to ping 192.168.24.8 for local network 192.168.24.0/24.
2020-03-31 03:32:38 | Ping to 192.168.24.8 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Failed to start Hiera: RuntimeError: Config file /etc/puppetlabs/puppet/hiera.yaml not found
2020-03-31 03:32:38 | Testing NTP...FAILURE
2020-03-31 03:32:38 | stdout_lines: <omitted>
~~~
https://logserver.rdoproject.org/98/714598/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/08be6c9/logs/overcloud-controller-0/var/log/dnf.log.txt.gz
~~~
2020-03-31T02:18:06Z INFO Downloading Packages:
2020-03-31T02:18:06Z DEBUG error: Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=centos (https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=centos).
2020-03-31T02:18:06Z DDEBUG Cleaning up.
2020-03-31T02:18:06Z INFO The downloaded packages were saved in cache until the next successful transaction.
2020-03-31T02:18:06Z INFO You can remove cached packages by executing 'dnf clean packages'.
2020-03-31T02:18:06Z SUBDEBUG
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 224, in do_transaction
self.download_packages(install_pkgs, self.output.progress, total_cb)
File "/usr/lib/python3.6/site-packages/dnf/base.py", line 1159, in download_packages
self._download_remote_payloads(payloads, drpm, progress, callback_total)
File "/usr/lib/python3.6/site-packages/dnf/base.py", line 1089, in _download_remote_payloads
raise dnf.exceptions.DownloadError(errors._irrecoverable)
dnf.exceptions.DownloadError: Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=centos
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 65, in main
return _main(base, args, cli_class, option_parser_class)
File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 98, in _main
return cli_run(cli, base)
File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 122, in cli_run
ret = resolving(cli, base)
File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 166, in resolving
base.do_transaction(display=displays)
File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 231, in do_transaction
raise dnf.exceptions.Error(errstr)
dnf.exceptions.Error: Error downloading packages:
Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=centos
2020-03-31T02:18:06Z CRITICAL Error: Error downloading packages:
Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=centos
.
.
2020-03-31T03:34:22Z SUBDEBUG
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 65, in main
return _main(base, args, cli_class, option_parser_class)
File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 94, in _main
cli.configure(list(map(ucd, args)), option_parser())
File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 964, in configure
self._configure_repos(opts)
File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 769, in _configure_repos
raise dnf.exceptions.RepoError(msg % repo)
dnf.exceptions.RepoError: Unknown repo: 'delorean-*-deps-a'
2020-03-31T03:34:22Z CRITICAL Error: Unknown repo: 'delorean-*-deps-a'
~~~
The above ovb job failure seems related to mirror issue.
- Issue faced during overcloud image modify[0].
- Repo setup script was executed but some task failed[1] during repo setup.
- Task[2] (Install mock package) didn't finished
- Future tasks for disabling epel and removing epel didn't ran.
- Later puppet hiera from epel got installed and caused that NTP Error[3], because block[4] was not executed due to puppet error so it reported ntp failure.
Expectations/ Possible Improvements:-
* It was expected that repo_setup.sh should fail if it failed to install some packages and all-nodes.sh[3] shouldn't have been executed.
* epel repo removal should not be skipped.
* Using infra mirrors instead of fedora mirror should improve the situation.
[0] https:/ /opendev. org/openstack/ tripleo- quickstart- extras/ src/branch/ master/ roles/modify- image/tasks/ manual. yml#L58- L59
[1]
https:/ /logserver. rdoproject. org/98/ 714598/ 2/openstack- check/tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t001/08be6c9/ logs/undercloud /home/zuul/ repo_setup. sh.1585621037. log.txt. gz
~~~ /mirrors. fedoraproject. org/metalink? repo=epel- 8&arch= x86_64& infra=genclo& content= cento
2020-03-31 02:18:06 | Error: Error downloading packages:
2020-03-31 02:18:06 | Status code: 503 for https:/
~~~
More details here:-
* https:/ /logserver. rdoproject. org/98/ 714598/ 2/openstack- check/tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t001/08be6c9/ logs/overcloud- controller- 0/var/log/ dnf.log. txt.gz
[2] https:/ /opendev. org/openstack/ tripleo- quickstart/ src/branch/ master/ config/ release/ tripleo- ci/CentOS- 8/master. yml#L39
[3] https:/ /logserver. rdoproject. org/98/ 714598/ 2/openstack- check/tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t001/08be6c9/ logs/undercloud /home/zuul/ overcloud_ deploy. log.txt. gz
~~~ controller- 0]: FAILED! => changed=true /puppet/ hiera.yaml not found
2020-03-31 03:32:38 | fatal: [overcloud-
2020-03-31 03:32:38 | msg: non-zero return code
2020-03-31 03:32:38 | rc: 1
2020-03-31 03:32:38 | stderr: |-
2020-03-31 03:32:38 | Shared connection to 192.168.24.8 closed.
2020-03-31 03:32:38 | stderr_lines: <omitted>
2020-03-31 03:32:38 | stdout: |-
2020-03-31 03:32:38 | Trying to ping default gateway 10.0.0.1...Ping to 10.0.0.1 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Trying to ping 192.168.24.8 for local network 192.168.24.0/24.
2020-03-31 03:32:38 | Ping to 192.168.24.8 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Failed to start Hiera: RuntimeError: Config file /etc/puppetlabs
~~~
[4] https:/ /opendev. org/openstack/ tripleo- heat-templates/ src/branch/ master/ validation- scripts/ all-nodes. sh#L107