CI: FAIL when installing undercloud, no repos are reachable from puppet

[Sagi (Sergey) Shnaidman]

CI jobs fail in undercloud installation step, puppet can not install packages.

2016-06-22 05:45:33.000 | + puppet apply --detailed-exitcodes /etc/puppet/manifests/puppet-stack-config.pp
2016-06-22 05:45:33.000 | + awk '{print strftime("%Y-%m-%d %H:%M:%S") " - "$0}'
2016-06-22 05:45:52.000 | 2016-06-22 05:45:39 - Could not retrieve fact='fqdn_external', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:45:52.000 | 2016-06-22 05:45:39 - Could not retrieve fact='fqdn_external', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:45:52.000 | 2016-06-22 05:45:39 - Could not retrieve fact='fqdn_internalapi', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:45:52.000 | 2016-06-22 05:45:39 - Could not retrieve fact='fqdn_internalapi', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:45:52.000 | 2016-06-22 05:45:39 - Could not retrieve fact='fqdn_storage', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:45:52.000 | 2016-06-22 05:45:39 - Could not retrieve fact='fqdn_storage', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:45:52.000 | 2016-06-22 05:45:39 - Could not retrieve fact='fqdn_storagemgmt', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:45:52.000 | 2016-06-22 05:45:39 - Could not retrieve fact='fqdn_storagemgmt', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:45:52.000 | 2016-06-22 05:45:39 - Could not retrieve fact='fqdn_tenant', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:45:52.000 | 2016-06-22 05:45:39 - Could not retrieve fact='fqdn_tenant', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:45:52.000 | 2016-06-22 05:45:39 - Could not retrieve fact='fqdn_management', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:45:52.000 | 2016-06-22 05:45:39 - Could not retrieve fact='fqdn_management', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:45:52.000 | 2016-06-22 05:45:39 - Notice: Compiled catalog for instack in environment production in 0.21 seconds
2016-06-22 05:45:52.000 | 2016-06-22 05:45:40 - Warning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.
2016-06-22 05:45:52.000 | 2016-06-22 05:45:40 - (at /usr/share/ruby/vendor_ruby/puppet/type.rb:816:in `set_default')
2016-06-22 05:45:52.000 | 2016-06-22 05:45:43 - Notice: /Stage[main]/Main/Exec[package-upgrade]/returns: executed successfully
2016-06-22 05:45:52.000 | 2016-06-22 05:45:49 - Notice: /Stage[main]/Main/Package[openvswitch]/ensure: created
2016-06-22 05:45:52.000 | 2016-06-22 05:45:49 - Notice: /Stage[main]/Main/Service[openvswitch]/ensure: ensure changed 'stopped' to 'running'
2016-06-22 05:45:52.000 | 2016-06-22 05:45:52 - Notice: /Stage[main]/Main/Exec[os-net-config]/returns: executed successfully
2016-06-22 05:45:52.000 | 2016-06-22 05:45:52 - Notice: Finished catalog run in 11.55 seconds
2016-06-22 05:45:52.000 | + rc=2
2016-06-22 05:45:52.000 | + set -e
2016-06-22 05:45:52.000 | + echo 'puppet apply exited with exit code 2'
2016-06-22 05:45:52.000 | puppet apply exited with exit code 2
2016-06-22 05:45:52.000 | + '[' 2 '!=' 2 -a 2 '!=' 0 ']'
2016-06-22 05:45:52.000 | + FACTER_step=2
2016-06-22 05:45:52.000 | + puppet_apply puppet apply --detailed-exitcodes /etc/puppet/manifests/puppet-stack-config.pp
2016-06-22 05:45:52.000 | + set +e
2016-06-22 05:45:52.000 | + puppet apply --detailed-exitcodes /etc/puppet/manifests/puppet-stack-config.pp
2016-06-22 05:45:52.000 | + awk '{print strftime("%Y-%m-%d %H:%M:%S") " - "$0}'
2016-06-22 05:46:28.000 | 2016-06-22 05:45:58 - Could not retrieve fact='fqdn_external', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:46:28.000 | 2016-06-22 05:45:58 - Could not retrieve fact='fqdn_external', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:46:28.000 | 2016-06-22 05:45:58 - Could not retrieve fact='fqdn_internalapi', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:46:28.000 | 2016-06-22 05:45:58 - Could not retrieve fact='fqdn_internalapi', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:46:28.000 | 2016-06-22 05:45:58 - Could not retrieve fact='fqdn_storage', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:46:28.000 | 2016-06-22 05:45:58 - Could not retrieve fact='fqdn_storage', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:46:28.000 | 2016-06-22 05:45:58 - Could not retrieve fact='fqdn_storagemgmt', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:46:28.000 | 2016-06-22 05:45:58 - Could not retrieve fact='fqdn_storagemgmt', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:46:28.000 | 2016-06-22 05:45:58 - Could not retrieve fact='fqdn_tenant', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:46:28.000 | 2016-06-22 05:45:58 - Could not retrieve fact='fqdn_tenant', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:46:28.000 | 2016-06-22 05:45:58 - Could not retrieve fact='fqdn_management', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:46:28.000 | 2016-06-22 05:45:58 - Could not retrieve fact='fqdn_management', resolution='<anonymous>': undefined method `empty?' for nil:NilClass
2016-06-22 05:46:28.000 | 2016-06-22 05:45:59 - Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
2016-06-22 05:46:28.000 | 2016-06-22 05:46:05 - Warning:

..........

Error: Execution of '/bin/yum -d 0 -e 0 -y install openstack-glance' returned 1: Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2016-06-22 05:46:43.000 | 2016-06-22 05:46:43 -
2016-06-22 05:46:43.000 | 2016-06-22 05:46:43 -
2016-06-22 05:46:43.000 | 2016-06-22 05:46:43 - Error downloading packages:
2016-06-22 05:46:43.000 | 2016-06-22 05:46:43 - python-keystonemiddleware-4.5.2-0.20160611051312.1c3eafb.el7.centos.noarch: [Errno 256] No more mirrors to try.
2016-06-22 05:46:43.000 | 2016-06-22 05:46:43 - Error: /Stage[main]/Glance/Package[openstack-glance]/ensure: change from absent to present failed: Execution of '/bin/yum -d 0 -e 0 -y install openstack-glance' returned 1: Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2016-06-22 05:46:43.000 | 2016-06-22 05:46:43 -
2016-06-22 05:46:43.000 | 2016-06-22 05:46:43 -
2016-06-22 05:46:43.000 | 2016-06-22 05:46:43 - Error downloading packages:
2016-06-22 05:46:43.000 | 2016-06-22 05:46:43 - python-keystonemiddleware-4.5.2-0.20160611051312.1c3eafb.el7.centos.noarch: [Errno 256] No more mirrors to try.

..........

05:50:33.795545 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Warning: /Firewall[998 log all]: Skipping because of failed dependencies
2016-06-22 05:50:33.795645 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[keystone] has failures: true
2016-06-22 05:50:33.795718 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency File[/var/www/cgi-bin/nova] has failures: true
2016-06-22 05:50:33.795785 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[openstack-glance] has failures: true
2016-06-22 05:50:33.795848 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[python-nova] has failures: true
2016-06-22 05:50:33.795910 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[nova-api] has failures: true
2016-06-22 05:50:33.795972 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[nova-cert] has failures: true
2016-06-22 05:50:33.796056 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[nova-compute] has failures: true
2016-06-22 05:50:33.796124 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[nova-conductor] has failures: true
2016-06-22 05:50:33.796205 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[nova-scheduler] has failures: true
2016-06-22 05:50:33.796273 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[neutron] has failures: true
2016-06-22 05:50:33.796338 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[neutron-plugin-ml2] has failures: true
2016-06-22 05:50:33.796403 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[neutron-ovs-agent] has failures: true
2016-06-22 05:50:33.796475 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[heat-common] has failures: true
2016-06-22 05:50:33.796541 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[heat-api] has failures: true
2016-06-22 05:50:33.796604 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[heat-api-cfn] has failures: true
2016-06-22 05:50:33.796667 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[heat-engine] has failures: true
2016-06-22 05:50:33.796730 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[ironic-common] has failures: true
2016-06-22 05:50:33.796793 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[mistral-common] has failures: true
2016-06-22 05:50:33.796856 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[swift-proxy] has failures: true
2016-06-22 05:50:33.796918 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[ironic-api] has failures: true
2016-06-22 05:50:33.796983 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[ironic-conductor] has failures: true
2016-06-22 05:50:33.797047 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[ironic-inspector] has failures: true
2016-06-22 05:50:33.797110 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[mistral-api] has failures: true
2016-06-22 05:50:33.797184 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[mistral-engine] has failures: true
2016-06-22 05:50:33.797255 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[mistral-executor] has failures: true
2016-06-22 05:50:33.797319 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[zaqar-common] has failures: true
2016-06-22 05:50:33.797383 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Package[openstack-tempest] has failures: true
2016-06-22 05:50:33.797448 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: /Firewall[999 drop all]: Dependency Service[ovs-cleanup-service] has failures: true
2016-06-22 05:50:33.797520 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
2016-06-22 05:50:33.797572 | 2016-06-22 05:50:33.000 | 2016-06-22 05:50:23 - Notice: Finished catalog run in 564.40 seconds
2016-06-22 05:50:33.797608 | 2016-06-22 05:50:33.000 | + rc=6
2016-06-22 05:50:33.797638 | 2016-06-22 05:50:33.000 | + set -e
2016-06-22 05:50:33.797677 | 2016-06-22 05:50:33.000 | + echo 'puppet apply exited with exit code 6'
2016-06-22 05:50:33.797713 | 2016-06-22 05:50:33.000 | puppet apply exited with exit code 6
2016-06-22 05:50:33.797748 | 2016-06-22 05:50:33.000 | + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
2016-06-22 05:50:33.797775 | 2016-06-22 05:50:33.000 | + exit 6
2016-06-22 05:50:33.797862 | 2016-06-22 05:50:33.000 | [2016-06-22 05:50:33,735] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
2016-06-22 05:50:33.797888 | 2016-06-22 05:50:33.000 |
2016-06-22 05:50:33.797934 | 2016-06-22 05:50:33.000 | [2016-06-22 05:50:33,736] (os-refresh-config) [ERROR] Aborting...
2016-06-22 05:50:33.797970 | 2016-06-22 05:50:33.000 | Traceback (most recent call last):
2016-06-22 05:50:33.798007 | 2016-06-22 05:50:33.000 | File "<string>", line 1, in <module>
2016-06-22 05:50:33.798063 | 2016-06-22 05:50:33.000 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1039, in install
2016-06-22 05:50:33.798096 | 2016-06-22 05:50:33.000 | _run_orc(instack_env)
2016-06-22 05:50:33.798152 | 2016-06-22 05:50:33.000 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 917, in _run_orc
2016-06-22 05:50:33.798212 | 2016-06-22 05:50:33.000 | _run_live_command(args, instack_env, 'os-refresh-config')
2016-06-22 05:50:33.798278 | 2016-06-22 05:50:33.000 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 450, in _run_live_command
2016-06-22 05:50:33.798325 | 2016-06-22 05:50:33.000 | raise RuntimeError('%s failed. See log for details.' % name)
2016-06-22 05:50:33.798370 | 2016-06-22 05:50:33.000 | RuntimeError: os-refresh-config failed. See log for details.
2016-06-22 05:50:33.798417 | 2016-06-22 05:50:33.000 | Command 'instack-install-undercloud' returned non-zero exit status 1

[Sagi (Sergey) Shnaidman (sshnaidm)]

Logs:

http://logs.openstack.org/83/332483/2/check-tripleo/gate-tripleo-ci-centos-7-ha/5e98113/logs/undercloud/var/log/undercloud_install.txt.gz#_2016-06-22_05_45_52_000

[Sagi (Sergey) Shnaidman (sshnaidm)]

The rdo commit https://github.com/redhat-openstack/rdoinfo/commit/b4db304e973fe7e8043c2eac2fcc1b91d9c3b005 has broken repositories. Was fixed on RDO side.