bootstrap_admin_node.sh script breaks dns settings

[Artem Hrechanychenko]

Detailed bug description:
 bootstrap_admin_node.sh failed.
 Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install ami-creator' returned 1: Error downloading packages:
  pykickstart-1.99.66.6-1.el7.noarch: [Errno 256] No more mirrors to try.
  dosfstools-3.0.20-9.el7.x86_64: [Errno 256] No more mirrors to try.
  system-config-keyboard-1.4.0-4.el7.noarch: [Errno 256] No more mirrors to try.
  system-config-keyboard-base-1.4.0-4.el7.noarch: [Errno 256] No more mirrors to try.
  squashfs-tools-4.3-0.21.gitaae0aff4.el7.x86_64: [Errno 256] No more mirrors to try.
  ami-creator-0.3-1.el7~mos1.noarch: [Errno 256] No more mirrors to try.
  1:python-imgcreate-13.4.5-2.el7~mos1.x86_64: [Errno 256] No more mirrors to try.
Error: /Stage[main]/Main/Package[ami-creator]/ensure: change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install ami-creator' returned 1: Error downloading packages:
  pykickstart-1.99.66.6-1.el7.noarch: [Errno 256] No more mirrors to try.
  dosfstools-3.0.20-9.el7.x86_64: [Errno 256] No more mirrors to try.
  system-config-keyboard-1.4.0-4.el7.noarch: [Errno 256] No more mirrors to try.
  system-config-keyboard-base-1.4.0-4.el7.noarch: [Errno 256] No more mirrors to try.
  squashfs-tools-4.3-0.21.gitaae0aff4.el7.x86_64: [Errno 256] No more mirrors to try.
  ami-creator-0.3-1.el7~mos1.noarch: [Errno 256] No more mirrors to try.
  1:python-imgcreate-13.4.5-2.el7~mos1.x86_64: [Errno 256] No more mirrors to try.
Debug: Class[Main]: The container Stage[main] will propagate my refresh event
Debug: Exec[initialize atop_current](provider=posix): Executing '/etc/cron.daily/atop_retention'
Debug: Executing '/etc/cron.daily/atop_retention'
Notice: /Stage[main]/Osnailyfacter::Atop/Exec[initialize atop_current]: Triggered 'refresh' from 1 events
Debug: /Stage[main]/Osnailyfacter::Atop/Exec[initialize atop_current]: The container Class[Osnailyfacter::Atop] will propagate my refresh event
Debug: Class[Osnailyfacter::Atop]: The container Stage[main] will propagate my refresh event
Debug: Finishing transaction 23018440
Debug: Storing state
Debug: Stored state in 0.04 seconds
Notice: Finished catalog run in 73.16 seconds
Debug: Using settings: adding file resource 'rrddir': 'File[/var/lib/puppet/rrd]{:path=>"/var/lib/puppet/rrd", :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: /File[/var/lib/puppet/rrd]/seluser: Found seluser default 'system_u' for /var/lib/puppet/rrd
Debug: /File[/var/lib/puppet/rrd]/selrole: Found selrole default 'object_r' for /var/lib/puppet/rrd
Debug: /File[/var/lib/puppet/rrd]/seltype: Found seltype default 'puppet_var_lib_t' for /var/lib/puppet/rrd
Debug: /File[/var/lib/puppet/rrd]/selrange: Found selrange default 's0' for /var/lib/puppet/rrd
Debug: Finishing transaction 45576300
Debug: Received report to process from nailgun.test.domain.local
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Processing report from nailgun.test.domain.local with processor Puppet::Reports::Store
The were failures while running task: host with exit code: 6
ERROR: Fuel node deployment FAILED! Check /var/log/puppet/bootstrap_admin_node.log for details

Steps to reproduce:
 1) Deploy master node using Centos node, fuel-release - http://mirror.fuel-infra.org/mos-repos/centos/mos-master-centos7/os/x86_64/Packages/fuel-release-10.0~b1-2.el7~mos1.noarch.rpm

 2) provide additional mos repo
    [root@nailgun ~]# cat /etc/yum.repos.d/custom.repo
    [mos]
    name=mos
    baseurl=http://mirror.fuel-infra.org/mos-repos/centos/mos-master-centos7/os/x86_64/
    enabled=1
    gpgcheck=0
    priority=10
    skip_if_unavailable=1

 3) provide additional deb repo using
    [root@nailgun ~]# cat /root/default_deb_repos.yaml
---
- name: "mos"
  uri: http://mirror.fuel-infra.org/mos-repos/ubuntu/master
  suite: mos-master
  section: "main restricted"
  type: "deb"
  priority: 1000

 4) yum install -y fuel-release-10.0~b1-2.el7~mos1.noarch.rpm
 5) yum install -y fuel-setup

 6) run bootstrap_admin_node.sh

Expected results:
 Fuel services installed successfully

Actual result:
 The were failures while running task: host with exit code: 6
ERROR: Fuel node deployment FAILED! Check /var/log/puppet/bootstrap_admin_node.log for details

Reproducibility:
 every time

Workaround:
  change DNS nameserver in /etc/resolv.conf 10.109.21.2 to 8.8.8.8

Impact:

 Fuel-qa must be able to run BVT, Swarm not using ISO

Description of the environment:

1)(venv-nailgun-tests-3.0) agrechanichenko@cz7322:~$ pip freeze |grep fuel-devops
fuel-devops==3.0.0

 2)[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_release:
 10.0
cat /etc/fuel_openstack_version:
 newton-10.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-misc-10.0~b1-1.noarch
 fuel-10.0~b1-2.el7~mos1.noarch
 fuel-bootstrap-cli-10.0~b1-1.el7~mos4.noarch
 fuel-setup-10.0~b1-2.el7~mos1.noarch
 fuel-notify-10.0~b1-1.noarch
 fuel-library10.0-10.0~b1-1.noarch
 network-checker-10.0~b1-1.el7~mos0.noarch
 shotgun-10.0~b1-1.el7~mos0.noarch
 fuel-agent-10.0~b1-1.el7~mos4.noarch
 fuel-openstack-metadata-10.0~b1-1.el7~mos6.noarch
 fuel-ui-10.0~b1-1.el7~mos10.noarch
 fuel-migrate-10.0~b1-1.noarch
 fuel-release-10.0~b1-2.el7~mos1.noarch
 python-fuelclient-10.0.0~b1-1.el7~mos2.noarch
 fuelmenu-10.0~b1-1.el7~mos0.noarch
 fuel-utils-10.0~b1-1.noarch

3) Repo-id : 10.0_auxiliary
Repo-name : 10.0_auxiliary
Repo-revision: 1469266217
Repo-updated : Sat Jul 23 09:30:17 2016
Repo-pkgs : 0
Repo-size : 0
Repo-baseurl : file:///var/www/nailgun/newton-10.0/centos/auxiliary/
Repo-expire : 21,600 second(s) (last: Sat Jul 23 09:30:18 2016)
Repo-filename: /etc/yum.repos.d/10.0_auxiliary.repo

Repo-id : base/7/x86_64
Repo-name : CentOS-7 - Base
Repo-revision: 1449700451
Repo-updated : Wed Dec 9 22:35:45 2015
Repo-pkgs : 8,887
Repo-size : 6.4 G
Repo-mirrors : http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=genclo
Repo-baseurl : http://mirror.hosting90.cz/centos/7.2.1511/os/x86_64/ (9 more)
Repo-expire : 21,600 second(s) (last: Sat Jul 23 09:24:49 2016)
Repo-excluded: 120
Repo-filename: /etc/yum.repos.d/CentOS-Base.repo

Repo-id : extras/7/x86_64
Repo-name : CentOS-7 - Extras
Repo-revision: 1467739008
Repo-updated : Tue Jul 5 17:17:21 2016
Repo-pkgs : 318
Repo-size : 936 M
Repo-mirrors : http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=extras&infra=genclo
Repo-baseurl : http://mirror.hosting90.cz/centos/7.2.1511/extras/x86_64/ (9 more)
Repo-expire : 21,600 second(s) (last: Sat Jul 23 09:24:49 2016)
Repo-excluded: 38
Repo-filename: /etc/yum.repos.d/CentOS-Base.repo

Repo-id : mos
Repo-name : mos
Repo-revision: 1469104950
Repo-updated : Thu Jul 21 12:42:49 2016
Repo-pkgs : 1,243
Repo-size : 1.6 G
Repo-baseurl : http://mirror.fuel-infra.org/mos-repos/centos/mos-master-centos7/os/x86_64/
Repo-expire : 21,600 second(s) (last: Sat Jul 23 09:24:50 2016)
Repo-filename: /etc/yum.repos.d/custom.repo

Repo-id : mos10.0-base/7
Repo-name : mos10.0-base
Repo-revision: 1459339594
Repo-updated : Wed Mar 30 12:06:34 2016
Repo-pkgs : 0
Repo-size : 0
Repo-mirrors : http://mirror.fuel-infra.org/mos-repos/centos/mos10.0-centos7/mos-mirrors-os.txt
Repo-baseurl : http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos10.0-centos7/os/x86_64/ (1 more)
Repo-expire : 21,600 second(s) (last: Sat Jul 23 09:24:50 2016)
Repo-filename: /etc/yum.repos.d/mos-os.repo

Repo-id : mos10.0-security/7
Repo-name : mos10.0-security
Repo-revision: 1459339601
Repo-updated : Wed Mar 30 12:06:41 2016
Repo-pkgs : 0
Repo-size : 0
Repo-mirrors : http://mirror.fuel-infra.org/mos-repos/centos/mos10.0-centos7/mos-mirrors-security.txt
Repo-baseurl : http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos10.0-centos7/security/x86_64/ (1 more)
Repo-expire : 21,600 second(s) (last: Sat Jul 23 09:24:51 2016)
Repo-filename: /etc/yum.repos.d/mos-security.repo

Repo-id : mos10.0-updates/7
Repo-name : mos10.0-updates
Repo-revision: 1459339599
Repo-updated : Wed Mar 30 12:06:39 2016
Repo-pkgs : 0
Repo-size : 0
Repo-mirrors : http://mirror.fuel-infra.org/mos-repos/centos/mos10.0-centos7/mos-mirrors-updates.txt
Repo-baseurl : http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos10.0-centos7/updates/x86_64/ (1 more)
Repo-expire : 21,600 second(s) (last: Sat Jul 23 09:24:51 2016)
Repo-filename: /etc/yum.repos.d/mos-updates.repo

Repo-id : updates/7/x86_64
Repo-name : CentOS-7 - Updates
Repo-revision: 1469029255
Repo-updated : Wed Jul 20 15:45:56 2016
Repo-pkgs : 1,994
Repo-size : 5.5 G
Repo-mirrors : http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates&infra=genclo
Repo-baseurl : http://mirror.hosting90.cz/centos/7.2.1511/updates/x86_64/ (9 more)
Repo-expire : 21,600 second(s) (last: Sat Jul 23 09:24:51 2016)
Repo-excluded: 32
Repo-filename: /etc/yum.repos.d/CentOS-Base.repo

[Artem Hrechanychenko (agrechanichenko)]

Update

looks like problem in dns

[root@nailgun ~]# yum clean all
Loaded plugins: fastestmirror, priorities
Bad id for repo: CentOS-7 - Extras, byte = 8
Cleaning repos: 10.0_auxiliary CentOS-Plus CentOS-contrib base extras mos mos10.0-base mos10.0-security mos10.0-updates updates
Cleaning up everything
Cleaning up list of fastest mirrors
[root@nailgun ~]# yum install -y ami-creator
Loaded plugins: fastestmirror, priorities
Bad id for repo: CentOS-7 - Extras, byte = 8
10.0_auxiliary | 2.9 kB 00:00:00
http://mirror.centos.org/centos/7.2.1511//centosplus/x86_64/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: mirror.centos.org; Unknown error"
Trying other mirror.
http://mirror.centos.org/centos/7.2.1511//cr/x86_64/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: mirror.centos.org; Unknown error"
Trying other mirror.
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=genclo error was
14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error"

 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

Cannot find a valid baseurl for repo: base/7/x86_64
[root@nailgun ~]# ping ya.ru
ping: unknown host ya.ru
[root@nailgun ~]# cat /etc/resolv.conf
search test.domain.local
domain test.domain.local
nameserver 10.109.21.2

Before starting bootstrap_admin_node.sh dns nameserver for searching was 8.8.8.8

[root@nailgun ~]# cat /etc/hiera/astute.yaml |grep DNS
"DNS_DOMAIN": "test.domain.local"
"DNS_SEARCH": "test.domain.local"
"DNS_UPSTREAM": "10.109.21.1"

after changing 10.109.21.2 to 8.8.8.8 in /etc/resolv.conf all works fine

[Vladimir Kozhukalov (kozhukalov)]

By default fuel-menu uses 8.8.8.8 dns address. As far as I know fuel-qa modifies astute.yaml and puts this address 10.109.21.2 there. Please modify this line https://github.com/openstack/fuel-qa/blob/master/fuelweb_test/helpers/fuel_actions.py#L185

[Artem Hrechanychenko (agrechanichenko)]

No, fuel-qa method write to astute yaml libvirt gateway address, 10.109.x.1

[Artem Hrechanychenko (agrechanichenko)]

cat /etc/hierra/astute.yaml |grep "DNS"

"DNS_DOMAIN": "test.domain.local"
"DNS_SEARCH": "test.domain.local"
"DNS_UPSTREAM": "10.109.21.1

[Artem Hrechanychenko (agrechanichenko)]

so promlem is next:

1) from bootstrap_admin_node.sh >> etc/puppet/modules/fuel/examples/deploy.sh
2) deploy.sh >> host: https://github.com/openstack/fuel-library/blob/master/deployment/puppet/fuel/examples/deploy.sh#L19
3) changing /etc/resolv.conf : https://github.com/openstack/fuel-library/blob/master/deployment/puppet/fuel/examples/host.pp#L71-L76
4)/etc/resolv.conf <<<: %cobbler_host , - master node IP >> https://github.com/openstack/fuel-library/blob/master/deployment/puppet/fuel/templates/resolv.conf.erb

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/347259

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/347259
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=524bb61b65f2582bad636578379427250e245237
Submitter: Jenkins
Branch: master

commit 524bb61b65f2582bad636578379427250e245237
Author: Vladimir Kozhukalov <email address hidden>
Date: Tue Jul 26 13:49:47 2016 +0300

    Reconfigure resolv.conf once cobbler is installed

    Once cobbler is ready and working
    we need to re-configure /etc/resolv.conf so it points to
    cobbler.

    Change-Id: I43cf1607dcfcd5e19daeb30512a0b18ee8d6a7be
    Closes-Bug: #1606181

[Vladimir Kozhukalov (kozhukalov)]

This bug should not be fixed for 9.1 since we deliver 9.1 as MU for 9.0. This bug is only important for initial deployment.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/372602

[Dmitry Pyzhov (dpyzhov)]

The issue doesn't affect our main use cases. Let's merge the fix in 9.2

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/372602
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=32bec8708c08b9d695ed143e49403b0c4a2d43c8
Submitter: Jenkins
Branch: stable/mitaka

commit 32bec8708c08b9d695ed143e49403b0c4a2d43c8
Author: Vladimir Kozhukalov <email address hidden>
Date: Tue Jul 26 13:49:47 2016 +0300

    Reconfigure resolv.conf once cobbler is installed

    Once cobbler is ready and working
    we need to re-configure /etc/resolv.conf so it points to
    cobbler.

    Change-Id: I43cf1607dcfcd5e19daeb30512a0b18ee8d6a7be
    Closes-Bug: #1606181
    (cherry picked from commit 524bb61b65f2582bad636578379427250e245237)

[Alexey. Kalashnikov (akalashnikov)]

Verified on 9.2

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0 release.