bootstrap admin node failed

[Sergey Kulanov]

We have several issues on master node bootstrapping. It seems that they are not related and probably needs to be split:

https://product-ci.infra.mirantis.net/job/8.0-kilo.ubuntu.smoke_neutron/146/

1. fuel-snapshot-2015-10-19_13-15-46/10.109.0.2/var/log/boot.log
Starting docker: ESC[60G[ESC[0;32m OK ESC[0;39m]^M
Starting rhsmcertd...ESC[60G[ESC[0;32m OK ESC[0;39m]^M
/etc/rc3.d/S99local: command substitution: line 54: syntax error near unexpected token `)'
/etc/rc3.d/S99local: command substitution: line 54: `get_ethernet_interfaces())'

2. fuel-snapshot-2015-10-19_13-15-46/10.109.0.2/var/log/puppet/bootstrap_admin_node.log
ESC[1;31mError: Cannot create /var/www/nailgun/2015.1.0-8.0/centos/auxiliary; parent directory /var/www/nailgun/2015.1.0-8.0/centos does not existESC[0m
ESC[1;31mError: /Stage[main]/Nailgun::Auxiliaryrepos/File[/var/www/nailgun/2015.1.0-8.0/centos/auxiliary/]/ensure: change from absent to directory failed: Cannot create /var/www/nailgun/2015.1.0-8.0/centos/auxiliary; parent directory /var/www/nailgun/2015.1.0-8.0/centos does not existESC[0m
ESC[0;36mDebug: /Stage[main]/Nailgun::Auxiliaryrepos/File[/var/www/nailgun/2015.1.0-8.0/centos/auxiliary/]: Nothing to manage: no ensure and the resource doesn't existESC[0m
ESC[mNotice: /Stage[main]/Nailgun::Auxiliaryrepos/Package[createrepo]: Dependency File[/var/www/nailgun/2015.1.0-8.0/centos/auxiliary/] has failures: trueESC[0m
ESC[1;31mWarning: /Stage[main]/Nailgun::Auxiliaryrepos/Package[createrepo]: Skipping because of failed dependenciesESC[0m

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/237144

[Sergey Kulanov (skulanov)]

Fix to fuel-main:
[root@nailgun ~]# cat /var/log/boot.log
  Welcome to CentOS
Starting udev: [ OK ]
Setting hostname nailgun.test.domain.local: [ OK ]
Setting up Logical Volume Management: 5 logical volume(s) in volume group "os" now active
                                                           [ OK ]
Checking filesystems
/dev/mapper/os-root: clean, 49807/642112 files, 311707/2564096 blocks
/dev/vda3: clean, 35/51200 files, 37399/204800 blocks
/dev/mapper/os-var: clean, 3294/1236992 files, 1284844/4947968 blocks
/dev/mapper/os-varlibdocker: clean, 11/1485120 files, 137233/5939200 blocks
/dev/mapper/os-varlog: clean, 32/1261568 files, 123337/5038080 blocks
                                                           [ OK ]
Remounting root filesystem in read-write mode: [ OK ]
Mounting local filesystems: [ OK ]
Enabling /etc/fstab swaps: [ OK ]
Entering non-interactive startup
Calling the system activity data collector (sadc)...
Starting monitoring for VG os: 5 logical volume(s) in volume group "os" monitored
                                                           [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0: Determining if ip address 10.109.25.2 is already in use for device eth0...
                                                           [ OK ]
Starting auditd: [ OK ]
Starting portreserve: [ OK ]
Starting system logger: [ OK ]
Starting kdump: [FAILED]
Mounting filesystems: [ OK ]
Retrigger failed udev events [ OK ]
Adding udev persistent rules [ OK ]
Starting postfix: [ OK ]
Starting crond: [ OK ]
Starting cgconfig service: [ OK ]
Starting docker: [ OK ]
Starting rhsmcertd... [ OK ]

[Sergey Kulanov (skulanov)]

the second issue was solved by reverting https://github.com/openstack/fuel-web/commit/2e948eaa0152c886f280c5f42e88500a73c75011

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/237144
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=c0c7176b1245a4188f93082fd79e0502ce872387
Submitter: Jenkins
Branch: master

commit c0c7176b1245a4188f93082fd79e0502ce872387
Author: Sergey Kulanov <email address hidden>
Date: Mon Oct 19 22:01:00 2015 +0300

    Fix function call in /etc/rc.local script

    Change-Id: Icb2906de493ee0b100add11847331f152bea35ae
    Related-bug: #1507728

[Grigory Mikhailov (gmikhailov)]

Verified on ISO #154.
Environment created via "dos.py" script.
Admin node boot log: "/var/log/boot.log" looks as expected:

[root@nailgun log]# cat boot.log
                Welcome to CentOS
Starting udev: [ OK ]
Setting hostname nailgun.domain.local: [ OK ]
Setting up Logical Volume Management: 5 logical volume(s) in volume group "os" now active
                                                           [ OK ]
Checking filesystems
/dev/mapper/os-root: clean, 54901/642112 files, 323885/2564096 blocks
/dev/vda3: clean, 35/51200 files, 37411/204800 blocks
/dev/mapper/os-var: clean, 3244/1236992 files, 1290695/4947968 blocks
/dev/mapper/os-varlibdocker: clean, 11/1488032 files, 137415/5947392 blocks
/dev/mapper/os-varlog: clean, 32/1261568 files, 123337/5046272 blocks
                                                           [ OK ]
Remounting root filesystem in read-write mode: [ OK ]
Mounting local filesystems: [ OK ]
Enabling /etc/fstab swaps: [ OK ]
Entering non-interactive startup
Calling the system activity data collector (sadc)...
Starting monitoring for VG os: 5 logical volume(s) in volume group "os" monitored
                                                           [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0: Determining if ip address 10.109.0.2 is already in use for device eth0...
                                                           [ OK ]
Starting auditd: [ OK ]
Starting portreserve: [ OK ]
Starting system logger: [ OK ]
Starting kdump: [FAILED]
Mounting filesystems: [ OK ]
Retrigger failed udev events [ OK ]
Adding udev persistent rules [ OK ]
Starting postfix: [ OK ]
Starting crond: [ OK ]
Starting cgconfig service: [ OK ]
Starting docker: [ OK ]
Starting rhsmcertd... [ OK ]

release_versions:
  2015.1.0-8.0:
    VERSION:
      api: '1.0'
      astute_sha: 959b06c5ef8143125efd1727d350c050a922eb12
      build_id: '154'
      build_number: '154'
      feature_groups: - mirantis
      fuel-agent_sha: 07560a9fc3ce5301ace04d2d3e5d68db6ee4f8d5
      fuel-createmirror_sha: a034dcb06520df58a7338816900a431a6b61d83f
      fuel-library_sha: 1e690ed95452297294c710a2f5886ef671d6b6da
      fuel-nailgun-agent_sha: 3e9d17211d65c80bf97c8d83979979f6c7feb687
      fuel-nailgun_sha: 37a535e11a7939e206ffababf3ecf12504cd91c4
      fuel-ostf_sha: 9690a2de829d3b063ed1e64b0b10dde39f711dc0
      fuel-upgrade_sha: 1e894e26d4e1423a9b0d66abd6a79505f4175ff6
      fuelmain_sha: 45b79c9121a08c2b467a3246dc1fa714e4c2043d
      fuelmenu_sha: 8a32c53c1fa13b036000f589f96e876277dbd071...