cannot install fuel from package on Centos node due to failed cobbler_sync task if hostname != fuel

[Artem Hrechanychenko]

Detailed bug description:
Installing Fuel services on Centos node failed with "The were failures while running task: cobbler with exit code: 6
[root@localhost ~]# ERROR: Fuel node deployment FAILED! Check /var/log/puppet/bootstrap_admin_node.log for details"

from /var/log/puppet/bootstrap_admin_node.log:

Notice: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: httpd does not appear to be running and proxying cobbler, or SELinux is in the way. Original traceback:
Notice: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: File "/usr/lib/python2.7/site-packages/cobbler/cli.py", line 252, in check_setup
Notice: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: s.ping()
Notice: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
Notice: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: return self.__send(self.__name, args)
Notice: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
Notice: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: verbose=self.__verbose
Notice: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
Notice: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: return self.single_request(host, handler, request_body, verbose)
Notice: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: File "/usr/lib64/python2.7/xmlrpclib.py", line 1321, in single_request
Notice: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: response.msg,
Notice: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: ProtocolError: <ProtocolError for 10.20.0.2:80/cobbler_api: 503 Service Unavailable>
Error: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]: Failed to call refresh: cobbler sync returned 155 instead of one of [0]

Reason of this traceback:
Debug: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: Exec try 15/20
Debug: Exec[cobbler_sync](provider=posix): Executing 'cobbler sync'
Debug: Executing 'cobbler sync'
Debug: /Stage[main]/Cobbler::Server/Exec[cobbler_sync]/returns: Sleeping for 3.0 seconds between tries

cobbler sync don't executed 20 times in a row

from logs: - httpd does not appear to be running and proxying cobbler, or SELinux is in the way.

httpd exist on node and SELinux in permissive mode

[root@localhost ~]# yum install httpd
Loaded plugins: fastestmirror, priorities
Loading mirror speeds from cached hostfile
 * base: mirror.logol.ru
 * extras: mirror.corbina.net
 * mos9.0-base: mirror.seed-cz1.fuel-infra.org
 * mos9.0-security: mirror.seed-cz1.fuel-infra.org
 * mos9.0-updates: mirror.seed-cz1.fuel-infra.org
 * updates: mirror.logol.ru
Package httpd-2.4.6-40.el7.centos.x86_64 already installed and latest version
Nothing to do
[root@localhost ~]# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28
[root@localhost ~]# sestatus -b | grep httpd | grep on$
httpd_builtin_scripting on
httpd_enable_cgi on
httpd_graceful_shutdown on

Steps to reproduce:
1) Create VM - 2VCPU, 4096 RAM 50HDD, Centos Minimal Install, 3 network interfaces
2) download http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7/os/x86_64/Packages/fuel-release-9.0.0-1.mos6316.noarch.rpm
3) yum install fuel-release ..path_to_fuel_release
4) yum install fuel-setup
5) bootstrap_admin_node.sh
6) change local mos repo http://127.0.0.1:8080/ubuntu/x86_64 mos9.0 main restricted to http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0 mos9.0 main restricted

Expected results:
Fuel services was installed on Centos node

Actual result:
 Failed to call refresh: cobbler sync returned 155 instead of one of [0]

Reproducibility:
100% on KVM, VBOX

Workaround:
Change /etc/hosts to
"ip fuel.domain.tld fuel"

Impact:
Separate master node deployment feature

Description of the environment:
     Operation system: Linux localhost.localdomain 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[Ivan Ponomarev (ivanzipfer)]

I little confused with 6) change local mos repo 127.0.0... to mirror.fuel-infra...... mos.9.0 in fuelmenu

Why are you trying to change local repository?

[Artem Hrechanychenko (agrechanichenko)]

Hi!
It's scenario for separate master node deployment feature

In this case we install fuel services on centos7 node.
To perform this action need to install fuel-release.rpm from mos9.0 repo, after that need to install fuel-setup package from mos repos.

After that need to execute shell script - bootstrap_admin_nodes.sh

that scrip download all packages to node and install fuel services.

as part of installing process - fuelmenu.
In Bootstrap section of fuelmenu mos repo reffer to http://127.0.0.1:8080/ubuntu/x86_64 mos9.0 main restricted and this repository not exist at this time so bootstrap not be built.

So need to change not existing local repo to online repository

[Sergey Kulanov (skulanov)]

Resolved by updating fuel packages

[Sergey Kulanov (skulanov)]

+ master node should have name different from localhost

[ivano (l-ivan)]

@skulanov in which package rel is the fix?