[9.0][9.1][Upgrade] Mos-updates 9.0 repo breaks new 9.0 master node installation

[Miroslav Anashkin]

Version:
Released Mirantis Fuel 9.0 ISO
[root@fuel ~]# cat /etc/fuel_build_id
685

Description and Steps to reproduce:
As soon as Fuel 9.1 released, mos-updates repository is no longer empty.

If we try to install new 9.0 now, the following happening:

0. Fuel master node requires internet connection in order to create bootstrap image. Usually we provide such connection.

1. bootstrap_admin_node.sh configures all the repositories, including mos-updates and mos-security before installing the Fuel-related components (Cobbler, Nailgun etc.)

2. During the deployment process Puppet scripts install the required packages and then attempt to install Cobbler. Since newer Cobbler version exists in mos-updates repository - it is got selected by default.

Actual results:
Cobbler installation fails due to incorrect dependencies and the remained package installation fails by dependency to Cobbler.
Please find details in the attached bootstrap_admin_node.log

Expected result:
Master node deployment finished successfully.

Workaround:
Please see the next comment.
https://bugs.launchpad.net/fuel/+bug/1634706/comments/2
Workaround basically works, however some newer version packages may be installed.

[Miroslav Anashkin (manashkin)]

Workaround:
The idea is to move mos-updates and mos-security repositories configuration to the later stage, after the all reauired packages installed.

Deploy the 9.0 master node as is.
Replace the /usr/sbin/bootstrap_admin_node.sh
on the master node with the attached version.
Run the following command (under nohup if you run it via ssh console - the script you are going to run restarts networking service):

$ nohup /usr/sbin/bootstrap_admin_node.sh

[Vitaly Sedelnik (vsedelnik)]

Invalid for 10.0 as we are not going to create mos-updates repo for 10.0. Confirmed for 9.x.

[Alexandr Kostrikov (akostrikov-mirantis)]

http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7/updates/x86_64/Packages/cobbler-2.6.9-1.el7~mos11.noarch.rpm

There is such package. Are You sure there were no connection problems?

[Anton Chevychalov (achevychalov)]

I tried reproduce that problem in lab and face no issue. It seems to we there is additional factor that trigger the issue.

[Vitaly Sedelnik (vsedelnik)]

Miroslav, please provide exact steps to reproduce the issue and the list of repositories enabled on master node.

[Denis Meltsaykin (dmeltsaykin)]

Miroslav, according to our observations there is no such behavior in 9.0/9.1. Please clarify what steps did you take to achieve such a situation?

[Denis Meltsaykin (dmeltsaykin)]

Also there are such messages in the log:
Error: Execution of '/bin/yum -d 0 -e 0 -y install cobbler' returned 1: Error downloading packages:
  cobbler-2.6.9-1.el7~mos11.noarch: [Errno 256] No more mirrors to try.
Error: /Stage[main]/Cobbler::Packages/Package[cobbler]/ensure: change from absent to present failed: Execution of '/bin/yum -d 0 -e 0 -y install cobbler' returned 1: Error downloading packages:
  cobbler-2.6.9-1.el7~mos11.noarch: [Errno 256] No more mirrors to try.

it seems like something was broken in your internet connection or firewall, as this package is fully downloadable from all the mirrors.

[Miroslav Anashkin (manashkin)]

The following pieces of code are responsible for this issue:

In bootstrap_admin_node.sh:
########################################################################
363 # Enable online base MOS repos (security, updates) if we run an ISO installation
364 [ -f /etc/fuel_build_id ] && \
365 yum-config-manager --enable mos${FUEL_RELEASE}-security mos${FUEL_RELEASE}-updates --save
...
525 # apply puppet
526 /etc/puppet/modules/fuel/examples/deploy.sh || fail
########################################################################

[Miroslav Anashkin (manashkin)]

BTW, we got reports that Fuel 9.1 community ISO version has the same issue.

[Daniel (ddanielr)]

Attempting to do an install using the fuel 9.0 community stable release also fails.

When attempting to install fuel 9.0 on a Dell R710 the puppet install fails when attempting to install the python-openstackclient package stating a dependency conflict.

Error: Package: 1:python-openstackclient-2.2.0-1.el7~mos2.noarch (mos9.0-updates)
           Requires: python-openstacksdk >= 0.8.1
           Available: python2-openstacksdk-0.7.4-1.el7.noarch (mos)
               python-openstacksdk = 0.7.4-1.el7

After further review it appears that yum is selecting the python-openstackclient-2.2.x package instead of the python-openstackclient-2.1.x package and this is causing dependency issues.

Explicitly installing the 2.1.x version of the python-openstackclient package is successful.

Also, disabling the mos9.0-updates.repo removes the offending package version

Attached is my install log file.

Steps to replicate:

1. Download fuel 9.0 community iso
2. Mount using iDRAC virtual media and boot virtual media
3. Select fuel install option 1
4. Verify that network settings are correct & press F8 to start the installer
5. Puppet install fails.

[Denis Meltsaykin (dmeltsaykin)]

Miroslav, please provide an env. We cannot reproduce the issue on 9.0GA product ISO. The code you are referring to is totally ok and works as intended.

[Vitaly Sedelnik (vsedelnik)]

The bug stays in Incomplete for more than 2 months, setting to Invalid for 9.2