MOS 9.0 community ISO installation fails due to wrong version of python-openstacksdk on ISO

[Marcin Jerzak]

Detailed bug description:

After installing MOS with FUEL from:
http://seed.fuel-infra.org/fuelweb-community-release/fuel-community-9.0.iso.torrent

Unable to boot slave nodes over PXE - error:

  Invalid or corrupt kernel image.

Master node bootstrap errors:

/var/log/puppet/bootstrap_admin_node.log

Error: Execution of '/bin/yum -d 0 -e 0 -y install python-openstackclient' returned 1: Error: Package: 1:python-openstackclient-2.2.0-1.el7~mos2.noarch (mos9.0-updates)
           Requires: python-openstacksdk >= 0.8.1
           Available: python2-openstacksdk-0.7.4-1.el7.noarch (mos)
               python-openstacksdk = 0.7.4-1.el7
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
Error: /Stage[main]/Openstacklib::Openstackclient/Package[python-openstackclient]/ensure: change from absent to present failed: Execution of '/bin/yum -d 0 -e 0 -y install python-openstackclient' returned 1: Error: Package: 1:python-openstackclient-2.2.0-1.el7~mos2.noarch (mos9.0-updates)
           Requires: python-openstacksdk >= 0.8.1
           Available: python2-openstacksdk-0.7.4-1.el7.noarch (mos)
               python-openstacksdk = 0.7.4-1.el7
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

...

Error: Could not prefetch keystone_domain provider 'openstack': Command openstack_command is missing

Steps to reproduce:

1. Install MOS with FUEL from http://seed.fuel-infra.org/fuelweb-community-release/fuel-community-9.0.iso.torrent with default settings
as a KVM virtual machine

2. Boot OpenStack slave node over PXE

Expected results:

MOS installation is successful and slave node boots over PXE without errors.

Actual results:

MOS installation is corrupted and slave node cannot boot over PXE.

Workaround:

mv /etc/yum.repos.d/mos-updates.repo /etc/yum.repos.d/mos-updates.repo.back
/bin/yum clean all
/bin/yum install python-openstackclient
/etc/puppet/modules/fuel/examples/deploy.sh
fuel-bootstrap -v --debug build --activate

Impact:

Impossible to install MOS 9.0

Description of the environment:

KVM on bare metal on Ubuntu 14.04.5 LTS
Master - 1 CPU, 4GB RAM, 100 GB HD
Slave - 1 CPU, 2GB RAM, 100 GB HD

[Marcin Jerzak (mjerzak)]

It is also impossible to upgrade:

#update-prepare prepare master

...

Error: Package: fuel-nailgun-9.0.0-1.mos8940.noarch (mirror.fuel-infra.org_mos-repos_centos_mos9.0-centos7_snapshots_proposed-latest_x86_64_)
           Requires: python-uwsgidecorators >= 2.0.12

[Ivan Udovichenko (iudovichenko)]

For some reason mos9.0-base repository is not enabled. But actually it contains the required version of python2-openstacksdk [1]

[root@nailgun ~]# cat /etc/yum.repos.d/mos-os.repo
[mos$fuelver-base]
name=mos$fuelver-base
#baseurl=http://mirror.fuel-infra.org/mos-repos/centos/mos$fuelver-centos$releasever/os/x86_64/
mirrorlist=http://mirror.fuel-infra.org/mos-repos/centos/mos$fuelver-centos$releasever/mos-mirrors-os.txt
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/fuel-gpg/RPM-GPG-KEY-mos
skip_if_unavailable=1

Right after I turned it on, I was able to install the package (python-openstackclient) [2].

[1] http://mirror.seed-us1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/os/x86_64/Packages/python2-openstacksdk-0.8.2-1.el7.noarch.rpm
[2] http://paste.openstack.org/show/593989/

[Alexey Shtokolov (ashtokolov)]

The ISO contains the local mos9.0-base repo. This repo should be built during the ISO build. So the question is about ISO building.

[Ivan Udovichenko (iudovichenko)]

ci-team please check Alexey's comment ^^^

[Dmitry Kaigarodеsev (dkaiharodsev)]

since we're talking about 9.0-community ISO, which released in April 2016 -
all artifacts and logs from that build has been erased few month ago.

build were using following script:
https://github.com/fuel-infra/jenkins-jobs/blob/d0c3d96ebbe8f707aaf8fb2af40324f472f24621/common/scripts/all.sh

which basically does "make $make_args iso listing" by using following repo:
https://github.com/openstack/fuel-main/tree/e894a217591321d6abb4fb83347a7053f6079a13

summarizing:
unfortunately, we cannot investigate facts about how that ISO has been build, we can try to reproduce that build on https://custom-ci.infra.mirantis.net/view/9.0/job/9.0.custom.iso/ job
with specifying all related commits made on release date (Apr 21 2016)

concerns:
- not sure that we really able to fully reproduce that build
- why did we catch this issue only now?

[Alexey Shtokolov (ashtokolov)]

So we have an updated python-openstackclient from mos9.0-updates which has "python-openstacksdk >= 0.8.1" dependencies, but there is no fresh python-openstacksdk in mos9.0-updates and community ISO contains python-openstacksdk 0.7.4-1.el7 (edited)

We need to put fresh python-openstacksdk-0.8.2-1.el7 to mos9.0-updates repo as well

[Alexey Shtokolov (ashtokolov)]

> not sure that we really able to fully reproduce that build
All repos have the 9.0 tag to reproduce this built.

> why did we catch this issue only now?
I think community ISO was actively used couple months after the release (Apr2016). And new python-openstackclient was published 17-Aug-2016.

[Alexey Shtokolov (ashtokolov)]

Guys, can we put fresh python-openstacksdk-0.8.2-1.el7 to mos9.0-updates repo?

[Roman Vyalov (r0mikiam)]

Alexey, sorry but we cannot to update mos9.0-updates repository because it may affect our 9.1 release. Because of this i am proposing to enable mos9.0-base repositories as workaround

[Roman Vyalov (r0mikiam)]

also maybe we need to enable this repository as default ?

because we have a workaround I decreased the priority to High

[Alexey Shtokolov (ashtokolov)]

Marcin, could you please try the workaround Roman proposed?

[Vitaly Sedelnik (vsedelnik)]

Setting to Invalid as the bug stays in Incomplete for more than a month