7.0-build.late.artifacts job puts fuel-library6.1into fuel-docker-images

[Vadim Rovachev]

If we add proposed repo [1] to build master node and update master node then we have wrong keystone container and swarm test has failed like a [2].
If we add proposed repo [1] to build master node and before update master node we downgrade fuel-docker-images package then all ok.

[1] - http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos7.0-centos6-fuel/proposed/
[2] - https://patching-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.ha_neutron/4/consoleFull

[Denis Meltsaykin (dmeltsaykin)]

What have we found out during investigation:

* fuel-docker-images package is being built by https://patching-ci.infra.mirantis.net/view/7.0.patching/job/7.0-build.late.artifacts/
* it has no per-commit-version (i.e. all the builds have the same version 7.0.0-2)
* at some point (somewhere around the build #17) keystone container is shipped with fuel-library6.1, what is terribly wrong, as it should be fuel-library7.0 there in container
* it's very hard to determine how exactly keystone container is being built, as the output of the late-artifacts job looks like mixed because of parallelism. It looks like fuel-library is preinstalled, as all the scripts use it early from the start of the container.
* there was related change https://github.com/fuel-infra/jenkins-jobs/commit/5cd143e9ded32452dd553a1a2d471664a276b4c3 but it's still unclear wheter this CR is the reason of failure or not
* there is some amount of silly code during the build, that may behave really unexpected, like:

Step 2 : RUN rm -rf /etc/yum.repos.d/*;echo -e "[nailgun]\nname=Nailgun Local Repo\nbaseurl=http://$(route -n | awk '/^0.0.0.0/ { print $2 }'):49193/os/x86_64/\ngpgcheck=0" > /etc/yum.repos.d/nailgun.repo;yum clean expire-cache;yum update -y; echo python-fuelclient supervisor postgresql-libs | xargs -n1 yum install -y --quiet

[Vitaly Sedelnik (vsedelnik)]

Renamed and raised to Critical as this issue block executing swarm for 7.0-mu-1

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Sergey Kulanov <email address hidden>
Review: https://review.fuel-infra.org/13145

[Sergey Kulanov (skulanov)]

from logs:

https://github.com/openstack/fuel-main/blob/stable/7.0/docker/module.mk#L71

please see attached logs:

It seems we have two issues:
1)we have fuel-library7.0 in mirror, but downloading fuel-library6.1 instead(!?)
Retrieving http://127.0.0.1/os/x86_64/Packages/fuel-library6.1-6.1.0-1.mos6739.noarch.rpm ...OK
Retrieving http://127.0.0.1/os/x86_64/Packages/fuel-misc-7.0.0-7242.1.git87d3b4f.noarch.rpm ...OK

2)
error: Failed dependencies:
        system-logos is needed by (installed) plymouth-0.8.3-27.el6.centos.1.x86_64
        system-logos >= 7.92.1-1 is needed by (installed) httpd-2.2.15-39.el6.centos.x86_64
error: Failed dependencies:
        initscripts >= 8.83-1 is needed by (installed) plymouth-0.8.3-27.el6.centos.1.x86_64
        initscripts >= 5.20 is needed by (installed) openssh-5.3p1-104.el6_6.1.x86_64
        initscripts is needed by (installed) postgresql-9.3.5-1PGDG.el6.mira3.x86_64
        initscripts is needed by (installed) xinetd-2:2.3.14-39.el6_4.x86_64
        initscripts >= 8.36 is needed by (installed) httpd-2.2.15-39.el6.centos.x86_64
        initscripts >= 5.86-1 is needed by (installed) kbd-1.15-11.el6.x86_64
        initscripts >= 8.04-1 is needed by (installed) hal-0.5.14-14.el6.x86_64
        initscripts >= 8.63-1 is needed by (installed) dracut-004-356.el6_6.3.noarch
        initscripts is needed by (installed) nginx-1.0.15-4.el6.mira2.x86_64
        initscripts is needed by (installed) ruby21-puppet-3.4.2-1.el6.mira3.noarch
        initscripts is needed by (installed) openstack-keystone-2014.2.2-1.mos3851.noarch
        /sbin/service is needed by (installed) rsyslog-5.8.10-10.el6_6.x86_64
        /sbin/service is needed by (installed) postfix-2:2.6.6-6.el6_5.x86_64
        /sbin/service is needed by (installed) cronie-1.4.4-12.el6.x86_64
        /sbin/service is needed by (installed) mdadm-3.3-6.el6_6.1.x86_64
        /sbin/service is needed by (installed) xinetd-2:2.3.14-39.el6_4.x86_64
        /sbin/service is needed by (installed) tftp-server-0.49-7.el6.mira2.x86_64
        /sbin/service is needed by (installed) cobbler-2.4.4-1.el6.mira3.noar

attaching log:

I also propose to apply patch and re-check the build:
https://review.fuel-infra.org/#/c/13145/

[Sergey Kulanov (skulanov)]

Log from: https://github.com/openstack/fuel-main/blob/stable/7.0/docker/module.mk#L71

[Sergey Kulanov (skulanov)]

Folks, returning back to mos-maintenance, since we need decision/review:
1. Update job with https://review.fuel-infra.org/#/c/13145/
2. Rebuild job https://patching-ci.infra.mirantis.net/view/7.0.patching/job/7.0-build.late.artifacts/19/
3. Check artifacts for issues and then make further decision who can help us

[Sergey Kulanov (skulanov)]

btw, I also found the same artifact from the release job:
https://product-ci.infra.mirantis.net/view/7.0/job/7.0.all/301/

I hope this is exactly from this build, since no cleaning and new jobs were run on this node:

-rw-r--r-- 1 jenkins jenkins 7179086 Sep 22 20:27 fuel-centos-build.log

we have no such issues

[Sergey Kulanov (skulanov)]

So I've rebuild late artifacts without proposed mirrors:

     https://patching-ci.infra.mirantis.net/view/7.0.patching/job/7.0-build.late.artifacts/

 and it seems we resolve issue with fuel-library6.1 installation

but, we still have the same issue:
Changing password for user root.
50+0 records in
50+0 records out
25600 bytes (26 kB) copied, 0.00561767 s, 4.6 MB/s
passwd: all authentication tokens updated successfully.
Locking password for user root.
passwd: Success
error: Failed dependencies:
        system-logos is needed by (installed) plymouth-0.8.3-27.el6.centos.1.x86_64
        system-logos >= 7.92.1-1 is needed by (installed) httpd-2.2.15-39.el6.centos.x86_64
error: Failed dependencies:
        initscripts >= 8.83-1 is needed by (installed) plymouth-0.8.3-27.el6.centos.1.x86_64
        initscripts >= 5.20 is needed by (installed) openssh-5.3p1-104.el6_6.1.x86_64
        initscripts is needed by (installed) postgresql-9.3.5-1PGDG.el6.mira3.x86_64
        initscripts is needed by (installed) xinetd-2:2.3.14-39.el6_4.x86_64
        initscripts >= 8.36 is needed by (installed) httpd-2.2.15-39.el6.centos.x86_64
        initscripts >= 5.86-1 is needed by (installed) kbd-1.15-11.el6.x86_64
        initscripts >= 8.04-1 is needed by (installed) hal-0.5.14-14.el6.x86_64
        initscripts >= 8.63-1 is needed by (installed) dracut-004-356.el6_6.3.noarch
        initscripts is needed by (installed) nginx-1.0.15-4.el6.mira2.x86_64
        initscripts is needed by (installed) ruby21-puppet-3.4.2-1.el6.mira3.noarch
        initscripts is needed by (installed) openstack-keystone-2014.2.2-1.mos3851.noarch
        /sbin/service is needed by (installed) rsyslog-5.8.10-10.el6_6.x86_64
        /sbin/service is needed by (installed) postfix-2:2.6.6-6.el6_5.x86_64
        /sbin/service is needed by (installed) cronie-1.4.4-12.el6.x86_64
        /sbin/service is needed by (installed) mdadm-3.3-6.el6_6.1.x86_64
        /sbin/service is needed by (installed) xinetd-2:2.3.14-39.el6_4.x86_64
        /sbin/service is needed by (installed) tftp-server-0.49-7.el6.mira2.x86_64
        /sbin/service is needed by (installed) cobbler-2.4.4-1.el6.mira3.noarch

I've checked again and we have the same issue for release ISO, so probably this should be address in separate bug and should not be blocker for swarm.

So finally we need to investigate why with proposed mirror we are installing fuel-library6.1

[Sergey Kulanov (skulanov)]

Checked, The above stacktrace should not be critical, since this in %post section and regarding cleaning unused stuff

https://github.com/openstack/fuel-main/blob/stable/7.0/docker/fuel-centos-build/fuel-centos.ks#L170

So finally we need to investigate why with proposed mirror we are installing fuel-library6.1

[Sergey Kulanov (skulanov)]

yum always installs fuel-library6.1 instead of fuel-library7.0 because a new dependency appeared: python-fuelclient [1]

So it seems:
   Since both fuel-library6.1 and fuel-library7.0 provides the same package: `fuel-library` yum select those which has less dependencies [2]

To check, I've created pathset [3] which removes python-fuelclient and add the resulted repo [4]. As the result the correct version of the package was installed [5]. So to resolve current issue we need to use fuel-library7.0 explicitly

[1]. https://github.com/openstack/fuel-library/blob/stable/7.0/specs/fuel-library7.0.spec#L17
[2]. http://paste.openstack.org/show/477331/
[3]. https://review.openstack.org/#/c/239190/
[4]. http://perestroika-repo-tst.infra.mirantis.net/review/CR-239190/mos-repos/centos/mos7.0-centos6-fuel/proposed/x86_64/
[5]. http://paste.openstack.org/show/477332/
[6]. https://github.com/openstack/fuel-main/blob/stable/7.0/docker/fuel-centos-build/fuel-centos.ks#L75

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/239192

[Sergey Kulanov (skulanov)]

We don't have such issue for 8.0 since we don't have fuel-library in [1] and as the result there is only one package `fuel-library` in mirror

[1]. https://github.com/openstack/fuel-main/blob/master/requirements-rpm.txt

[Sergey Kulanov (skulanov)]

Fixed with proposed patchset:

Retrieving http://127.0.0.1/os/x86_64/Packages/fuel-library7.0-7.0.0-7240.1.gitad2d6b9.noarch.rpm ...OK
Retrieving http://127.0.0.1/os/x86_64/Packages/fuel-misc-7.0.0-7240.1.gitad2d6b9.noarch.rpm ...OK

you can take packages from http://patching-slave06-bud.infra.mirantis.net/fwm/7.0-build.late.artifacts/23/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13145
Submitter: Aleksandra Fedorova <email address hidden>
Branch: master

Commit: b89590dca36c49046f5a12b779289146ffc1097b
Author: Sergey Kulanov <email address hidden>
Date: Mon Oct 26 13:37:09 2015

[patching-ci] Simplify debug for build-late-artifacts job

For both 6.1 and 7.0
* Turn off parallel build
* Publish logs for "fuel/centos base image" as artifacts

Change-Id: Ib6223788b306dabfce3b4f2109e06ee127d924d2
Related-bug: #1508579

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/239192
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=6d684dfdcd79f7c1ae2a89d7cf581f9f460b238f
Submitter: Jenkins
Branch: stable/7.0

commit 6d684dfdcd79f7c1ae2a89d7cf581f9f460b238f
Author: Sergey Kulanov <email address hidden>
Date: Sat Oct 24 22:31:53 2015 +0300

    Define fuel-library7.0 package explicitly in fuel-centos.ks

    Since both fuel-library6.1 and fuel-library7.0 provides the same
    package: `fuel-library` yum select those which has less
    dependencies, in our case it's fuel-library6.1 which is wrong.

    Related-change: https://review.openstack.org/#/c/231584/

    Change-Id: I0bcb6b37e499a3d5275d1f2294de4bff0fdc42b6
    Closes-bug: #1508579