7.0-build.late.artifacts job puts fuel-library6.1into fuel-docker-images

Bug #1508579 reported by Vadim Rovachev on 2015-10-21
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Critical
Sergey Kulanov

Bug Description

If we add proposed repo [1] to build master node and update master node then we have wrong keystone container and swarm test has failed like a [2].
If we add proposed repo [1] to build master node and before update master node we downgrade fuel-docker-images package then all ok.

[1] - http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos7.0-centos6-fuel/proposed/
[2] - https://patching-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.ha_neutron/4/consoleFull

tags: added: non-release system-tests
Changed in fuel:
importance: Undecided → High
milestone: none → 7.0-updates
Changed in fuel:
milestone: 7.0-updates → 7.0-mu-1
summary: - Sworm tests 7.0 for updates has failed because of fuel-docker-images
+ Swarm tests 7.0 for updates has failed because of fuel-docker-images
package
description: updated
Changed in fuel:
status: New → Confirmed
assignee: nobody → MOS Maintenance (mos-maintenance)

What have we found out during investigation:

* fuel-docker-images package is being built by https://patching-ci.infra.mirantis.net/view/7.0.patching/job/7.0-build.late.artifacts/
* it has no per-commit-version (i.e. all the builds have the same version 7.0.0-2)
* at some point (somewhere around the build #17) keystone container is shipped with fuel-library6.1, what is terribly wrong, as it should be fuel-library7.0 there in container
* it's very hard to determine how exactly keystone container is being built, as the output of the late-artifacts job looks like mixed because of parallelism. It looks like fuel-library is preinstalled, as all the scripts use it early from the start of the container.
* there was related change https://github.com/fuel-infra/jenkins-jobs/commit/5cd143e9ded32452dd553a1a2d471664a276b4c3 but it's still unclear wheter this CR is the reason of failure or not
* there is some amount of silly code during the build, that may behave really unexpected, like:

Step 2 : RUN rm -rf /etc/yum.repos.d/*;echo -e "[nailgun]\nname=Nailgun Local Repo\nbaseurl=http://$(route -n | awk '/^0.0.0.0/ { print $2 }'):49193/os/x86_64/\ngpgcheck=0" > /etc/yum.repos.d/nailgun.repo;yum clean expire-cache;yum update -y; echo python-fuelclient supervisor postgresql-libs | xargs -n1 yum install -y --quiet

Vitaly Sedelnik (vsedelnik) wrote :

Renamed and raised to Critical as this issue block executing swarm for 7.0-mu-1

summary: - Swarm tests 7.0 for updates has failed because of fuel-docker-images
- package
+ 7.0-build.late.artifacts job puts fuel-library6.1into fuel-docker-images
Changed in fuel:
assignee: MOS Maintenance (mos-maintenance) → Fuel CI team (fuel-ci)
importance: High → Critical

Related fix proposed to branch: master
Change author: Sergey Kulanov <email address hidden>
Review: https://review.fuel-infra.org/13145

Changed in fuel:
status: Confirmed → In Progress
Changed in fuel:
assignee: Fuel CI team (fuel-ci) → Sergey Kulanov (skulanov)
Sergey Kulanov (skulanov) wrote :

from logs:

https://github.com/openstack/fuel-main/blob/stable/7.0/docker/module.mk#L71

please see attached logs:

It seems we have two issues:
1)we have fuel-library7.0 in mirror, but downloading fuel-library6.1 instead(!?)
Retrieving http://127.0.0.1/os/x86_64/Packages/fuel-library6.1-6.1.0-1.mos6739.noarch.rpm ...OK
Retrieving http://127.0.0.1/os/x86_64/Packages/fuel-misc-7.0.0-7242.1.git87d3b4f.noarch.rpm ...OK

2)
error: Failed dependencies:
        system-logos is needed by (installed) plymouth-0.8.3-27.el6.centos.1.x86_64
        system-logos >= 7.92.1-1 is needed by (installed) httpd-2.2.15-39.el6.centos.x86_64
error: Failed dependencies:
        initscripts >= 8.83-1 is needed by (installed) plymouth-0.8.3-27.el6.centos.1.x86_64
        initscripts >= 5.20 is needed by (installed) openssh-5.3p1-104.el6_6.1.x86_64
        initscripts is needed by (installed) postgresql-9.3.5-1PGDG.el6.mira3.x86_64
        initscripts is needed by (installed) xinetd-2:2.3.14-39.el6_4.x86_64
        initscripts >= 8.36 is needed by (installed) httpd-2.2.15-39.el6.centos.x86_64
        initscripts >= 5.86-1 is needed by (installed) kbd-1.15-11.el6.x86_64
        initscripts >= 8.04-1 is needed by (installed) hal-0.5.14-14.el6.x86_64
        initscripts >= 8.63-1 is needed by (installed) dracut-004-356.el6_6.3.noarch
        initscripts is needed by (installed) nginx-1.0.15-4.el6.mira2.x86_64
        initscripts is needed by (installed) ruby21-puppet-3.4.2-1.el6.mira3.noarch
        initscripts is needed by (installed) openstack-keystone-2014.2.2-1.mos3851.noarch
        /sbin/service is needed by (installed) rsyslog-5.8.10-10.el6_6.x86_64
        /sbin/service is needed by (installed) postfix-2:2.6.6-6.el6_5.x86_64
        /sbin/service is needed by (installed) cronie-1.4.4-12.el6.x86_64
        /sbin/service is needed by (installed) mdadm-3.3-6.el6_6.1.x86_64
        /sbin/service is needed by (installed) xinetd-2:2.3.14-39.el6_4.x86_64
        /sbin/service is needed by (installed) tftp-server-0.49-7.el6.mira2.x86_64
        /sbin/service is needed by (installed) cobbler-2.4.4-1.el6.mira3.noar

attaching log:

I also propose to apply patch and re-check the build:
https://review.fuel-infra.org/#/c/13145/

Sergey Kulanov (skulanov) wrote :

Folks, returning back to mos-maintenance, since we need decision/review:
1. Update job with https://review.fuel-infra.org/#/c/13145/
2. Rebuild job https://patching-ci.infra.mirantis.net/view/7.0.patching/job/7.0-build.late.artifacts/19/
3. Check artifacts for issues and then make further decision who can help us

Changed in fuel:
assignee: Sergey Kulanov (skulanov) → MOS Maintenance (mos-maintenance)
status: In Progress → Confirmed
Sergey Kulanov (skulanov) wrote :

btw, I also found the same artifact from the release job:
https://product-ci.infra.mirantis.net/view/7.0/job/7.0.all/301/

I hope this is exactly from this build, since no cleaning and new jobs were run on this node:

-rw-r--r-- 1 jenkins jenkins 7179086 Sep 22 20:27 fuel-centos-build.log

we have no such issues

Sergey Kulanov (skulanov) wrote :

So I've rebuild late artifacts without proposed mirrors:

     https://patching-ci.infra.mirantis.net/view/7.0.patching/job/7.0-build.late.artifacts/

 and it seems we resolve issue with fuel-library6.1 installation

but, we still have the same issue:
Changing password for user root.
50+0 records in
50+0 records out
25600 bytes (26 kB) copied, 0.00561767 s, 4.6 MB/s
passwd: all authentication tokens updated successfully.
Locking password for user root.
passwd: Success
error: Failed dependencies:
        system-logos is needed by (installed) plymouth-0.8.3-27.el6.centos.1.x86_64
        system-logos >= 7.92.1-1 is needed by (installed) httpd-2.2.15-39.el6.centos.x86_64
error: Failed dependencies:
        initscripts >= 8.83-1 is needed by (installed) plymouth-0.8.3-27.el6.centos.1.x86_64
        initscripts >= 5.20 is needed by (installed) openssh-5.3p1-104.el6_6.1.x86_64
        initscripts is needed by (installed) postgresql-9.3.5-1PGDG.el6.mira3.x86_64
        initscripts is needed by (installed) xinetd-2:2.3.14-39.el6_4.x86_64
        initscripts >= 8.36 is needed by (installed) httpd-2.2.15-39.el6.centos.x86_64
        initscripts >= 5.86-1 is needed by (installed) kbd-1.15-11.el6.x86_64
        initscripts >= 8.04-1 is needed by (installed) hal-0.5.14-14.el6.x86_64
        initscripts >= 8.63-1 is needed by (installed) dracut-004-356.el6_6.3.noarch
        initscripts is needed by (installed) nginx-1.0.15-4.el6.mira2.x86_64
        initscripts is needed by (installed) ruby21-puppet-3.4.2-1.el6.mira3.noarch
        initscripts is needed by (installed) openstack-keystone-2014.2.2-1.mos3851.noarch
        /sbin/service is needed by (installed) rsyslog-5.8.10-10.el6_6.x86_64
        /sbin/service is needed by (installed) postfix-2:2.6.6-6.el6_5.x86_64
        /sbin/service is needed by (installed) cronie-1.4.4-12.el6.x86_64
        /sbin/service is needed by (installed) mdadm-3.3-6.el6_6.1.x86_64
        /sbin/service is needed by (installed) xinetd-2:2.3.14-39.el6_4.x86_64
        /sbin/service is needed by (installed) tftp-server-0.49-7.el6.mira2.x86_64
        /sbin/service is needed by (installed) cobbler-2.4.4-1.el6.mira3.noarch

I've checked again and we have the same issue for release ISO, so probably this should be address in separate bug and should not be blocker for swarm.

So finally we need to investigate why with proposed mirror we are installing fuel-library6.1

Sergey Kulanov (skulanov) wrote :

Checked, The above stacktrace should not be critical, since this in %post section and regarding cleaning unused stuff

https://github.com/openstack/fuel-main/blob/stable/7.0/docker/fuel-centos-build/fuel-centos.ks#L170

So finally we need to investigate why with proposed mirror we are installing fuel-library6.1

Changed in fuel:
assignee: MOS Maintenance (mos-maintenance) → Sergey Kulanov (skulanov)
status: Confirmed → In Progress
Sergey Kulanov (skulanov) wrote :

yum always installs fuel-library6.1 instead of fuel-library7.0 because a new dependency appeared: python-fuelclient [1]

So it seems:
   Since both fuel-library6.1 and fuel-library7.0 provides the same package: `fuel-library` yum select those which has less dependencies [2]

To check, I've created pathset [3] which removes python-fuelclient and add the resulted repo [4]. As the result the correct version of the package was installed [5]. So to resolve current issue we need to use fuel-library7.0 explicitly

[1]. https://github.com/openstack/fuel-library/blob/stable/7.0/specs/fuel-library7.0.spec#L17
[2]. http://paste.openstack.org/show/477331/
[3]. https://review.openstack.org/#/c/239190/
[4]. http://perestroika-repo-tst.infra.mirantis.net/review/CR-239190/mos-repos/centos/mos7.0-centos6-fuel/proposed/x86_64/
[5]. http://paste.openstack.org/show/477332/
[6]. https://github.com/openstack/fuel-main/blob/stable/7.0/docker/fuel-centos-build/fuel-centos.ks#L75

Sergey Kulanov (skulanov) wrote :

We don't have such issue for 8.0 since we don't have fuel-library in [1] and as the result there is only one package `fuel-library` in mirror

[1]. https://github.com/openstack/fuel-main/blob/master/requirements-rpm.txt

Reviewed: https://review.fuel-infra.org/13145
Submitter: Aleksandra Fedorova <email address hidden>
Branch: master

Commit: b89590dca36c49046f5a12b779289146ffc1097b
Author: Sergey Kulanov <email address hidden>
Date: Mon Oct 26 13:37:09 2015

[patching-ci] Simplify debug for build-late-artifacts job

For both 6.1 and 7.0
* Turn off parallel build
* Publish logs for "fuel/centos base image" as artifacts

Change-Id: Ib6223788b306dabfce3b4f2109e06ee127d924d2
Related-bug: #1508579

Reviewed: https://review.openstack.org/239192
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=6d684dfdcd79f7c1ae2a89d7cf581f9f460b238f
Submitter: Jenkins
Branch: stable/7.0

commit 6d684dfdcd79f7c1ae2a89d7cf581f9f460b238f
Author: Sergey Kulanov <email address hidden>
Date: Sat Oct 24 22:31:53 2015 +0300

    Define fuel-library7.0 package explicitly in fuel-centos.ks

    Since both fuel-library6.1 and fuel-library7.0 provides the same
    package: `fuel-library` yum select those which has less
    dependencies, in our case it's fuel-library6.1 which is wrong.

    Related-change: https://review.openstack.org/#/c/231584/

    Change-Id: I0bcb6b37e499a3d5275d1f2294de4bff0fdc42b6
    Closes-bug: #1508579

Changed in fuel:
status: In Progress → Fix Committed
Changed in fuel:
milestone: 7.0-mu-1 → 7.0-updates
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers