[pike] [ci] containers-multinode-upgrades-pike job is broken due to missing puppet class.

[Sofer Athlan-Guyot]

Hi,

the jobs fails with:

    Nov 24 09:55:36 upstream-centos-7-2-node-rdo-cloud-tripleo-50567-21398 puppet-user[116213]: Evaluation Error: Error while evaluating a Function Call, Could not find class ::tripleo::profile::base::docker for upstream-centos-7-2-node-rdo-cloud-tripleo-50567-21398.localdomain at /var/lib/tripleo-config/puppet_step_config.pp:6:1 on node upstream-centos-7-2-node-rdo-cloud-tripleo-50567-21398.localdomain

(can be seen there[1])

[1] https://logs.rdoproject.org/27/516627/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z9591fed2009d4c6891dc0ac9f31ccc74/subnode-2/var/log/journal.txt.gz#_Nov_24_09_55_36

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

So digging into the log we can see that puppet-tripleo is "installed" at Nov 24 09:17:51 Installed: puppet-tripleo-6.5.6-0.20171122071003.1a0fbeb.el7.centos.noarch. But this version doesn't have the manifest/profile/base/docker.pp file hence the error.

Still need to determine why this version.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

So for the moment I would say that the version of puppet installed is not the right. Who could help debug the repo setup and package delivered ?

[Jose Luis Franco (jfrancoa)]

I have the feeling, that the source of the issue has to do something with this patch, which was merged on 22nd Nov: https://review.openstack.org/#/c/519407/

In fact, if you check the logs from the 21st, the job fails for a different reason: https://logs.rdoproject.org/13/522013/1/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Za68238e87e4246ecb7f8850cc2674f29/undercloud/home/jenkins/failed_upgrade.log.txt.gz

While, the first job execution after the patch got merged (22/11 @ 06:53) starts failing because it can't find the docker puppet class: https://logs.rdoproject.org/01/521501/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Zd58d2a50ccdd425a86b2fd8bd31dafaa/undercloud/home/jenkins/failed_upgrade.log.txt.gz

When comparing both journal logs, we can see that one updates the packages, while the other doesn't:

Journal 22/11 @ 02:46 ([Update all packages] task ran)
------------------------------------------------------
https://logs.rdoproject.org/43/522043/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z081f531e693a49d6a99eab9e64660a70/subnode-2/var/log
/journal.txt.gz#_Nov_22_01_42_28

Journal 22/11 @ 14:05 (No [Update all packages] task ran)
---------------------------------------------------------
https://logs.rdoproject.org/01/521501/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Zd58d2a50ccdd425a86b2fd8bd31dafaa/subnode-2/var/log/journal.txt.gz#_Nov_22_12_12_53

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Hi,

so we can see that the generated playbook doesn't have the yum upgrade task on the host[1][2]. This prevent the puppet-tripleo package to be upgraded and causes the error.

[1] the "Update all packages" task here : https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/puppet/services/tripleo-packages.yaml#L86..L87
[2] rendered playbook: https://logs.rdoproject.org/13/524113/1/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z292150e9a99d404e8e8c102d6453e7be/subnode-2/var/lib/heat-config/heat-config-ansible/4c286b96-7418-47b1-b042-d92ab35c0d97_playbook.yaml.txt.gz (best view with jq ....)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/524258

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/524259

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

We can see the yum update package reintroduced during upgrade testing there[1] and the actual upgrade there[2]

So that patch fix the issue. Review that tested it is[3]

[1] https://logs.rdoproject.org/08/523408/6/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z144e9e1f762c4aeba5d375dca6ef5631/subnode-2/var/lib/heat-config/heat-config-ansible/899b4849-b5f0-44b8-a27e-28937bb148f0_playbook.yaml.txt.gz

[2] https://logs.rdoproject.org/08/523408/6/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z144e9e1f762c4aeba5d375dca6ef5631/subnode-2/var/log/yum.log.txt.gz

[3] https://review.openstack.org/#/c/523408/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/524258
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ba273f23a9d698c9b6f2c7b802a83ee78ebe0ffa
Submitter: Zuul
Branch: master

commit ba273f23a9d698c9b6f2c7b802a83ee78ebe0ffa
Author: Sofer Athlan-Guyot <email address hidden>
Date: Thu Nov 30 18:09:50 2017 +0100

    Reintroduce TripleoPackages in the default Controller service.

    Without this service, no package upgrade can happen on the host. This
    breaks upgrade ci. For instance the puppet modules (for instance) are not
    upgraded at all, neither is os-net-config and so on…

    Change-Id: Ic050bd58065904f1c55d6a71c7bd4b09ba72d19d
    Closes-Bug: #1734353

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/524259
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=338c78367a217f849b511ec8f2360b5bf64d52b5
Submitter: Zuul
Branch: stable/pike

commit 338c78367a217f849b511ec8f2360b5bf64d52b5
Author: Sofer Athlan-Guyot <email address hidden>
Date: Thu Nov 30 18:09:50 2017 +0100

    Reintroduce TripleoPackages in the default Controller service.

    Without this service, no package upgrade can happen on the host. This
    breaks upgrade ci. For instance the puppet modules (for instance) are not
    upgraded at all, neither is os-net-config and so on…

    Change-Id: Ic050bd58065904f1c55d6a71c7bd4b09ba72d19d
    Closes-Bug: #1734353
    (cherry picked from commit ba273f23a9d698c9b6f2c7b802a83ee78ebe0ffa)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b3 development milestone.