sc000 upgrade job goes N -> N instead of N-1 -> N

[Jiří Stránský]

It seems we lost correct handling of overcloud_release and/or overcloud_docker_registry_namespace in upgrade jobs. E.g. upgrade jobs on master don't deploy queens containers like they should, they already start on master.

E.g. http://logs.openstack.org/87/589487/2/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/82b2d19/logs/undercloud/home/zuul/

In upgrade jobs, overcloud_prep_containers.log.txt.gz should always download 1 release older images than upgrade_overcloud_prep_containers.log.txt.gz, but currently it downloads the same release.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

hi,

so the issue is there http://logs.openstack.org/87/589487/2/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/82b2d19/job-output.txt.gz#_2018-08-09_10_50_03_763013 ... we're skipping the task that set the proper value for deployment.

So either container_build_id is defined or use_overcloud_mixed_upgrade is false

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Wrong, just ignore my previous comment, the skip doesn't happen during deployment of the overcloud ... the real one is there [1] and it's not skipped.

[1] http://logs.openstack.org/87/589487/2/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/82b2d19/job-output.txt.gz#_2018-08-09_11_26_43_972408

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

The setting of the facts worked well actually[1], so the problem is elsewhere.

[1] http://logs.openstack.org/87/589487/2/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/82b2d19/logs/ara_oooq/result/5600258c-4eaa-458b-8ae2-bd0d6d551aa0/

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

So the problem is that we don't use the new docker_registry_namespace_used value as according we use the containers-prepare-parameter.yaml[1] which is created during undercloud deployment[2] with the assumption that overcloud and undercloud match.

We have to recreate that file during mixed upgrade:
 - when we deploy overcloud;
 - when we upgrade

[1] https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L33..L48

[2] https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-deploy/tasks/create-scripts.yml#L107..L115

[Matt Young (halcyondude)]

patch posted:

https://review.openstack.org/#/c/590912

Don't use the undercloud containers during mixed upgrade.

When doing mixed upgrade we don't want to use the container definition
file generated during undercloud deployment.

The problem is that at that time (undercloud deployment) the build_id
and docker_registry_namespace_used are (rightly) pointing to master.
But then during mixed upgrade overcloud deployment we need the N-1
version docker's images.

Change-Id: Ice13dceba7417e1ef827f4ab44fd08d2d13922fd
Closes-Bug: #1786488

[Matt Young (halcyondude)]

unclear why LP isn't being updated from commit messages, sofer's patch isn't the first I've seen this happen to in the past day

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/590912

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/590912
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=07d14c4f68f3541a55ed12b0121cc8b1a2f10e97
Submitter: Zuul
Branch: master

commit 07d14c4f68f3541a55ed12b0121cc8b1a2f10e97
Author: Sofer Athlan-Guyot <email address hidden>
Date: Fri Aug 10 18:41:50 2018 +0200

    Don't use the undercloud containers during mixed upgrade.

    When doing mixed upgrade we don't want to use the container definition
    file generated during undercloud deployment.

    The problem is that at that time (undercloud deployment) the build_id
    and docker_registry_namespace_used are (rightly) pointing to master.
    But then during mixed upgrade overcloud deployment we need the N-1
    version docker's images.

    Change-Id: Ice13dceba7417e1ef827f4ab44fd08d2d13922fd
    Closes-Bug: #1786488

[Jose Luis Franco (jfrancoa)]

Moving the LP bug into progress, as the upgrades job is still using the same docker registry release when upgrading:

-Overcloud deploy: [tripleoqueens]
http://logs.openstack.org/59/585159/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/e144942/logs/undercloud/home/zuul/overcloud-prep-containers.sh

-Overcloud upgrade: [tripleoqueens]
http://logs.openstack.org/59/585159/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/e144942/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz

[Jose Luis Franco (jfrancoa)]

Opening a new LP

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/595143

[OpenStack Infra (hudson-openstack)]

Change abandoned by Athlan-Guyot sofer (<email address hidden>) on branch: master
Review: https://review.openstack.org/595143
Reason: I don't remember if it's still need or not and the lp is kinda confusing. So let's abandon this.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.