rocky tripleo-ci-centos-7-undercloud-upgrades fail "container-update : Check if containers-prepare-parameter.yaml exists"

[Marios Andreou]

The tripleo-ci-centos-7-undercloud-upgrades job is failing at [1][2][3] (many examples [4]) during the upgrade with trace like:

        * 2019-12-05 18:43:18.514385 | primary | TASK [container-update : Check if containers-prepare-parameter.yaml exists] ****
        2019-12-05 18:43:18.560253 | primary | Thursday 05 December 2019 18:43:18 +0000 (0:00:00.145) 0:04:12.649 *****
        2019-12-05 18:43:19.121018 | primary | fatal: [undercloud]: FAILED! => {
        2019-12-05 18:43:19.121176 | primary | "changed": false,
        2019-12-05 18:43:19.121256 | primary | "failed_when_result": true,
        2019-12-05 18:43:19.121306 | primary | "stat": {
        2019-12-05 18:43:19.121368 | primary | "exists": false
        2019-12-05 18:43:19.121404 | primary | }
        2019-12-05 18:43:19.121434 | primary | }

Looks to be related to the fixes made for https://launchpad.net/bugs/1853812 in particular see comment at [5]
This blocks the rocky gate

[1] https://01349028e7e7f00f1019-2920023e186e60bebe7fd9c4edba4c4d.ssl.cf2.rackcdn.com/697334/1/check/tripleo-ci-centos-7-undercloud-upgrades/36978cb/job-output.txt
[2] https://cdd8793c5a5b75d5a8cc-cf05ec96aa1131dfad9173080b1236b7.ssl.cf2.rackcdn.com/697334/1/check/tripleo-ci-centos-7-undercloud-upgrades/297adea/job-output.txt
[3] https://becfa8f6521da15baf66-344a6ad21c722e33f9112420ebbef28e.ssl.cf1.rackcdn.com/696847/1/check/tripleo-ci-centos-7-undercloud-upgrades/012b7d7/job-output.txt
[4] http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-undercloud-upgrades&branch=stable%2Frocky
[5] https://review.opendev.org/#/c/695234/6/config/general_config/featureset050.yml

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/698223

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/698225

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/698227

[Carlos Camacho (ccamacho)]

Hey Jose Luis I started to check this https://review.opendev.org/#/c/698219/ and the environment is using actually fs47, there is another problem which is the validation you submited to render the container file if not present.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jose Luis Franco (<email address hidden>) on branch: master
Review: https://review.opendev.org/698227
Reason: failing job uses fs47, being addressed here https://review.opendev.org/#/c/698219/

[Carlos Camacho (ccamacho)]

Hello, here you have the patch testing the undercloud upgrade passing https://review.opendev.org/#/c/698220/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698223
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=66ce22d6ede34330f59cf4eb3f6bd4f76c68db9a
Submitter: Zuul
Branch: master

commit 66ce22d6ede34330f59cf4eb3f6bd4f76c68db9a
Author: Jose Luis Franco Arza <email address hidden>
Date: Tue Dec 10 13:04:07 2019 +0100

    Add containerized_undercloud_upgrade variable.

    When upgrading from Queens to Rocky, we are jumping from a
    non-containerized undercloud into a containerized one.
    However, this scenario hasn't been considered yet. When
    submitting [0] we add the logic to upgrade always to a
    containerized undercloud, we need to have a variable that
    allows us differentiate between the type (containerized or
    non-containrized) of undercloud we deploy and upgrade to.

    [0] - Ibcdfd71b9280615aedc4d59ef0c566c475f15d0b

    Related-Bug: #1855856

    Change-Id: I30b4789adcf348c636d3679523075b22f8956ff1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698225
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=0de4ed518d3c41c951cb193a555086d9a8344a58
Submitter: Zuul
Branch: master

commit 0de4ed518d3c41c951cb193a555086d9a8344a58
Author: Jose Luis Franco Arza <email address hidden>
Date: Tue Dec 10 13:21:45 2019 +0100

    Render containerized-prepare-parameters.yaml during upgrade if not found.

    Generally, if we upgrade from a containerized undercloud the
    containers-prepare-paramters.yaml file will be already present (as
    it is being rendered during the undercloud-deploy). However, if we
    upgrade the Undercloud from Queens (non-containerized) to Rocky
    (containerized) the file won't be present. We need to render it
    during the container-update role so we can set the right values.

    Change-Id: I3fd411ef753bdd1f4ce5d6befa726d82e9073458
    Related-Bug: #1855856

[Marios Andreou (marios-b)]

almost done just waiting for https://review.opendev.org/#/c/698219/ to clear the gate

[Marios Andreou (marios-b)]

damn gate now blocked on https://bugs.launchpad.net/tripleo/+bug/1856520

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698219
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=5589386151d268a2a74fd481c1420cefc66d57dd
Submitter: Zuul
Branch: master

commit 5589386151d268a2a74fd481c1420cefc66d57dd
Author: Carlos Camacho <email address hidden>
Date: Tue Dec 10 12:25:59 2019 +0100

    Configure UC rocky upgrade vars

    We are using fs47 to run the UC upgrade without containers.
    This should change depending on the version.

    Depends-On: https://review.opendev.org/#/c/698225
    Closes-Bug: #1855856

    Change-Id: Iae6d1a331e55b5f56f667e7bf84d79bf7e01f3c6

[Marios Andreou (marios-b)]

ok should be good now and we have some green runs at http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-undercloud-upgrades&branch=stable%2Frocky

bug is already @ fix-released