centos-8 multinode-oooq-container-updates fails during converge "Failed containers: keystone_bootstrap"

[Marios Andreou]

At [1][2][3] the tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is failing during the converge step with trace like

 2021-06-17 01:55:21 | 2021-06-17 01:55:21.379561 | | WARNING | Failure running exec 'keystone_bootstrap'. rc=255, stdout=, stderr=Error: can only create exec sessions on running containers: container state improper
 2021-06-17 01:55:21 | 2021-06-17 01:55:21.381411 | bc764e10-1505-d63c-eddc-0000000013f6 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 | centos-8-stream-rax-ord-0025151891 | error={"changed": false, "msg": "Failed containers: keystone_bootstrap"}

This has started happening within the last 12 hours or so [4], so it must be something that merged very recently (and where we aren't gating with the update job, e.g. tripleo-ansible but I have not yet confirmed this). This is a master gate blocker. Last successful runs from yesterday there [5]

[1] https://98f884550bb898129005-dc55dfdaf5877a4d04198310cb0fd46a.ssl.cf1.rackcdn.com/796712/1/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/36713ad/logs/undercloud/home/zuul/overcloud_update_converge.log
[2] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_467/778915/40/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/467d23f/logs/undercloud/home/zuul/overcloud_update_converge.log
[3] https://0cbbb3ecadec96097039-d3b9488846fd47e876cca24d70452d35.ssl.cf1.rackcdn.com/796531/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/bfe3551/logs/undercloud/home/zuul/overcloud_update_converge.log
[4] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates
[5] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates&result=SUCCESS

[Marios Andreou (marios-b)]

Not confirmed yet but I suspect this may have started with https://review.opendev.org/c/openstack/tripleo-ansible/+/791317 Container manage module

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/796805

[Marios Andreou (marios-b)]

posted https://review.opendev.org/c/openstack/tripleo-ansible/+/796805 to wire up the minor update job to run in tripleo-ansible

[Marios Andreou (marios-b)]

also adding the promotion-blocker tag so it will be tracked as CIX it isn't currently on the board

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/796857

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/796857
Committed: https://opendev.org/openstack/tripleo-ansible/commit/6a24ca732529b282315da114273f6f79001d0a44
Submitter: "Zuul (22348)"
Branch: master

commit 6a24ca732529b282315da114273f6f79001d0a44
Author: Alex Schultz <email address hidden>
Date: Thu Jun 17 13:28:41 2021 +0000

    Revert "Container manage module"

    This reverts commit 22bbe1311094be600b9fed426f1ee0a983b8e111.

    This appears to have broken updates. Let's revert for now.

    Change-Id: I680b38643151a2a16205e15edd3ea8703214a2d8
    Related-Bug: #1932261

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/796805
Committed: https://opendev.org/openstack/tripleo-ansible/commit/e8570786776612deac68e332b51331fac3a755e1
Submitter: "Zuul (22348)"
Branch: master

commit e8570786776612deac68e332b51331fac3a755e1
Author: Marios Andreou <email address hidden>
Date: Thu Jun 17 12:07:41 2021 +0300

    Add minor update job to tripleo-ansible zuul layout

    This wires up the master upgrades template so we will gate on
    the minor update job in tripleo-ansible to prevent issues like
    related-bug.

    Related-Bug: 1932261
    Change-Id: I3711d0ef8cf8daecde2f9593bf3b8214ddc647ca

[Marios Andreou (marios-b)]

revert merged and the job is back to green https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates thanks for fast actions to clear this