containerized UC upgrade job failing on container_cli discrepancy

[Jiří Stránský]

During upgrade container_cli can be Podman but containers are still running on Docker, and the upgrade fails on the catch-all online data migration task:

2018-11-06 11:35:41 | TASK [Ensure all online data migrations for Ironic have been applied] **********
2018-11-06 11:35:42 | fatal: [centos-7-vexxhost-sjc1-0000362761]: FAILED! => {"changed": true, "cmd": ["podman", "exec", "ironic_api", "ironic-dbsync", "--config-file", "/etc/ironic/ironic.conf", "online_data_migrations"], "delta": "0:00:00.455726", "end": "2018-11-06 11:35:42.224177", "msg": "non-zero return code", "rc": 125, "start": "2018-11-06 11:35:41.768451", "stderr": "unable to exec into ironic_api: no container with name or ID ironic_api found: no such container", "stderr_lines": ["unable to exec into ironic_api: no container with name or ID ironic_api found: no such container"], "stdout": "", "stdout_lines": []}

This currently breaks the undercloud containerized upgrade CI job.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/616146

[Emilien Macchi (emilienm)]

Removing the alert, the job is non voting.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/616146
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=978c5978ae92bf9ec60b7c25d5e9a6d305c55421
Submitter: Zuul
Branch: master

commit 978c5978ae92bf9ec60b7c25d5e9a6d305c55421
Author: Jiri Stransky <email address hidden>
Date: Wed Nov 7 11:19:34 2018 +0100

    Handle difference between future and current container_cli

    During upgrade we may have container_cli be Podman but the containers
    may still be running on Docker. Handle this situation in the upgrade
    tasks which are the last-resort online data migration if user forgot
    to trigger them earlier, as they seem to be hitting this issue.

    We must support both options at the same time, because the upgrade
    code must be idempotent (re-runnable). When running upgrade 1st time,
    the containers will be running in Docker, when re-running the upgrade
    (e.g. because a part of it failed), the containers will be running in
    Podman.

    Once we converge onto a single solution and do not have to support
    migration, this commit can be reverted.

    Change-Id: I933ce754f081ee87ec53d5f8d9c901ab71dceb1e
    Closes-Bug: #1802085

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.2.0 release.