invalid arguments you must use just one container during upgrade

[Grzegorz Grasza]

A queens to train fast forward upgrade fails on one of the controllers with the following:

2020-06-19 06:05:34 | TASK [restart pacemaker resource for haproxy] **********************************
2020-06-19 06:05:34 | Friday 19 June 2020 06:05:33 -0400 (0:00:00.241) 0:01:48.907 ***********
2020-06-19 06:05:34 | skipping: [controller-0] => {"changed": false, "skip_reason": "Conditional result was False"}
2020-06-19 06:05:34 | skipping: [controller-1] => {"changed": false, "skip_reason": "Conditional result was False"}
2020-06-19 06:05:34 | skipping: [controller-2] => {"changed": false, "skip_reason": "Conditional result was False"}
2020-06-19 06:05:34 |
2020-06-19 06:05:34 | TASK [copy certificate from host to container] *********************************
2020-06-19 06:05:34 | Friday 19 June 2020 06:05:33 -0400 (0:00:00.236) 0:01:49.144 ***********
2020-06-19 06:05:34 | skipping: [controller-2] => {"changed": false, "skip_reason": "Conditional result was False"}
2020-06-19 06:05:34 | fatal: [controller-1]: FAILED! => {"changed": true, "cmd": "podman cp /etc/pki/tls/private/overcloud_endpoint.pem ad9612da1a4c\n0d7665445110:/etc/pki/tls/private/overcloud_endpoint.pem", "delta": "0:00:00.098841", "end": "2020-06-19 10:05:34.085772", "msg": "non-zero return code", "rc": 127, "start": "2020-06-19 10:05:33.986931", "stderr": "Error: invalid arguments /etc/pki/tls/private/overcloud_endpoint.pem, ad9612da1a4c you must use just one container\n/bin/sh: line 1: 0d7665445110:/etc/pki/tls/private/overcloud_endpoint.pem: No such file or directory", "stderr_lines": ["Error: invalid arguments /etc/pki/tls/private/overcloud_endpoint.pem, ad9612da1a4c you must use just one container", "/bin/sh: line 1: 0d7665445110:/etc/pki/tls/private/overcloud_endpoint.pem: No such file or directory"], "stdout": "", "stdout_lines": []}
2020-06-19 06:05:34 | changed: [controller-0] => {"changed": true, "cmd": "podman cp /etc/pki/tls/private/overcloud_endpoint.pem 3b4bafa53b98:/etc/pki/tls/private/overcloud_endpoint.pem", "delta": "0:00:00.473766", "end": "2020-06-19 10:05:34.391347", "rc": 0, "start": "2020-06-19 10:05:33.917581", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
2020-06-19 06:05:34 |
2020-06-19 06:05:34 | NO MORE HOSTS LEFT *************************************************************
2020-06-19 06:05:34 |
2020-06-19 06:05:34 | PLAY RECAP *********************************************************************
2020-06-19 06:05:34 | controller-0 : ok=103 changed=24 unreachable=0 failed=0 skipped=51 rescued=0 ignored=0
2020-06-19 06:05:34 | controller-1 : ok=95 changed=23 unreachable=0 failed=1 skipped=51 rescued=0 ignored=0
2020-06-19 06:05:34 | controller-2 : ok=95 changed=23 unreachable=0 failed=0 skipped=52 rescued=0 ignored=0
2020-06-19 06:05:34 |
2020-06-19 06:05:34 | Friday 19 June 2020 06:05:34 -0400 (0:00:00.791) 0:01:49.935 ***********

More logs available in https://bugzilla.redhat.com/show_bug.cgi?id=1849211

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/737486

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/738756

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738756
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=51e697362557947cd2def74b34b17aeec28bf94b
Submitter: Zuul
Branch: stable/ussuri

commit 51e697362557947cd2def74b34b17aeec28bf94b
Author: Grzegorz Grasza <email address hidden>
Date: Mon Jun 22 12:08:19 2020 +0200

    Fix Error: invalid arguments you must use just one container

    The podman ps command can return multiple containers, which
    might happen during an upgrade, when an old container lingers on.
    We want to copy the cert into all of them.

    Change-Id: I335b75feb9b01fa0e6a80d24b8c6ea79ca6480f9
    Closes-Bug: #1884731
    Resolves: rhbz#1849211
    (cherry picked from commit d37579fd787185b292b5d547f5ac3cf17dd2fd9e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/737249
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d37579fd787185b292b5d547f5ac3cf17dd2fd9e
Submitter: Zuul
Branch: master

commit d37579fd787185b292b5d547f5ac3cf17dd2fd9e
Author: Grzegorz Grasza <email address hidden>
Date: Mon Jun 22 12:08:19 2020 +0200

    Fix Error: invalid arguments you must use just one container

    The podman ps command can return multiple containers, which
    might happen during an upgrade, when an old container lingers on.
    We want to copy the cert into all of them.

    Change-Id: I335b75feb9b01fa0e6a80d24b8c6ea79ca6480f9
    Closes-Bug: #1884731
    Resolves: rhbz#1849211

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/737486
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4ffe8c3cc1b7038e471bffddbc53818ef1813f5e
Submitter: Zuul
Branch: stable/train

commit 4ffe8c3cc1b7038e471bffddbc53818ef1813f5e
Author: Grzegorz Grasza <email address hidden>
Date: Mon Jun 22 12:08:19 2020 +0200

    Fix Error: invalid arguments you must use just one container

    The podman ps command can return multiple containers, which
    might happen during an upgrade, when an old container lingers on.
    We want to copy the cert into all of them.

    Change-Id: I335b75feb9b01fa0e6a80d24b8c6ea79ca6480f9
    Closes-Bug: #1884731
    Resolves: rhbz#1849211
    (cherry picked from commit d37579fd787185b292b5d547f5ac3cf17dd2fd9e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.