During upgrade_tasks on bootstrap node ansible is unable to find ceph-mon if controller has FQDN hostname

[Francesco Pantano]

When running the the upgrade_tasks and post_upgrade_tasks for ceph during FFU upgrade from queens to train will if ceph-mon-${HOSTNAME} does not match the containers name during system_upgrade on the bootstrap node.
In a setup integrated with IdM this can be a FQDN, while the default and supported container name for ceph uses shortnames.

Here the output:

2020-10-23 04:31:40,455 p=5846 u=mistral n=ansible | failed: [controller-0 -> 192.168.24.33] (item=nobackfill) => {"ansible_loop_var": "item", "changed": true, "cmd": "docker exec -u root ceph-mon-${HOSTNAME} ceph osd set nobackfill", "delta": "0:00:00.025770", "end": "2020-10-23 04:31:40.395243", "item": "nobackfill", "msg": "non-zero return code", "rc": 1, "start": "2020-10-23 04:31:40.369473", "stderr": "Error response from daemon: No such container: ceph-mon-controller-0.redhat.local", "stderr_lines": ["Error response from daemon: No such container: ceph-mon-controller-0.redhat.local"], "stdout": "", "stdout_lines": []}
2020-10-23 04:31:40,658 p=5846 u=mistral n=ansible | failed: [controller-0 -> 192.168.24.33] (item=norebalance) => {"ansible_loop_var": "item", "changed": true, "cmd": "docker exec -u root ceph-mon-${HOSTNAME} ceph osd set norebalance", "delta": "0:00:00.028196", "end": "2020-10-23 04:31:40.612835", "item": "norebalance", "msg": "non-zero return code", "rc": 1, "start": "2020-10-23 04:31:40.584639", "stderr": "Error response from daemon: No such container: ceph-mon-controller-0.redhat.local", "stderr_lines": ["Error response from daemon: No such container: ceph-mon-controller-0.redhat.local"], "stdout": "", "stdout_lines": []}
2020-10-23 04:31:40,860 p=5846 u=mistral n=ansible | failed: [controller-0 -> 192.168.24.33] (item=nodeep-scrub) => {"ansible_loop_var": "item", "changed": true, "cmd": "docker exec -u root ceph-mon-${HOSTNAME} ceph osd set nodeep-scrub", "delta": "0:00:00.025621", "end": "2020-10-23 04:31:40.821615", "item": "nodeep-scrub", "msg": "non-zero return code", "rc": 1, "start": "2020-10-23 04:31:40.795994", "stderr": "Error response from daemon: No such container: ceph-mon-controller-0.redhat.local", "stderr_lines": ["Error response from daemon: No such container: ceph-mon-controller-0.redhat.local"], "stdout": "", "stdout_lines": []}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/761930

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/761930
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4e0f61d9e2affd107b0ac9a195e23f2a71083383
Submitter: Zuul
Branch: master

commit 4e0f61d9e2affd107b0ac9a195e23f2a71083383
Author: Francesco Pantano <email address hidden>
Date: Mon Nov 9 14:39:40 2020 +0100

    Properly compute hostname when looking for the ceph-mon container

    There are some cases when the return value of the ${HOSTNAME) variable
    is different from {{ ansible_hostname }}, especially when tls-everywhere
    framework is involved.
    This purpose of this review is to fix the FFU section that needs to know
    the hostname to find the running container on the delegated host.

    Change-Id: Ib269a25fcf0db66ad3144c3e20031a4da6e2e2fb
    Closes-Bug: #1903537

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/762283

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/762284

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/762285

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762283
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7ffaf82e7dd02132e28c9092becb7e9a973a95a2
Submitter: Zuul
Branch: stable/victoria

commit 7ffaf82e7dd02132e28c9092becb7e9a973a95a2
Author: Francesco Pantano <email address hidden>
Date: Mon Nov 9 14:39:40 2020 +0100

    Properly compute hostname when looking for the ceph-mon container

    There are some cases when the return value of the ${HOSTNAME) variable
    is different from {{ ansible_hostname }}, especially when tls-everywhere
    framework is involved.
    This purpose of this review is to fix the FFU section that needs to know
    the hostname to find the running container on the delegated host.

    Change-Id: Ib269a25fcf0db66ad3144c3e20031a4da6e2e2fb
    Closes-Bug: #1903537
    (cherry picked from commit 4e0f61d9e2affd107b0ac9a195e23f2a71083383)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762285
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5a0dbe2406a76710cc9844fdfb8188e6d2338607
Submitter: Zuul
Branch: stable/train

commit 5a0dbe2406a76710cc9844fdfb8188e6d2338607
Author: Francesco Pantano <email address hidden>
Date: Mon Nov 9 14:39:40 2020 +0100

    Properly compute hostname when looking for the ceph-mon container

    There are some cases when the return value of the ${HOSTNAME) variable
    is different from {{ ansible_hostname }}, especially when tls-everywhere
    framework is involved.
    This purpose of this review is to fix the FFU section that needs to know
    the hostname to find the running container on the delegated host.

    Change-Id: Ib269a25fcf0db66ad3144c3e20031a4da6e2e2fb
    Closes-Bug: #1903537
    (cherry picked from commit 4e0f61d9e2affd107b0ac9a195e23f2a71083383)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 13.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 14.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.4.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.