Error at task "generate ceph-ansible group vars mgrs" when deploy ceph with ceph-ansible

[Juan Badia Payno]

Deploy ceph with ceph-ansible fails at the task "generate ceph-ansible group vars mgrs" with the error "Destination directory /home/stack/overcloud-deploy/overcloud/config-download/overcloud/ceph-ansible/group_vars does not exist"

#######
Reproduce the error:
 - Deployed the environment with tripleo-quickstart
   - Modified the config/nodes/3ctlr_2comp_3ceph.yml file:
      Memory: undercloud + controller: 16384, compute + ceph: 8192
      CPU: undercloud 8, controller: 4, default: 2
   - Deploying using the container register quay.io
   On the undercloud:
    - Install ceph-ansible
    - deploy the overcloud ceph-ansible/ceph-ansible.yaml

#### * Deploy the overcloud!
## ::
openstack overcloud deploy --stack overcloud --override-ansible-cfg /home/stack/custom_ansible.cfg \
    --templates /usr/share/openstack-tripleo-heat-templates \
    --libvirt-type qemu --timeout 90 -e /home/stack/cloud-names.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml -e /home/stack/overcloud-baremetal-deployed.yaml --deployed-server -e /home/stack/containers-prepare-parameter.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /home/stack/network-environment.yaml -e /home/stack/overcloud_storage_params.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml -e /home/stack/enable-tls.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml -e /home/stack/inject-trust-anchor.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml --validation-warnings-fatal -e /home/stack/overcloud-topology-config.yaml -e /home/stack/overcloud-selinux-config.yaml -e /usr/share/openstack-tripleo-heat-templates/ci/environments/ovb-ha.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml --disable-validations \
    "$@" && status_code=0 || status_code=$?

###### ansible.log
2021-05-25 16:43:03,759 p=852799 u=stack n=ansible | PLAY [External deployment step 1] **********************************************
2021-05-25 16:43:03,772 p=852799 u=stack n=ansible | 2021-05-25 16:43:03.771556 | 00abc5a4-4b92-39d8-01c4-0000000000ae | TASK | External deployment step 1
2021-05-25 16:43:03,792 p=852799 u=stack n=ansible | 2021-05-25 16:43:03.791719 | 00abc5a4-4b92-39d8-01c4-0000000000ae | OK | External deployment step 1 | undercloud | result={
    "changed": false,
    "msg": "Use --start-at-task 'External deployment step 1' to resume from this task"
}
2021-05-25 16:43:03,926 p=852799 u=stack n=ansible | 2021-05-25 16:43:03.925354 | c72f754e-b14a-4c48-b08d-283e25dd6222 | INCLUDED | /home/stack/overcloud-deploy/overcloud/config-download/overcloud/external_deploy_steps_tasks_step1.yaml | undercloud
2021-05-25 16:43:03,937 p=852799 u=stack n=ansible | 2021-05-25 16:43:03.937236 | 00abc5a4-4b92-39d8-01c4-000000005d09 | TASK | set ceph-ansible group vars mgrs
2021-05-25 16:43:03,969 p=852799 u=stack n=ansible | 2021-05-25 16:43:03.968794 | 00abc5a4-4b92-39d8-01c4-000000005d09 | OK | set ceph-ansible group vars mgrs | undercloud
2021-05-25 16:43:03,976 p=852799 u=stack n=ansible | 2021-05-25 16:43:03.975790 | 00abc5a4-4b92-39d8-01c4-000000005d0a | TASK | generate ceph-ansible group vars mgrs
2021-05-25 16:43:04,449 p=852799 u=stack n=ansible | 2021-05-25 16:43:04.448416 | 00abc5a4-4b92-39d8-01c4-000000005d0a | FATAL | generate ceph-ansible group vars mgrs | undercloud | error={"changed": false, "checksum": "b10166b36c277d505708ceaa7971a894b2da8662", "msg": "Destination directory /home/stack/overcloud-deploy/overcloud/config-download/overcloud/ceph-ansible/group_vars does not exist"}
2021-05-25 16:43:04,457 p=852799 u=stack n=ansible | PLAY RECAP *********************************************************************
2021-05-25 16:43:04,457 p=852799 u=stack n=ansible | localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
2021-05-25 16:43:04,458 p=852799 u=stack n=ansible | overcloud-cephstorage-0 : ok=142 changed=62 unreachable=0 failed=0 skipped=66 rescued=0 ignored=0
2021-05-25 16:43:04,458 p=852799 u=stack n=ansible | overcloud-cephstorage-1 : ok=136 changed=62 unreachable=0 failed=0 skipped=63 rescued=0 ignored=0
2021-05-25 16:43:04,459 p=852799 u=stack n=ansible | overcloud-cephstorage-2 : ok=136 changed=62 unreachable=0 failed=0 skipped=63 rescued=0 ignored=0
2021-05-25 16:43:04,459 p=852799 u=stack n=ansible | overcloud-controller-0 : ok=194 changed=99 unreachable=0 failed=0 skipped=84 rescued=0 ignored=0
2021-05-25 16:43:04,460 p=852799 u=stack n=ansible | overcloud-controller-1 : ok=189 changed=99 unreachable=0 failed=0 skipped=84 rescued=0 ignored=0
2021-05-25 16:43:04,460 p=852799 u=stack n=ansible | overcloud-controller-2 : ok=189 changed=99 unreachable=0 failed=0 skipped=84 rescued=0 ignored=0
2021-05-25 16:43:04,461 p=852799 u=stack n=ansible | overcloud-novacompute-0 : ok=169 changed=88 unreachable=0 failed=0 skipped=77 rescued=0 ignored=0
2021-05-25 16:43:04,461 p=852799 u=stack n=ansible | overcloud-novacompute-1 : ok=169 changed=88 unreachable=0 failed=0 skipped=77 rescued=0 ignored=0
2021-05-25 16:43:04,462 p=852799 u=stack n=ansible | undercloud : ok=20 changed=8 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
2021-05-25 16:43:04,467 p=852799 u=stack n=ansible | 2021-05-25 16:43:04.467518 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

######
the ceph-ansible directory does not exist on the config-download/overcloud directory
(undercloud) [stack@undercloud overcloud]$ pwd
/home/stack/config-download/overcloud
(undercloud) [stack@undercloud overcloud]$ ls -lhrt | grep -i ceph
drwx------. 2 stack stack 4.0K May 25 16:38 CephStorage
(undercloud) [stack@undercloud overcloud]$ cd /home/stack/overcloud-deploy/overcloud/config-download/overcloud
(undercloud) [stack@undercloud overcloud]$ ls -lhrt | grep -i ceph
drwx------. 2 stack stack 4.0K May 25 16:38 CephStorage

#####
(undercloud) [stack@undercloud ~]$ cat /etc/yum.repos.d/quickstart-centos-ceph-pacific.repo
[quickstart-centos-ceph-pacific]
name=quickstart-centos-ceph-pacific
baseurl=http://mirror.centos.org/centos/8/storage/x86_64/ceph-pacific/
gpgcheck=0
enabled=1

#####

[Francesco Pantano (fmount)]

This bug sounds like pretty similar to [1], which is supposed to be fixed by [2].
If you inspect at [3], during step1 variables are dumped before the directory is created, and this is the root cause of the issue here.

[1] https://bugs.launchpad.net/tripleo/+bug/1925373
[2] https://review.opendev.org/c/openstack/heat/+/787662
[3] http://ix.io/3nTW

[Francesco Pantano (fmount)]

The example [3] of comment#1 shows how [4] is not reflected in
"external_deploy_steps_tasks_step1.yaml", putting the mgr block [5] before [6]

[4] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-mgr.yaml#L119

[5] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-mgr.yaml#L126-L140

[6] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-base.yaml#L635

[Rabi Mishra (rabi)]

heat-engine container does not have fix for https://bugs.launchpad.net/tripleo/+bug/1925373

[Rabi Mishra (rabi)]

It's pretty old.

[stack@undercloud ~]$ sudo podman images | grep heat-engin
192.168.24.1:8787/tripleomaster/openstack-heat-engine current-tripleo bc30957138db 2 months ago 997 MB

[Juan Badia Payno (jbadiapa)]

I think this error is no longer valid.
The issue was as Rabi commented that the containers on quay were quite old.
The quay containers are refreshing more often, so I'm going to wait a few week and close it if there is no rejections.