'overcloud export ceph' fails ceph_client.yml not found

[John Fulton]

openstack overcloud export ceph --stackopenstack overcloud export ceph --stack central --output-file central_ceph_external.yaml
/usr/lib/python3.9/site-packages/ansible/_vendor/__init__.py:42: UserWarning: One or more Python packages bundled by this ansible-core distribution were already loaded (pyparsing). This may result in undefined behavior.
  warnings.warn('One or more Python packages bundled by this ansible-core distribution were already '
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 34, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_export_ceph.py", line 105, in take_action
    cephs.append(export.export_ceph(stack,
  File "/usr/lib/python3.9/site-packages/tripleoclient/export.py", line 215, in export_ceph
    with open(file, 'r') as ff:
FileNotFoundError: [Errno 2] No such file or directory: '/home/stack/overcloud-deploy/central/config-download/central/cephadm/ceph_client.yml'
[Errno 2] No such file or directory: '/home/stack/overcloud-deploy/central/config-download/central/cephadm/ceph_client.yml'

[John Fulton (jfulton-org)]

Analysis of system where bug was encountered:

No such file or directory:
  '/home/stack/overcloud-deploy/central/config-download/central/cephadm/ceph_client.yml

Instead we have this:

  /home/stack/ceph_client.yaml

Let's think about how TripleO controls where this file is:

What should happen if I just read the code:

A. THT sets this to [1]

  "{{ playbook_dir }}/cephadm/ceph_client.yml"

B. tripleo-ansible saves [2] this to a variable which defaults it to [3] [4]:

  tripleo_ceph_client_vars: "/home/stack/ceph_client.yaml"

C. tripleo-pythonclient looks for it in [5]:

  ceph_client = "cephadm/ceph_client.yml"
  file = os.path.join(config_download_dir, stack, ceph_client)

A and C are not inconsistent because:

  playbook_dir == (config_download_dir + stack)

B is not consistent w/ A+C, so I should update that if nothing else. Was that the root cause?

Let's look at if THT had that set in the deployment:

  config-download/central/cephadm/cephadm-extra-vars-heat.yml:tripleo_ceph_client_vars: /home/stack/ceph_client.yaml

There:

(undercloud) [stack@site-undercloud-0 templates]$ grep CephClientConfigVars *.yaml
overcloud-ceph-deployed.yaml: CephClientConfigVars: /home/stack/ceph_client.yaml
(undercloud) [stack@site-undercloud-0 templates]$

The default from tripleo-ansible was exported and that's why THT didn't override it.

Definitely a bug. Probably needs two patches:

1. tripleo-ansible: new default consistent with what THT would produce
2. python-tripleoclient: also look in /home/stack/ceph_client.yaml related-bug (for current users)

[1] https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/deployment/cephadm/ceph-base.yaml#L335-L337

[2] https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_cephadm/tasks/export.yaml#L86-L92

[3] https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_cephadm/defaults/main.yml#L37

[4] https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ceph_client/defaults/main.yml#L25

[5] https://github.com/openstack/python-tripleoclient/blob/stable/wallaby/tripleoclient/export.py#L214

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/846441

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/846443

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/846441
Committed: https://opendev.org/openstack/python-tripleoclient/commit/fda3233451db678642fd57f920487db7ca63dc30
Submitter: "Zuul (22348)"
Branch: master

commit fda3233451db678642fd57f920487db7ca63dc30
Author: John Fulton <email address hidden>
Date: Fri Jun 17 16:11:59 2022 -0400

    Fall back to alt path for ceph client data during export

    The patch to fix the related bug (not this patch) fixes an
    inconsistency where the ceph client data is somtimes exported
    to the wrong path. If anyone deployed before they got the fix
    to the related bug, then this patch will workaround that
    problem if they export later.

    When 'openstack overcloud export ceph' is run look in the
    correct default location for the ceph client data and if it
    is not found, then fall back to incorrect location.

    Change-Id: I0b8c747481e561645eefea50cb77ca806a7f0de2
    Related-Bug: #1978846

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/846556

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/846556
Committed: https://opendev.org/openstack/python-tripleoclient/commit/7af5a55c819b5509a5dcaff7262538308ecd7c33
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7af5a55c819b5509a5dcaff7262538308ecd7c33
Author: John Fulton <email address hidden>
Date: Fri Jun 17 16:11:59 2022 -0400

    Fall back to alt path for ceph client data during export

    The patch to fix the related bug (not this patch) fixes an
    inconsistency where the ceph client data is somtimes exported
    to the wrong path. If anyone deployed before they got the fix
    to the related bug, then this patch will workaround that
    problem if they export later.

    When 'openstack overcloud export ceph' is run look in the
    correct default location for the ceph client data and if it
    is not found, then fall back to incorrect location.

    Change-Id: I0b8c747481e561645eefea50cb77ca806a7f0de2
    Related-Bug: #1978846
    (cherry picked from commit fda3233451db678642fd57f920487db7ca63dc30)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/846443
Committed: https://opendev.org/openstack/tripleo-ansible/commit/93cc215909db617aece85cb53c0279d04c04c69a
Submitter: "Zuul (22348)"
Branch: master

commit 93cc215909db617aece85cb53c0279d04c04c69a
Author: John Fulton <email address hidden>
Date: Fri Jun 17 16:31:02 2022 -0400

    Set tripleo_ceph_client_vars consistently with THT

    The TripleO Heat Templates parameter CephClientConfigVars
    defaults to "{{ playbook_dir }}/cephadm/ceph_client.yml".
    This parameter correlates with the the Ansible variable
    tripleo_ceph_client_vars whose default is not the same.
    This inconsistency can cause bug 1978846. This patch
    changes tripleo-ansible so that they both have the same
    default.

    Change-Id: I58a8a1d058a76c1733c94c2be96bf59bb553946f
    Closes-Bug: #1978846

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/847998

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/847998
Committed: https://opendev.org/openstack/tripleo-ansible/commit/5e4ab31eabfaca477ffbe9a71abd4c8aaa4624c2
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 5e4ab31eabfaca477ffbe9a71abd4c8aaa4624c2
Author: John Fulton <email address hidden>
Date: Fri Jun 17 16:31:02 2022 -0400

    Set tripleo_ceph_client_vars consistently with THT

    The TripleO Heat Templates parameter CephClientConfigVars
    defaults to "{{ playbook_dir }}/cephadm/ceph_client.yml".
    This parameter correlates with the the Ansible variable
    tripleo_ceph_client_vars whose default is not the same.
    This inconsistency can cause bug 1978846. This patch
    changes tripleo-ansible so that they both have the same
    default.

    Change-Id: I58a8a1d058a76c1733c94c2be96bf59bb553946f
    Closes-Bug: #1978846
    (cherry picked from commit 93cc215909db617aece85cb53c0279d04c04c69a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-ansible 5.0.0 release.