Overcloud deployment/update fails due to Read-only file system error in keystone containers when tls_cacertdir is specified.

[Yamato Tanaka]

Description
===========
When tls_cacertdir is specified in a template file, overcloud deployment/update fails at "Wait for containers to start for step 3 using paunch" task as below
~~~
(undercloud) [stack@undercloud ~]$ cat /home/stack/templates/ldap.yaml
parameter_defaults:
  KeystoneLDAPDomainEnable: true
  KeystoneLDAPBackendConfigs:
    testdomain:
        :
      tls_cacertdir: /etc/openldap/cacerts <====(*)
        :

(undercloud) [stack@undercloud ~]$ openstack overcloud deploy --templates \
> -e /home/stack/templates/ldap.yaml \
> ...
> ...

   :

2022-01-26 10:12:27,452 p=23585 u=mistral n=ansible | TASK [Wait for containers to start for step 3 using paunch] ********************

   :

2022-01-26 10:13:26,691 p=23585 u=mistral n=ansible | fatal: [overcloud-controller-2]: FAILED! => {"ansible_job_id": "386314127972.254349", "attempts": 19, "changed": false, "finished": 1, "msg": "Paunch failed with config_id tripleo_step3", "rc": 126, "stderr": "Error executing ['podman', 'container', 'exists', 'aodh_db_sync']: returned 1\nDid not find container with \"['podm ...

   :

PLAY RECAP *********************************************************************
overcloud-controller-0 : ok=278 changed=125 unreachable=0 failed=1 skipped=157 rescued=0 ignored=0
overcloud-controller-1 : ok=267 changed=125 unreachable=0 failed=1 skipped=153 rescued=0 ignored=0
overcloud-controller-2 : ok=267 changed=125 unreachable=0 failed=1 skipped=153 rescued=0 ignored=0
overcloud-novacompute-0 : ok=224 changed=87 unreachable=0 failed=0 skipped=143 rescued=0 ignored=0
undercloud : ok=26 changed=10 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0
~~~

During the deployment/update, "/var/lib/config-data/puppet-generated/keystone/etc/openldap/cacerts" is create automatically because of "tls_cacertdir" setting.
~~~
[root@overcloud-controller-0 ~]# find /var/lib/config-data/puppet-generated/keystone/etc/openldap/
/var/lib/config-data/puppet-generated/keystone/etc/openldap/
/var/lib/config-data/puppet-generated/keystone/etc/openldap/cacerts
~~~

Copying "/var/lib/config-data/puppet-generated/keystone/etc/openldap" to "/etc/openldap" causes "Read-only file system" error in keystone containers on controller nodes.
~~~
[root@overcloud-controller-0 ~]# tail -n 21 /var/log/containers/stdouts/keystone*
==> /var/log/containers/stdouts/keystone_db_sync.log <==
2022-02-10T01:45:54.579882868+00:00 stderr F ERROR:__main__:Unexpected error:
2022-02-10T01:45:54.579882868+00:00 stderr F Traceback (most recent call last):
2022-02-10T01:45:54.579882868+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 417, in main
2022-02-10T01:45:54.579882868+00:00 stderr F execute_config_strategy(config)
2022-02-10T01:45:54.579882868+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 383, in execute_config_strategy
2022-02-10T01:45:54.579882868+00:00 stderr F copy_config(config)
2022-02-10T01:45:54.579882868+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 306, in copy_config
2022-02-10T01:45:54.579882868+00:00 stderr F config_file.copy()
2022-02-10T01:45:54.579882868+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 150, in copy
2022-02-10T01:45:54.579882868+00:00 stderr F self._merge_directories(source, dest)
2022-02-10T01:45:54.579882868+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
2022-02-10T01:45:54.579882868+00:00 stderr F os.path.join(dest, to_copy))
2022-02-10T01:45:54.579882868+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 92, in _merge_directories
2022-02-10T01:45:54.579882868+00:00 stderr F self._set_properties(source, dest)
2022-02-10T01:45:54.579882868+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 117, in _set_properties
2022-02-10T01:45:54.579882868+00:00 stderr F self._set_properties_from_file(source, dest)
2022-02-10T01:45:54.579882868+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 122, in _set_properties_from_file
2022-02-10T01:45:54.579882868+00:00 stderr F shutil.copystat(source, dest)
2022-02-10T01:45:54.579882868+00:00 stderr F File "/usr/lib64/python3.6/shutil.py", line 205, in copystat
2022-02-10T01:45:54.579882868+00:00 stderr F follow_symlinks=follow)
2022-02-10T01:45:54.579882868+00:00 stderr F OSError: [Errno 30] Read-only file system

==> /var/log/containers/stdouts/keystone_init_log.log <==

==> /var/log/containers/stdouts/keystone.log <==
2022-02-10T01:46:27.038039580+00:00 stderr F ERROR:__main__:Unexpected error:
2022-02-10T01:46:27.038039580+00:00 stderr F Traceback (most recent call last):
2022-02-10T01:46:27.038039580+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 417, in main
2022-02-10T01:46:27.038039580+00:00 stderr F execute_config_strategy(config)
2022-02-10T01:46:27.038039580+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 383, in execute_config_strategy
2022-02-10T01:46:27.038039580+00:00 stderr F copy_config(config)
2022-02-10T01:46:27.038039580+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 306, in copy_config
2022-02-10T01:46:27.038039580+00:00 stderr F config_file.copy()
2022-02-10T01:46:27.038039580+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 150, in copy
2022-02-10T01:46:27.038039580+00:00 stderr F self._merge_directories(source, dest)
2022-02-10T01:46:27.038039580+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
2022-02-10T01:46:27.038039580+00:00 stderr F os.path.join(dest, to_copy))
2022-02-10T01:46:27.038039580+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 92, in _merge_directories
2022-02-10T01:46:27.038039580+00:00 stderr F self._set_properties(source, dest)
2022-02-10T01:46:27.038039580+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 117, in _set_properties
2022-02-10T01:46:27.038039580+00:00 stderr F self._set_properties_from_file(source, dest)
2022-02-10T01:46:27.038039580+00:00 stderr F File "/usr/local/bin/kolla_set_configs", line 122, in _set_properties_from_file
2022-02-10T01:46:27.038039580+00:00 stderr F shutil.copystat(source, dest)
2022-02-10T01:46:27.038039580+00:00 stderr F File "/usr/lib64/python3.6/shutil.py", line 205, in copystat
2022-02-10T01:46:27.038039580+00:00 stderr F follow_symlinks=follow)
2022-02-10T01:46:27.038039580+00:00 stderr F OSError: [Errno 30] Read-only file system
~~~

This issue doesn't occur in an older release because this issue was introduces by the following change.
  - Gerrit :https://review.opendev.org/c/openstack/tripleo-heat-templates/+/785404/

After the above change was merged, "/etc/openldap" is bind mounted with "ro" option in keystone containers
~~~
[root@overcloud-controller-0 config-data]# podman inspect keystone |grep /etc/openldap|tail -n1
                "/etc/openldap:/etc/openldap:ro,rprivate,rbind",
[root@overcloud-controller-0 config-data]# podman inspect keystone_db_sync |grep /etc/openldap|tail -n1
                "/etc/openldap:/etc/openldap:ro,rprivate,rbind",
~~~

Steps to reproduce
==================

I reproduced this issue by the following step.

1. deploy overcloud normally
2. create a new template file which contains "tls_cacertdir" line.
~~~
(undercloud) [stack@undercloud ~]$ cat /home/stack/templates/ldap.yaml
parameter_defaults:
  KeystoneLDAPDomainEnable: true
  KeystoneLDAPBackendConfigs:
    testdomain:
        :
      tls_cacertdir: /etc/openldap/cacerts <====(*)
        :
~~~
3. redeploy overcloud by running "openstack overcloud deploy" command which contains the above template file.

Expected result
===============
Deployment succeeds.

Actual result
=============
Deployment fails.

Environment
===========
this issue was found in a stable-train-based deployment (RHOSP16.1)

Logs & Configs
==============
this issue was initially reported in the following downstream bug : https://bugzilla.redhat.com/show_bug.cgi?id=2052800

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/829001

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/829001
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/daa48254ebbc1b107ec2b382355209c256dbea48
Submitter: "Zuul (22348)"
Branch: master

commit daa48254ebbc1b107ec2b382355209c256dbea48
Author: Yamato Tanaka <email address hidden>
Date: Mon Feb 14 17:02:36 2022 +0900

    Exclude /etc/openldap to avoid overriding ro file

    Currently /etc/openldap is bind-mounted by keystone containers
    with ro flag so it should be excluded from config files generaed
    by puppet.

    Closes-Bug: #1960781
    Change-Id: I8027b829e6b5aebfcd36f5025bb9c848cc96ce3f

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/830574

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/830575

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/830576

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/830577

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/830574
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/92d84e573fbd84cb397ab785bfe21531b5fb8511
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 92d84e573fbd84cb397ab785bfe21531b5fb8511
Author: Yamato Tanaka <email address hidden>
Date: Mon Feb 14 17:02:36 2022 +0900

    Exclude /etc/openldap to avoid overriding ro file

    Currently /etc/openldap is bind-mounted by keystone containers
    with ro flag so it should be excluded from config files generaed
    by puppet.

    Closes-Bug: #1960781
    Change-Id: I8027b829e6b5aebfcd36f5025bb9c848cc96ce3f
    (cherry picked from commit daa48254ebbc1b107ec2b382355209c256dbea48)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/830575
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/4dadf3ded45958eb42d4b9f5a49f0f595dc31dd2
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 4dadf3ded45958eb42d4b9f5a49f0f595dc31dd2
Author: Yamato Tanaka <email address hidden>
Date: Mon Feb 14 17:02:36 2022 +0900

    Exclude /etc/openldap to avoid overriding ro file

    Currently /etc/openldap is bind-mounted by keystone containers
    with ro flag so it should be excluded from config files generaed
    by puppet.

    Closes-Bug: #1960781
    Change-Id: I8027b829e6b5aebfcd36f5025bb9c848cc96ce3f
    (cherry picked from commit daa48254ebbc1b107ec2b382355209c256dbea48)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Marios Andreou <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/830576
Reason: abandoning per http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028026.html - so we can move EOL https://review.opendev.org/c/openstack/releases/+/834049

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 16.0.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/830577
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/754a9a7a44a5d6bc1d4c40aaf479a6937fc11e15
Submitter: "Zuul (22348)"
Branch: stable/train

commit 754a9a7a44a5d6bc1d4c40aaf479a6937fc11e15
Author: Yamato Tanaka <email address hidden>
Date: Mon Feb 14 17:02:36 2022 +0900

    Exclude /etc/openldap to avoid overriding ro file

    Currently /etc/openldap is bind-mounted by keystone containers
    with ro flag so it should be excluded from config files generaed
    by puppet.

    Closes-Bug: #1960781
    Change-Id: I8027b829e6b5aebfcd36f5025bb9c848cc96ce3f
    (cherry picked from commit daa48254ebbc1b107ec2b382355209c256dbea48)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 13.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates train-eol release.