Align/fix tht containerized services' mismatching bind mounts for config data vs puppet-generated one

[Bogdan Dobrelya]

Looking into the t-h-t services definitions, there is a mismatch for
some patterns for when we use /var/lib/config-data/ vs
/var/lib/config-data/puppet-generate in volumes, see [0] for the full diff. Each of the listed cases needs either comments (describing why we are deviating from the common approach) or a fix

Example of a mismatch:

1) /var/lib/config-data/puppet-generated is not mounted as /var/lib/kolla/config_files/src:

docker/services/octavia-housekeeping.yaml:
  - /var/lib/config-data/puppet-generated/octavia/etc/octavia:/etc/octavia/

2) /var/lib/config-data is mounted to /etc/* omitting the */puppet-generated path:

docker/services/cinder-api.yaml:
  - /var/lib/config-data/cinder/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro

I assume /var/lib/config-data/puppet-generate should only be
bind-mounted as /var/lib/kolla/config_files/src and never as /etc/? While
/var/lib/config-data should only be bind mounted to /etc and not
/var/lib/kolla/config_files/src ? I may be wrong, but the code needs
comments IMO, and maybe an additional yaml validation for that?

In case the mismatch is real, the idempotency issues may arise, which also impacts minor updates.

[0] https://pastebin.com/6c9eTzmK

[Bogdan Dobrelya (bogdando)]

The solution is for each of the listed cases (there is a 4 possible ways to bind-mount things):

- /var/lib/config-data/puppet-generated/foo/:/var/lib/kolla/config_files/src:ro (145 matches in tht, that is a known to be valid pattern by design, see https://hardysteven.blogspot.com/2018/06/tripleo-containerized-deployments.html )

- /var/lib/config-data/puppet-generated/foo/:/etc/foo (11 matches in tht, is it a (not) valid pattern?)

- /var/lib/config-data/foo/etc/foo:/var/lib/kolla/config_files/src:ro (0 mathes in tht, so considering it a not valid pattern)

- /var/lib/config-data/puppet-generated/foo/:/etc/foo: (43 matches in tht, is it a (not) valid pattern?)

either fix the mismatches ( 43 + 11 cases ) or add comments describing why it has to be so different for each of the case.

[Steve Baker (steve-stevebaker)]

Its not quite as simple as doing a tht grep for /var/lib/config-data. Here are some things to consider:

1) anything with a custom command can't use kolla_start.sh or KOLLA_CONFIG_STRATEGY=COPY_ALWAYS, so the only option is to directly mount whatever /var/lib/config-data/<service>/... files are needed directly. In many cases, puppet-generated can't be used because its sparse and only contains files modified by puppet.
   a) Most cases of 1) are db_sync containers, and I don't think this pattern is a problem
   b) Some services like memcached need a custom command, for specific reasons, so the fix is more involved than just a consistent bind-mount policy

2) For services using kolla_start.sh KOLLA_CONFIG_STRATEGY=COPY_ALWAYS there may be the odd file *also* directly bind mounted in because they need to exist during container start (I can't find any right now but I remember seeing them).

Maybe we should limit this bug to identifying 2) and making sure they're consistent about directly mounting /var/lib/config-data/<service>/... vs /var/lib/config-data/puppet-generated/<service>/

For the record, I don't see any idempotency issues with our current approach.

[Bogdan Dobrelya (bogdando)]

I'm fine with limiting this bug to identifying 2) and making sure they're consistent about directly mounting /var/lib/config-data/<service>/... vs /var/lib/config-data/puppet-generated/<service>/

Thanks for inputs, Steve! We might want to update docs / blogs (https://hardysteven.blogspot.com/2018/06/tripleo-containerized-deployments.html) as well

[Bogdan Dobrelya (bogdando)]

Related partial fix https://review.openstack.org/#/c/620868

[Bogdan Dobrelya (bogdando)]

It seems we have /etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors defined globally and failing to catch up puppet config changes for Keystone IdM from the /var/lib/config-data/puppet-generated/keystone/etc/pki/ca-trust/source/anchors/ directory.

And this also falls into the bug subject area, i.e. how to bind mount /etc...: vs /var/lib/config-data/(puppet-generated?)/etc...: things.

[Bogdan Dobrelya (bogdando)]

Related bug 1766178

[Bogdan Dobrelya (bogdando)]

But that's not sure, cuz that all used to be ran on the host, not in a config containers, and there is *nothing* to pick from /var/lib/...

[Bogdan Dobrelya (bogdando)]

thinking more of the fixes like in https://review.openstack.org/#/c/634649/2..3/docker/services/database/mysql.yaml@224 in the context of this issue, I came to a filter http://paste.openstack.org/show/eFwIo5kcp40DUiYYSA9Y/ that shows possible mismatches (look into "config-data" of the bind-mount paths used for docker_puppet_tasks vs docker_config's step_X sections). That is something we might want to align and probably could as well automate for the YAML checker.

[Bogdan Dobrelya (bogdando)]

So looking into that http://paste.openstack.org/show/eFwIo5kcp40DUiYYSA9Y/, yet another example is lines #203 vs #215 and #228. My gut feeling is that the latter must be with puppet-generated in it as well.

[Bogdan Dobrelya (bogdando)]

ok, so while looking into that list, we should keep in mind the "valid cases" as it's described in https://bugs.launchpad.net/tripleo/+bug/1788138/comments/2

[Bogdan Dobrelya (bogdando)]

I think one thing can simplify templates a lot a merging all configs under var/lib/config-data/foo and removing *any* use of /var/lib/config-data/puppet-generated/foo, but for debug purposes only (there is a comment that explains that a copy of files modified during puppet run is useful for debugging). I'm not sure when we started referencing those debugging purposes only files residing in the puppet-generated paths, but that had been apparently a wrong move, that creates this subject issue in the first place.

[Marios Andreou (marios-b)]

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

[Launchpad Janitor (janitor)]

[Expired for tripleo because there has been no activity for 60 days.]

[Bogdan Dobrelya (bogdando)]

Addressed by https://review.opendev.org/c/openstack/tripleo-heat-templates/+/791033