[RFE] support automatic t-h-t decoupling and versioning for mixed UC/OC cases (git integration for deployment plans and tht)

[Bogdan Dobrelya]

=============
Problem scope
=============

When testing (or doing in production perhaps) mixed versions deployments of UC/OC and mixed upgrades, we/operators have to decouple the --templates paths, historically in quickstart and other places, like custom wrappers, but not in the product (tripleoclient et al). That is needed to not break the default RPM directory for the installed tripleo heat templates package, which is a subject of serial yum operations performed for the mixed deployment/upgrade cases.

Custom scripts/CI/QA tools (quickstart/infrared) decoupling implementations does not scale and has to be maintained separately from tripleo, nor does this provide a solution for operators having these mixed use cases in production.

The --templates paths decoupling logic must be productized, therefore. Especially now that we also have underclouds installed from heat tenmplates, like overclouds do. The current model for --templates is only:

1. (no --templates args, uses default /usr/share location)
2. --templates foo (foo must exist, so we use it)

More background/facts on the RFE request (think of it as alternatives?):

 * there was a *-compat RPM for exactly this purpose (diverging the mixed tht paths) a while back. If we do not really want automate this in client, let's document the mixed upgrades scenario in the advanced installation guide then, including anything needed to be done to the *-compat things.
 * in CI previously we unpacked the RPM to a different path to enable that behavior.
 * ops are used to build custom wrappers around 'openstack overcloud deploy' that rsync-ed the content of the t-h-t located in /usr/share, and commited into a local git to run deployments based on a copy of the t-h-t and allowing it to diverge at will. Fact is: this is a strong advice @Tengu got from the Red Hat fast formation about Director, when he was in Barcelona for the Summit
 * upgrades team maintains a lot of custom overrides and scripts for quickstart to support mixed upgrades and there are issues arise like https://bugs.launchpad.net/tripleo/+bug/1781227, which really should better off fixed in the client, than quickstart/infrared/you-name-it.

=================
Proposed solution
=================

The t-h-t decoupling RFE expects additional case 3 supported in the tripleoclient --templates logic for the existing standalone/overcloud/undercloud deployment/upgrade commands:

3. -- templates foo (if foo does not exist, create and populate it from the default RPM directory effectively decoupling it for future use).

An initial attempt had been proposed in [0] (reverted), [1]. These correspond to partial implementing the step #3 in the example. In the end, there should be git (optionally, with gerrit) integration instead for the steps 3, 6, 7.c, 9, 10.c, 12.

There is implementation alternatives, like pushing patches instead of committing changes in the example steps below, each time the initial templates population (steps 3, 6) and rsync updates happening in the client automation.

The trigger for the git integration should be also another
CLI option perhaps, instead of overloading --templates missing logic.

[0] https://review.openstack.org/#/c/582529/
[1] https://review.openstack.org/#/c/582916/

=======
Example
=======

Assumes any customizations to the heat templates should be done by operators manually, before/after the committing steps.

(mixed deploy UC X, OC Y, where X < Y, or X > Y)
1. install openstack-tripleo-heat-templates RPM version X (it is in /usr/share/openstack-tripleo-heat-templates on undercloud)
2. depoy undercloud version X w/o custom --templates args
 #? cannot use default templates path (/usr/share rpm DIR), so tht gets populated into an *empty* (what else it can be?) --templates /tmp/notexisted from RPM dir
3. copy t-h-t used for the step 2 to some uc-tht path, and commit as a git repo uc-tht
4. install t-h-t RPM Y for overcloud of version Y (not exactly yum update'd, but may be another release repo), caveats:
 a. to be safe on installing other version, purge the /usr/share/openstack-tripleo-heat-templates, assuming we cannot fully trust the packaging post- actions?
5. deploy overcloud version Y w/o custom --templates args
6. copy t-h-t used for the step 6 to some oc-tht path, and commit as a git repo oc-tht
 #?, w/o messing into UC tht X (see the step 2 notes) , using whatever --templates dir it wants (quickstart still uses /usr/share... default, this cannot be changed yet)

(mixed upgrade)
7. update undercloud tht RPM to X', caveats:
 a. this overrides the /usr/share contents used to deploy overcloud Y before
 b. to be safe on upgrading, purge the /usr/share/openstack-tripleo-heat-templates, assuming we cannot fully trust the packaging post- actions?
 c. copy installed t-h-t to the uc-tht path,
8. upgrade undercloud to X', using --templates uc-tht
9. commit changes from the step 7.c in the repo uc-tht
10. update overcloud tht RPM to Y', caveats:
 a. this overrides the /usr/share contents used to deploy overcloud Y before
 b. to be safe on upgrading, purge the /usr/share/openstack-tripleo-heat-templates, assuming we cannot fully trust the packaging post- actions?
 c. copy installed t-h-t to the oc-tht path
11. upgrade overcloud to Y', using --templates oc-tht
12. commit changes from the step 10.c in the repo oc-tht
13. from now on, any future uc/oc maintanance command should keep using the diverged --templates paths
 a. caveats: all those purge/copy/commit dances are the must, each time, and become highly error-prone now that we have also undercloud heat installer instead of instack.
 b. the steps 3, 6, 7.c, 9, 10.c, 12 should be took off its high error-proneness by the client automation and built in git integration

[Steven Hardy (shardy)]

> "The --templates paths decoupling logic must be productized"

I think we need more details on this, e.g what does it mean for an operator?

Step by step examples of the CLI flow would probably help clarify this I think.

[Steven Hardy (shardy)]

https://bugs.launchpad.net/tripleo/+bug/1781227 appears to be either a packaging or CI issue, I'm still not clear how the tripleoclient patch helps, except that we can "trick" the client into using a single --templates command to switch between actually two different paths - this is really opaque to the operator and inconsistent with how --templates has worked for many releases.

[Steven Hardy (shardy)]

I think the detail added shows that primarily we need a way to co-install multiple versions of the templates (and probably images), there is the openstack-tripleo-heat-templates-compat RPM which was supposed to help with that, but I don't think it's well tested at this point, perhaps we can address that by starting to use it for upgrade CI?

[Steven Hardy (shardy)]

Separate from the mixed version requirement, there's been discussion around operator workflow that copies the packaged t-h-t to a git controlled directory, and whether we could automate some of that workflow (which doesn't directly solve the co-install problems but could be used as a way to jump between versions of RPM if we tagged e.g each version as a different release or branch)

I think we need more discussion on the two use-cases here, but regardless I'm not in favor of accepting a non-existent --templates directory to trigger any copy/fallback action, because it's confusing and inconsistent with the Ux which has existed for many releases.

[Thomas Herve (therve)]

Not much to add to the discussion here, just wondering about the git deployment details. It may be interesting to have a look at dulwich with the swift backend: https://github.com/dulwich/dulwich/blob/master/README.swift.md

[Bogdan Dobrelya (bogdando)]

Another case for t-h-t versioning is Edge clouds, when single undercloud should install and Day-2 manage multiple overclouds from different tht paths

[Marios Andreou (marios-b)]

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

[Launchpad Janitor (janitor)]

[Expired for tripleo because there has been no activity for 60 days.]