Server side environment merging requires client-side workarounds

[Steven Hardy]

I've been trying to adopt the server-side environment merging we added to heat for TripleO, where we make very heavy use of multiple environment_files.

There are a few usability/interface problems which have so far stalled progress in adopting these heat features (in particular we really want to use the merge strategies for parameter_defaults, which requires moving to server-side environment merging of a list of environment_files.

1. You still need to resolve the merged environment locally, because you can't get all the nested get_file: foo.sh, type: foo.yaml etc references without running the process_multiple_environments_and_files function:

https://github.com/openstack/python-heatclient/blob/master/heatclient/common/template_utils.py#L206

This works OK using the mutable env_list_tracker parameter if all your files are stored locally (or, if all of them are stored in a swift container), but if you upload an incomplete set of templates/files to a swift container, it's really inconvenient to discover which files e.g referenced via get_file are missing.

I think the best solution to this is to add a missing_files_list_tracker or similar, where we can allow get_file/type etc lookups to fail but just add them to a list, then we can upload the missing files to swift, re-run the resolution of the environment, and when missing_files_list_tracker comes back empty, we know the swift container has all files referenced by the environment files and templates.

2. No way to "preview" server side merging

Even after we fix the above interface to make handling templates in swift easier, there's still a local resolution step required, which is not certain to yield the exact same results as we can't resolve the merge_strategies things (e.g if you merged a parameter containing strings with local paths, which are then passed into a ResourceChain instead of resource_registry aliases, the local environment resolution will be wrong I think.

I think we need to either enhance the preview/validate APIs to allow returning the merged environment (and list of files/missing files), or add some new API that's capable of just taking a list of environment files and doing *all* of the things we're doing locally atm.

(An important part of this for the TripleO use-case will be accepting a swift container instead of a request with all the files)

3. No way to list environment files from a running stack

We've got openstack stack environment show, which allows seeing the merged environment, but there's no way to see the individual environment_files that were merged to create it, which means you must keep track of them locally. It would be good to add an API that allows you to easily retrieve the list of files (and optionally their contents).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/404534
Committed: https://git.openstack.org/cgit/openstack/python-heatclient/commit/?id=280b1edeeab3a9746ec6674d23fb7ea95b4b46fc
Submitter: Jenkins
Branch: master

commit 280b1edeeab3a9746ec6674d23fb7ea95b4b46fc
Author: rabi <email address hidden>
Date: Wed Nov 30 09:01:34 2016 +0530

    User server side env merging with osc plugin

    With osc client plugin, we don't seem to use server side
    env merging for stack create and update. However, it's
    there for template validation.

    This patch brings it in sync with heat cli.

    Change-Id: I8973fa499c51ca5f789efa6fb5c07bb6302324d7
    Related-Bug: #1645686