Triggering puppet runs with config_identifier is error prone and complicates templates

[Steve Baker]

As hieradata config happens in more places its going to get harder to propagate config_identifier back to the deployment which triggers puppet.

One possible solution would be to stop tracking this dependency in the templates, and instead enhance 40-hiera-datafiles to indicate that the puppet should be reapplied.

One possible implementation of this could:
- maintain a file of sha256sum /etc/puppet/hieradata/*
- after writing out hieradata, compare previous sha256sum with current
- if the sha256sum has changed, delete any /var/run/heat-config/deployed/<uuid>.json which corresponds to a group:puppet config
- 55-heat-config will run as normal, and the puppet apply will be re-executed

[Steve Baker (steve-stevebaker)]

I'm not going to self-triage this as I'd like some feedback on the approach

[Steven Hardy (shardy)]

So, yeah, as the person responsible for the current interface I agree it's not ideal, and I did consider something similar to what you described when I did it.

> if the sha256sum has changed, delete any /var/run/heat-config/deployed/<uuid>.json which corresponds to a group:puppet config

This is the problem - you have a bunch of deployment resources which are e.g CREATE_COMPLETE, then some hieradata changes, and we have to ensure all the deployments go e.g UPDATE_IN_PROGRESS, or it's impossible to detect any failure.

So, if we go with the checksum approach, we have to take that output and somehow wire it in to all the puppet-applying deployments, otherwise we have no failure path when the puppet re-apply happens.

[Steve Baker (steve-stevebaker)]

On 13/10/15 22:03, Steven Hardy wrote:
> So, yeah, as the person responsible for the current interface I agree
> it's not ideal, and I did consider something similar to what you
> described when I did it.
>
>> if the sha256sum has changed, delete any /var/run/heat-
> config/deployed/<uuid>.json which corresponds to a group:puppet config
>
> This is the problem - you have a bunch of deployment resources which are
> e.g CREATE_COMPLETE, then some hieradata changes, and we have to ensure
> all the deployments go e.g UPDATE_IN_PROGRESS, or it's impossible to
> detect any failure.
>
> So, if we go with the checksum approach, we have to take that output and
> somehow wire it in to all the puppet-applying deployments, otherwise we
> have no failure path when the puppet re-apply happens.
>
hmm, good point. I'll have a think about coupling a deployment with an
orc script that signals when hieradata changes.

[Steven Hardy (shardy)]

I made a start on a hiera heat hook:

https://review.openstack.org/#/c/234821/

WIP atm, but I was thinking we could have that include a checksum of the entire hieradata folder in the deploy_stdout, then have some way to force a Deployment to run every time, then you'd only need to wire in e.g one HieraChecksumDeployment deploy_stdout to catch any changes to the hieradata, which would be less error prone.

Other than a randomly generated input value I'm not quite sure of the cleanest way to force the Deployment to always reapply on update - it seems like a generally useful thing though.

[Steve Baker (steve-stevebaker)]

Yeah, I've been thinking about a force-update for deployment resources too. The only solution currently would be to expect the user to pass in a unique parameter on every run.

This may be inappropriate overloading, but an option which would be backportable would be adding an UPDATE_ALWAYS option to the action property

http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::SoftwareDeployment-prop-actions

and the implementation could do something similar to port REPLACE_ALWAYS

https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/neutron/port.py#L428

[Dan Prince (dan-prince)]

I'm not opposed to passing a unique ConfigIdentifier on each run. We could do this via tripleoclient today (generating a unique config identifier on each run), and perhaps eventually move towards using a Heat RandomString that did the same thing within the tripleo-heat-templates.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/244346

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/244350

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/244346
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a01849952a23a9c6132c782207b7af6c65f5df4d
Submitter: Jenkins
Branch: master

commit a01849952a23a9c6132c782207b7af6c65f5df4d
Author: Dan Prince <email address hidden>
Date: Wed Nov 11 17:46:50 2015 -0500

    Add DeployIdentifier overcloud parameter

    We've heard from end users that it is confusing that puppet
    isn't re-executed on a heat stack-update.

    This patch adds a new DeployIdentifier parameter which
    we can set via client tooling (tripleoclient) to a unique
    value so that on each heat stack-update we always execute
    all of our configuration deployments.

    Change-Id: Ic352ddd30807dc378e5e7b6c396bc53f5d6d5622
    Related-bug: #1505430

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/244350
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=ed945a9080556a20067be2a93831ff1f34b9ae9c
Submitter: Jenkins
Branch: master

commit ed945a9080556a20067be2a93831ff1f34b9ae9c
Author: Dan Prince <email address hidden>
Date: Wed Nov 11 17:52:13 2015 -0500

    Generate a unique DeployIdentifier on updates

    This patch updates 'overcloud deploy' commands so that
    updates (re-running deploy on an existing stack) generate
    a unique DeployIdentifier parameter. By passing an unique
    DeployIdentifier parameter into the overcloud stack we
    force all puppet config tasks (which are idempotent
    and safe to exec multiple times) again.

    We've had several issues with puppet not-rerunning on
    a variety of config related changes and additionally recieved
    user feedback that not re-running all puppet deployments on
    an update is confusing.

    Change-Id: Ia659181872a6537cc169f1d1531ebf5d36160183
    Closes-bug: #1505430
    Depends-On: Ic352ddd30807dc378e5e7b6c396bc53f5d6d5622

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/254815

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/254815
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9e010667c36c84b54d2f06603cff9b6926668854
Submitter: Jenkins
Branch: stable/liberty

commit 9e010667c36c84b54d2f06603cff9b6926668854
Author: Dan Prince <email address hidden>
Date: Wed Nov 11 17:46:50 2015 -0500

    Add DeployIdentifier overcloud parameter

    We've heard from end users that it is confusing that puppet
    isn't re-executed on a heat stack-update.

    This patch adds a new DeployIdentifier parameter which
    we can set via client tooling (tripleoclient) to a unique
    value so that on each heat stack-update we always execute
    all of our configuration deployments.

    Change-Id: Ic352ddd30807dc378e5e7b6c396bc53f5d6d5622
    Related-bug: #1505430

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/268126

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/268126
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=cac3fb356f9fb2e6ca53e68f6114e73040fbbc92
Submitter: Jenkins
Branch: stable/liberty

commit cac3fb356f9fb2e6ca53e68f6114e73040fbbc92
Author: Dan Prince <email address hidden>
Date: Wed Nov 11 17:52:13 2015 -0500

    Generate a unique DeployIdentifier on updates

    This patch updates 'overcloud deploy' commands so that
    updates (re-running deploy on an existing stack) generate
    a unique DeployIdentifier parameter. By passing an unique
    DeployIdentifier parameter into the overcloud stack we
    force all puppet config tasks (which are idempotent
    and safe to exec multiple times) again.

    We've had several issues with puppet not-rerunning on
    a variety of config related changes and additionally recieved
    user feedback that not re-running all puppet deployments on
    an update is confusing.

    Change-Id: Ia659181872a6537cc169f1d1531ebf5d36160183
    Closes-bug: #1505430
    Depends-On: Ic352ddd30807dc378e5e7b6c396bc53f5d6d5622
    (cherry picked from commit ed945a9080556a20067be2a93831ff1f34b9ae9c)