55-heat-config state is stored in /var/run, which is lost on reboot

[Steve Baker]

It would probably cause the user less surprise to not rerun every deployment on reboot, so I would suggest a better location for this state should be /var/lib/heat-config

[Steve Baker (steve-stevebaker)]

I don't think any migration logic is needed from /var/run/heat-config to /var/lib/heat-config since the hook is baked into the image and you don't get the new location without a reboot anyway

[Steve Baker (steve-stevebaker)]

On second thoughts, migration logic is appropriate if the heat-templates package is updated in-place

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/242924

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/242924
Committed: https://git.openstack.org/cgit/openstack/heat-templates/commit/?id=c002854fa4ef786069b323f8bd15a77d42447884
Submitter: Jenkins
Branch: master

commit c002854fa4ef786069b323f8bd15a77d42447884
Author: Steve Baker <email address hidden>
Date: Mon Nov 9 17:43:17 2015 +1300

    Move deployed dir from /var/run to /var/lib

    Deployments should not be run again when a server is rebooted because
    heat will likely not be listening for the deployment signals. This
    change moves the deployed directory from /var/run/heat-config/deployed
    to /var/lib/heat-config/deployed so that deployed state is persisted
    across reboots.

    There is migration logic to move the existing files from the old to the
    new location, this will only be run on the first run of 55-heat-config
    after its package is updated.

    Change-Id: I3d305a4ac5b68c29037760682d37e5b9a530828e
    Closes-Bug: #1513220

[James Slagle (james-slagle)]

thanks for the fix.

would it be possible to take this a bit further to have something that could recreate what had been in /var/run if it has already been deleted (due to reboot, etc).

It could look at the contents of /var/lib/os-collect-config and /var/lib/heat-config, assume everything was already applied so that it didn't rerun any deployments, but just generated what should be in /var/run/heat-config, then migrates that to /var/lib/heat-config.

[James Slagle (james-slagle)]

from shardy on irc:

12:17 < shardy> slagle: Ok, well the data used as an input to heat-config is still there (the heat metadata asociated with the server), so it must be possible to re-created some dummy deployments data
12:19 < shardy> https://github.com/openstack/tripleo-image-elements/blob/master/elements/os-refresh-config/os-refresh-config/post-configure.d/99-refresh-completed#L66
12:19 < shardy> e.g a loop like that which touches the file and stops heat-config running
12:19 * shardy checks what heat-config expects
12:20 < shardy> slagle: I think it's just checking for <deployment_id>.json
12:20 < shardy> so a loop touching the files is probably enough

I think this could work. We just need something so that the deployments are not re-applied if they've already been lost/deleted.

[James Slagle (james-slagle)]

Hey, so I took a shot at a script to restore /var/run/heat-config/deployed. It seems to do the right thing. This is something TripleO could use out of band as a manual fix.

But, I figured it could be generally useful to anyone encountering this issue, since it effect anyone using SoftwareConfig/SoftwareDeployment.

Is this something of interest?

I'm unsure where I might commit this, or where I would document a process on how to use it. Any ideas on that?

The script is attached.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/251551

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/251551
Committed: https://git.openstack.org/cgit/openstack/heat-templates/commit/?id=50c18db21e0577d31425bb000792c42f372b7246
Submitter: Jenkins
Branch: master

commit 50c18db21e0577d31425bb000792c42f372b7246
Author: James Slagle <email address hidden>
Date: Mon Nov 30 16:49:41 2015 -0500

    Add script so that deployments won't get rerun

    This script creates the necessary files under /var/run/heat-config so that the
    next time configure.d/55-heat-config is executed via os-refresh-config, no
    deployments that were already known are rerun.

    This is a workaround for https://bugs.launchpad.net/heat-templates/+bug/1513220
    in scenarios where /var/run/heat-config has already been lost due to a scenario
    such as a system reboot.

    This script is a manual workaround, and should only be executed on instances
    where it's known that the deployments do not need to be reapplied.

    Change-Id: I3ac4e280a39acb893a8ecc94712eb6265a1236d2
    Partial-Bug: #1513220