deployment may be executed more than once.

[Qiming Teng]

SoftwareDeployment is meant to be executed only once, i.e. IN_PROGRESS -> FAILED, or IN_PROGRESS->COMPLETED.

Currently, if one has two SoftwareDeployment (say, A and B) applied to the same Server, at different point in time, when B is executed, A may be re-executed again. The reason is that the deployment metadata for a Server is built by querying the heat-client, heat-api, ... and eventually the DB API.

We need to check if a SoftwareDeployment is already 'FAILED' or 'COMPLETE' when building the metadata for a server and exclude them.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/84658

[Steve Baker (steve-stevebaker)]

Deployments have been designed with the assumption that config workloads are idempotent, so executing the same deployment more than once is not necessarily a problem.

Also you can use the deployment name to specify execution order without specifying dependencies between deployment resources. Say you have 2 deployment resources which don't depend on each other, but do have an execution order defined by their names. Lets say the second resource gets deployed first:

- 20_run_me_second

Then later the first resource is deployed:

- 10_run_me_first
- 20_run_me_second

The second deployment might have a conditional which checks for state from the first, and then does something when it finds it. So this is a case where your fix would break this case.

I accept there may be some configs which are not idempotent, so the fix for that would be to pass in enough information so the config can decide whether to execute or not. This is tricky because os-collect-config will trigger os-refresh-config if any deployments data changes, so we can't just pass in IN_PROGRESS/COMPLETE. What we could do is set an input deploy_action_start which contains a timestamp. Configs could then read that input and decide whether to execute (by comparing to a previously stored deploy_action_start)

[Qiming Teng (tengqim)]

Steve, thanks for the comments.

I don't think it a good idea to assume any implicit dependencies/ordering among the deployments and I don't see the value of the assumption about idempotency. Let me try explain them in my not so good English.

1. Idempotency

When we permit a deployment to be applied more than once even if we already see it failed or complete, we have to put a warning somewhere so users will know it and be very careful when writing their SoftwareConfig.peroperties.config scripts. This means software packages may get downloaded again, a service may be restarted, a database table is recreated, a user account for the application is registered again ... or whatever. For the user, he/she has to make a lot more efforts due to the 'idempotency' assumption we have on deployments. Deployments are not just trivial meta-data, they are supposed to perform various actions by materializing the work demanded by a SoftwareConfig.

One possible solution is to make os-*-config much more sophisticated, so that it can figure out whether and when to apply a deployment. That means we need to change both the Heat side and the os-*-config logic. Heat will specify whether a SoftwareDeployment is supposed to be applied once, or to be applied whenever any other SoftwareDeployment has been changed. I don't think we are going that far down this road.

I am aware of the idempotency design of some CM tools. Even in those cases, the in-instance agent is smart enough to check whether a change should be applied. But I am not convinced that we really want to impose that requirement to os-*-config components.

2. Dependency

Having all deployments automatically reapplied whenever any changes happen to one of them? Personally, I think that is a side-effect (if you don't think it a bug) of always having all deployments listed in the metadata.

My understanding is that we are introducing other triggers for SoftwareDeployment beyond the existing { CREATE, UPDATE, ... } collection. We are in fact extending it to include { CREATE_OF_OTHER_SDS, UPDATE_TO_OTHER_SDS, ... }.

There are already many other ways to express dependencies among deployments, EXPLICITLY. If a user has one deployment (A) dependent on another (B), he/she is supposed to express this dependency in the template, rather than just having them appear in the same list in the metadata.

The user has the final decision what should be done when one of deployment is changed. If only A need to be changed, that is all that will be included in the list. If however, both A and B need to be changed, he will explicitly create two derived configs.

3. Ordering/Timing

As for ordering, I agree with you. The only thing we need to do is to let the users know: deployments, when applied simultaneously, are executed in name order.

Even after we have applied the first level of filter so that only IN_PROGRESS deployments to appear when metadata are built, there are still more work to do, as implied in your timestamp based proposal. A deployment A may be still underway (IN_PROGRESS) when another deployment is changed (which is also IN_PROGRESS). In that case, we may need one more status (inside the VM),...

[Thomas Spatzier (thomas-spatzier)]

I think requiring idempotency is a hard requirement. I can see this work for some automation, but if somebody just wants to use an existing script this is hard to achieve without a lot of effort. For those simple use cases it would be desirable to have the script executed only once.

Steve, when you talked about doing all this status checking on previously run deployments, did you mean to do this inside the user defined script? Or inside the hooks? The latter would be ok I guess, because then there would be one place to handle this. Having to implement this logic in each script could make things really complicated for users.

Just filtering out everything that is failed also does not seem like the best solution. The status should be there, but just needs to be properly handled.

My 2 cents.

[Clint Byrum (clint-fewbar)]

Thanks for the excellent summary. As a user of software deployments, I
_do not_ want them to go away just because I said I completed them. When
I reboot, I will likely want to grab the latest copy and re-apply to
ensure that the state of the system is correct.

That state is not for the deployment tool to refer to, it is for things
further down the dependency tree which want to wait for that state.

Excerpts from Qiming Teng's message of 2014-04-03 02:46:13 UTC:
> Steve, thanks for the comments.
>
> I don't think it a good idea to assume any implicit
> dependencies/ordering among the deployments and I don't see the value of
> the assumption about idempotency. Let me try explain them in my not so
> good English.
>
> 1. Idempotency
>
> When we permit a deployment to be applied more than once even if we
> already see it failed or complete, we have to put a warning somewhere so
> users will know it and be very careful when writing their
> SoftwareConfig.peroperties.config scripts. This means software packages
> may get downloaded again, a service may be restarted, a database table
> is recreated, a user account for the application is registered again ...
> or whatever. For the user, he/she has to make a lot more efforts due to
> the 'idempotency' assumption we have on deployments. Deployments are
> not just trivial meta-data, they are supposed to perform various actions
> by materializing the work demanded by a SoftwareConfig.
>
> One possible solution is to make os-*-config much more sophisticated, so
> that it can figure out whether and when to apply a deployment. That
> means we need to change both the Heat side and the os-*-config logic.
> Heat will specify whether a SoftwareDeployment is supposed to be applied
> once, or to be applied whenever any other SoftwareDeployment has been
> changed. I don't think we are going that far down this road.
>
> I am aware of the idempotency design of some CM tools. Even in those
> cases, the in-instance agent is smart enough to check whether a change
> should be applied. But I am not convinced that we really want to impose
> that requirement to os-*-config components.
>
> 2. Dependency
>
> Having all deployments automatically reapplied whenever any changes
> happen to one of them? Personally, I think that is a side-effect (if
> you don't think it a bug) of always having all deployments listed in the
> metadata.
>
> My understanding is that we are introducing other triggers for
> SoftwareDeployment beyond the existing { CREATE, UPDATE, ... }
> collection. We are in fact extending it to include {
> CREATE_OF_OTHER_SDS, UPDATE_TO_OTHER_SDS, ... }.
>
> There are already many other ways to express dependencies among
> deployments, EXPLICITLY. If a user has one deployment (A) dependent on
> another (B), he/she is supposed to express this dependency in the
> template, rather than just having them appear in the same list in the
> metadata.
>
> The user has the final decision what should be done when one of
> deployment is changed. If only A need to be changed, that is all that
> will be included in the list. If however, both A and B need to be
> changed, he will explicitly create two derived ...

[Qiming Teng (tengqim)]

Maybe we need to add a property to deployments to indicate the frequency desired. Something like 'per-boot', 'once', 'always'. IIRC, cloud-init has something similar.

Combined with the intended frequency and current status 'NEW', 'IN_PROGRESS', 'COMPLETE/FAILED', the os-*-config scripts can easily tell whether a deployment need to be executed.

A blueprint for this?

[Steve Baker (steve-stevebaker)]

Your desired behaviour can still be achieved without any heat changes. A patch could be made to the heat-script hook which consumes an input called deployment_run_policy, then you can just set that value in your config resource default, or deployment resource input_values.

[Qiming Teng (tengqim)]

Steve, if I am understanding this correctly, we cannot get the state of a SoftwareDeployment from inside a VM. That means heat-config or heat-config-script won't be able to tell whether a SD has been deployed. What they can get today is a 'derived config'.

I don't think we have to cache the statuses of SDs from the hooks side. Any thoughts?

[Steve Baker (steve-stevebaker)]

The issue is that heat-engine can never know the true state of what is inside compute resource, so it shouldn't try to guess. Instead it should always provide all of the deployment data it knows about so that the hooks or the config scripts inside the compute resource can make their own decision about what config to trigger.

If the hooks or scripts do not have enough information yet to make that decision then that is the fix we need to make in heat-engine. It seems to me that adding a deploy_action_start input would provide just enough information to make this work, but I'm sure there is other potential extra data which would work too.