heat stack-update wants all parameters and the template all over again

[Robert Collins]

If I'm changing one parameter in a stack - e.g. a service password definition, having to supply everything all over again is painful.

It would be awesome to be able to go
heat stack-update mycluster -P "AdminPassword=newvalue" and have that use the existing template in the live stack, and all the existing parameters just updating the value of the one parameter.

[Steve Baker (steve-stevebaker)]

This sounds like a feature that should be raised as a blueprint against the heat project.

[Robert Collins (lifeless)]

I'll let clint drive process :)

On 14 September 2013 08:24, Steve Baker <email address hidden> wrote:
> This sounds like a feature that should be raised as a blueprint against
> the heat project.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1224828
>
> Title:
> heat stack-update wants all parameters and the template all over again
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/python-heatclient/+bug/1224828/+subscriptions

--
Robert Collins <email address hidden>
Distinguished Technologist
HP Converged Cloud

[Angus Salkeld (asalkeld)]

just thinking how you would delete a parameter:

say we have "X=foo"
and the default is "X=this"
we can set to empty string "X="
how do we reset back to the default?

[Steve Baker (steve-stevebaker)]

I'm thinking there will need to be a toggle or a subcommand to differentiate a full update from a partial one.

Maybe the following:
heat stack-update --patch ...

And the resulting REST call would be a PATCH rather than a PUT:
https://restful-api-design.readthedocs.org/en/latest/methods.html#patch-vs-put

so to delete a parameter, a full PUT would be required, PATCH is just for changing existing values.

[huangtianhua (huangtianhua)]

Hi Steve Baker, the method to solve this bug is plan to add an api(PATCH)?

[Steve Baker (steve-stevebaker)]

Adding a PATCH api was my first thought here. But actually horizon has worked around this by pre-populating the update request with parameters and template fetched from a stack-show. Maybe we should just do this for stack-update cli too?

[Tim Schnell (tim-schnell)]

I think that we still need the ability to PATCH in an update. I think Horizon won't behave so well if some of the parameters come back with the "hidden" value (all asterisks). If we can PATCH the update then we won't require the user to remember the values for hidden parameters.

[Clint Byrum (clint-fewbar)]

One might argue that PATCH on the stack is actually the appropriate
action to use for update. After all, that is what we're doing to the
stack resource.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/107524

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/109042

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ton Ngo (<email address hidden>) on branch: master
Review: https://review.openstack.org/107524
Reason: Replaced by patch:
https://review.openstack.org/109042

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/109105

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/109284

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/109465

[Steve Baker (steve-stevebaker)]

Please remember that this bug is tracking to allow optional template and environment on stack update, not just the parameters.

[Ton Ngo (ton-i)]

Agree, I am finishing up the patches to handle the parameters, then will add new patches for environment and templates.
To reuse existing template, we will need to save a copy of the original template. As Zane pointed out, the current raw_template in the DB is not guaranteed to be the original template since it can be modified to reflect the real stack resources. Saving the original template would require a new column in the stack table and some code to manage it.
Since this costs some resource and adds some complexity, I had posted this question earlier on the ML for feedback . Clint felt the user should manage their own templates outside of Heat.
Does anyone else have feedback or suggestions?
Could there be other use cases requiring the original template?

[Pavlo Shchelokovskyy (pshchelo)]

Ton,
I really thought about adding just that extra column when trying to
implement cancel-update. In the end it looks like I managed to complete
without it, but IMO the implementation would be quite cleaner and more
straightforward.

Best,
Pavlo.

On Thu, Sep 4, 2014 at 1:25 AM, Ton Ngo <email address hidden> wrote:

> Agree, I am finishing up the patches to handle the parameters, then will
> add new patches for environment and templates.
> To reuse existing template, we will need to save a copy of the original
> template. As Zane pointed out, the current raw_template in the DB is not
> guaranteed to be the original template since it can be modified to reflect
> the real stack resources. Saving the original template would require a new
> column in the stack table and some code to manage it.
> Since this costs some resource and adds some complexity, I had posted this
> question earlier on the ML for feedback . Clint felt the user should
> manage their own templates outside of Heat.
> Does anyone else have feedback or suggestions?
> Could there be other use cases requiring the original template?
>
> --
> You received this bug notification because you are subscribed to heat.
> Matching subscriptions: heat bugs, heatclient
> https://bugs.launchpad.net/bugs/1224828
>
> Title:
> heat stack-update wants all parameters and the template all over again
>
> Status in Orchestration API (Heat):
> In Progress
> Status in Python client library for heat:
> In Progress
> Status in Tuskar:
> Triaged
>
> Bug description:
> If I'm changing one parameter in a stack - e.g. a service password
> definition, having to supply everything all over again is painful.
>
> It would be awesome to be able to go
> heat stack-update mycluster -P "AdminPassword=newvalue" and have that
> use the existing template in the live stack, and all the existing
> parameters just updating the value of the one parameter.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/heat/+bug/1224828/+subscriptions
>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/109042
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=b98c19fbbf0694924480ff9959a8db742e29dad6
Submitter: Jenkins
Branch: master

commit b98c19fbbf0694924480ff9959a8db742e29dad6
Author: Ton Ngo <email address hidden>
Date: Wed Jul 23 09:55:40 2014 -0700

    Pass flag to engine service to patch parameters

    For stack-update, add a new PATCH ReST API and a new CLI
    argument named existing-parameters to indicate that the set
    of parameters should be patched over the existing parameter
    from the DB. A new method in environment handles
    the patching.

    Partially-implements: blueprint troubleshooting-low-level-control
    Partial-Bug: 1224828
    Change-Id: I802e0dca44926be3a3f45fcaa995c866a4abf998

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/109284
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=b8ba69e05f8a51158f27142e354adc20a1e31149
Submitter: Jenkins
Branch: master

commit b8ba69e05f8a51158f27142e354adc20a1e31149
Author: Ton Ngo <email address hidden>
Date: Thu Jul 24 06:01:37 2014 -0700

    Pass list of parameters to engine service to reset

    For stack-update, a new argument named clear-parameters with
    a list is passed in from the CLI and the new PATCH ReST API
    to delete the indicated parameters from the existing parameters
    in the DB, allowing the default parameters in the template to
    be used. A new method in environment handles the reset.

    Partially-implements: blueprint troubleshooting-low-level-control
    Partial-Bug: 1224828
    Change-Id: Ia1270b679f27e264e6977c590d676b947c74c5da

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/109105
Committed: https://git.openstack.org/cgit/openstack/python-heatclient/commit/?id=6877ee1234869a59a9d32e47cf691b2b06472208
Submitter: Jenkins
Branch: master

commit 6877ee1234869a59a9d32e47cf691b2b06472208
Author: Winnie Tsang <email address hidden>
Date: Wed Aug 13 12:22:22 2014 -0700

    Add stack-update argument for patching parameters

    A new argument named existing for stack-update will be
    supported by the heat engine to use the parameters stored
    in the Db.

    Partially-implements: blueprint troubleshooting-low-level-control
    Partial-Bug: 1224828
    Change-Id: I30a2b223d7dd19a717ed989b419b1e6cb2ac143b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/109465
Committed: https://git.openstack.org/cgit/openstack/python-heatclient/commit/?id=8ad1ca3f4d8e0ef2ec07862ee23ee55335670430
Submitter: Jenkins
Branch: master

commit 8ad1ca3f4d8e0ef2ec07862ee23ee55335670430
Author: Winnie Tsang <email address hidden>
Date: Wed Aug 13 13:36:03 2014 -0700

    Add stack-update argument for reset parameters

    A new argument named clear-parameter for stack-update
    will be supported by the heat engine to reset the value of the
    parameter back to the default value. This argument can be
    specified multiple times.

    Partially-implements: blueprint troubleshooting-low-level-control
    Partial-Bug: 1224828
    Change-Id: Ib04dbc13978fdefd1bf3485f2654e139a3f1df8d

[Ton Ngo (ton-i)]

Per suggestion from Steve Baker and Angus, the -x option can be extended to allow reusing the existing template and registry.
Usage would be something like:
heat stack-create mystack -f mytemplate.yaml -e myenv.yaml
heat stack-update mystack -x -f mynewtemplate.yaml
heat stack-update mystack -x -e mynewenv.yaml
heat stack-update mystack -x

In other words, when -x is specified, the template and registry would become optional. If the template and/or registry are not specified, the existing one would be used.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/154618

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/154619

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/154620

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/154692

[Steven Hardy (shardy)]

What's the status of this, I see the patches appear to have stalled back in March, is anyone still actively working on this?

I'd like to see this fixed, so e.g you could do something like:

heat stack-update overcloud -P "Debug=True"

This would be a much nicer interface for large stacks when you only want to modify a small number of parameters.

[Steve Baker (steve-stevebaker)]

I believe we have working preservation of parameters with --existing, but I don't recall the status of not having to re-state the template and environments

[Ton Ngo (ton-i)]

As Steve mentioned, the existing parameters can be reused with the --existing option; you only need to specify the parameters you want to change.
For reusing the existing template and registry, there are several patches outstanding, but they have not been getting much reviews so it's not clear whether there is interest in these options. If there is, I can rebase them and get them going again.

[Steve Baker (steve-stevebaker)]

On 12/06/15 12:04, Ton Ngo wrote:
> If there is, I can rebase them and get them going again.
Yes please, I would like to see this feature in Liberty

[Ton Ngo (ton-i)]

Sounds great, we will resume this set of patches. For the environment file, currently only the parameters and registry are handled. I will look into handling the new hooks for stack breakpoint; this will probably require an additional patch.

[Steven Hardy (shardy)]

> Yes please, I would like to see this feature in Liberty

+1, I'm more than happy to commit to providing some review bandwidth if you're willing to rebase and start pushing the patches along again, I'm sorry if it was lack of reviews which stalled this - I think it will be a nice improvement when it lands, thanks!

[Steven Hardy (shardy)]

I've added Steve and I to the outstanding reviews - if you can rebase the heat and heatclient patches and confirm they're ready to test I can pull them and test via TripleO.

[Steven Hardy (shardy)]

AFAICT only the parameters part of this actually landed, so I'm setting back to in-progress to enable tracking of the remaining pieces.

Ton - any update on reviving the outstanding patches? Can we offer any assistance with that?

[Ton Ngo (ton-i)]

Thanks Steve, yes it should be in progress. I was tied up with the Magnum work recently, but will work on rebasing this set of patches this week.

[Moshe Elisha (melisha)]

+1 for the proposal summarized in comment #22 by ton-i.
This will be very useful especially in conjunction with the check stack feature.

[noa (noa-koffman)]

I see this feature should include passing the existing template and environment file,
is it also going to include the rest of the files passed when creating the stack? (such as custom type definitions and script files referred to by the template or environment file)?

[Steven Hardy (shardy)]

@noa (noa-koffman) - yes the desired end goal here is to allow passing only what has changed, e.g a PATCH update which allows you to pass e.g just one modified paramter, without any template, environment, files etc.

[Steven Hardy (shardy)]

Ok, looking at what needs to be revived with a view to rebasing:

Heat patches:

https://review.openstack.org/#/c/154618/ (I'm not sure this is still needed, as PATCH support for parameters already landed?)

https://review.openstack.org/#/c/154619 (support reusing existing environment)

https://review.openstack.org/#/c/154620 (support reusing existing template)

Heatclient patch:

https://review.openstack.org/#/c/154692/

As a first step I'd like to revive the patches enabling reusing existing environment/template.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/154619
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=fa6251de0d9dccd133b439cdafd69c4e0664be6b
Submitter: Jenkins
Branch: master

commit fa6251de0d9dccd133b439cdafd69c4e0664be6b
Author: Steven Hardy <email address hidden>
Date: Wed Jul 22 19:04:11 2015 +0100

    Reuse existing environment on PATCH update

    Extend the "existing" option to the engine update so we reuse
    all of the environment, not only the parameters, which means
    that when -x is specified during a stack-update we'll maintain
    any existing resource_registry (and files), then merge any
    environment passed during the update into those existing.

    This allows the users to optionally omit the environment when
    performing a stack update, which is particularly convenient
    if you're updating a stack which has a large or complex
    environment.

    Co-Authored-By: Ton Ngo <email address hidden>
    Partial-Bug: 1224828
    Change-Id: I3033063a06c3913a1ae89ea1ec1fdac688ebd2cf

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steven Hardy (<email address hidden>) on branch: master
Review: https://review.openstack.org/154618
Reason: This was combined with https://review.openstack.org/#/c/154619/ which has now merged

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/205755
Committed: https://git.openstack.org/cgit/openstack/python-heatclient/commit/?id=8ee2fc1b3d4df186d0bcc3e16353324a456a1fa0
Submitter: Jenkins
Branch: master

commit 8ee2fc1b3d4df186d0bcc3e16353324a456a1fa0
Author: Steven Hardy <email address hidden>
Date: Sat Jul 25 12:33:56 2015 +0100

    stack-update -x tolerate no template

    Replacement for the approach outlined in
    https://review.openstack.org/#/c/154692/

    This works with the following heat changes to enable
    full PATCH update functionality, including omitting
    the template or environment:

    https://review.openstack.org/#/c/205754/
    https://review.openstack.org/#/c/154619/

    Closes-Bug: #1224828
    Change-Id: I2a82936743badb69f0de5a6ca64f95ae63a35358

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/205754
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=a5cb850841ddea87f5e3702298295cc2a090f2fc
Submitter: Jenkins
Branch: master

commit a5cb850841ddea87f5e3702298295cc2a090f2fc
Author: Steven Hardy <email address hidden>
Date: Sat Jul 25 12:32:00 2015 +0100

    PATCH update reuse existing template

    This is a replacement for the approach outlined in
    https://review.openstack.org/#/c/154620, we reuse the previous
    template, even if the stack is in a failed state, by making use
    of the prev_raw_template now stored at the start of each update.

    Closes-Bug: 1224828
    Change-Id: I0294745dea0d25b91dbcc50a8bd55b3e1ea87d81

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ton Ngo (<email address hidden>) on branch: master
Review: https://review.openstack.org/154620
Reason: Superceded by patch https://review.openstack.org/#/c/205754/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/230055

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/230055
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=4f098762d640f3c8efef6869e47f264a06f795ff
Submitter: Jenkins
Branch: master

commit 4f098762d640f3c8efef6869e47f264a06f795ff
Author: Steven Hardy <email address hidden>
Date: Thu Oct 1 19:03:42 2015 +0100

    Add functional test coverage for PATCH updates

    Adds coverage for patch updates which didn't land previously with the
    code.

    Change-Id: I66b721d1af787e9d64b8b942818a6c8b4412fa7b
    Related-Bug: #1224828

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steven Hardy (<email address hidden>) on branch: master
Review: https://review.openstack.org/154692
Reason: This change has been obsoleted by I2a82936743badb69f0de5a6ca64f95ae63a35358

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/300676

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/300676
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=a00b2f890e604b662dfc43669776435f67be217e
Submitter: Jenkins
Branch: stable/liberty

commit a00b2f890e604b662dfc43669776435f67be217e
Author: Steven Hardy <email address hidden>
Date: Thu Oct 1 19:03:42 2015 +0100

    Add functional test coverage for PATCH updates

    Adds coverage for patch updates which didn't land previously with the
    code.

    Change-Id: I66b721d1af787e9d64b8b942818a6c8b4412fa7b
    Related-Bug: #1224828
    (cherry picked from commit 4f098762d640f3c8efef6869e47f264a06f795ff)