Can't ignore updates to OS::Nova::Server

[Steven Hardy]

When managing large groups of OS::Nova::Server resources inside e.g a ResourceGroup, you may want to change a property such that it only affects new instances of the resource.

Examples:
 - adding node-specific data to a mapping written via user_data
  - Changing the image to point to a new version, existing nodes are updated in-place

Currently we have image_update_policy, to which we could add an IGNORE option to cater for the latter example.

For the first example, we need a user_data_update_policy with a similar IGNORE option.

The use-case for this is allowing non-destructive stack updates for TripleO, but it's a general problem for maintaining large groups of servers over a long period of time, where a forced replacement or even rolling update are not desired.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/274101

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/274149

[Zane Bitter (zaneb)]

This problem is a consequence of two (IMHO) incorrect design decisions:

1) Not having separate Launch Configs for Autoscaling groups; and
2) Allowing ResourceGroup to exist

The good news is that Senlin corrects these mistakes.

The bad news is that apparently we are going to hack around it in the short term by adding non-local state that completely changes how Heat behaves in ways that increasingly remind one of http://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/#philosophy (search for "action at a distance").

[Steven Hardy (shardy)]

@Zane: I can appreciate your viewpoint, but given that ResourceGroup does exist, what would you suggest?

This is a blocker bug for TripleO because it's impossible to ever change anything in user_data for existing deployments, or you destroy every.single.node, which is invariably *never* what the operator actually wants.

What I have proposed is a simple and low-risk workaround for folks getting bitten by the incorrect design decisions you reference, which I see as a reasonable interim fix while we get to the point where folks could feasibly migrate to an alternative solution like Senlin.

AFAICT Senlin isn't a magic bullet here, firstly AIUI it's not really ready for production usage, you have to roll out an entirely new service, modify your templates, and even then I can't see how we'd migrate anything from exisitng ASG/RG resources anyway, so you're back to square 1 where the only option is to redeploy the world.

[Steve Baker (steve-stevebaker)]

I'm fine with a user_data_update_policy, its a perfectly reasonable thing to put in the template author's control.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steven Hardy (<email address hidden>) on branch: master
Review: https://review.openstack.org/274101
Reason: Heh, this has been described as pure-evil by zaneb and stevebaker doesn't like it either, so I'm going to abandon it in favour of one of their proposed alternatives.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steven Hardy (<email address hidden>) on branch: master
Review: https://review.openstack.org/274149

[Marios Andreou (marios-b)]

Hi all, I was wondering what the status is here. I see the two reviews by shardy at https://review.openstack.org/274101 and https://review.openstack.org/274149 have been abandoned (though sbaker has restored the dependent latter review).

From those reviews and the discussion above I'm lead to believe there is some debate around the approach being taken(!). I'm mainly interested in how we move forward. This bug blocks tripleo upgrades because of the user_data example shardy gives in the description (because we do this https://review.openstack.org/#/c/220057/ ).

Do you think it is possible to change shardy's existing reviews into something palatable to everyone? I can't see much actual discussion on the reviews (at least not of the sort to justify the abandonment) so given the discussion happened on irc/wherever else, I'm not sure how far the differing viewpoints diverge.

Assuming there is an alternative proposal can you please include a link here. Thanks for any info you can provide!

[Steven Hardy (shardy)]

Hi Marios - I debated this with Zane and Steve Baker, and was under the impression Steve was going to propose an alternative approach, and that my config-file approach was being rejected, hence abandoning the reviews.

My understanding is that Steve had an idea about a DiB element which would create the user, including retrieving the SSH key from the nova metadata (such as is done via cloud-init with the userdata approach).

I'm fine with doing that, but don't have bandwidth to work on that alternate solution, so I've assigned this to stevebaker.

I do think this is a heat issue tho, we removed the heat-admin user from our internal user-data, admittedly with a deprecation period, but we failed to provide any migration path for those relying on the feature.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/283311

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Baker (<email address hidden>) on branch: master
Review: https://review.openstack.org/283311
Reason: This is likely better done by the subclass building its own modified properties_schema

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/283379

[Marios Andreou (marios-b)]

sbaker: for clarity, if we were to land https://review.openstack.org/#/c/283379/ "Override OS::Nova::Server for user_data updates" into instack-undercloud which makes it so Server.USER_DATA is update_allowed should it fix the user_data issue in itself? I understand you're also going to try and land https://review.openstack.org/#/c/274149/3 "Add user_data_update_policy property to OS::Nova::Server" for Mitaka in heat, but afaics we can just use the former patch at #/c/283379 which fixes the USER_DATA itself.

WRT using the subclass of OS::Nova::Server in #/c/283379. AFAICS this is then always aliased with the def resource_mapping(): return {'OS::Nova::Server': ServerUpdateAllowed}. If we have an existing overcloud with 'normal' OS::Nova::Server and then update after adding this to your undercloud, will the existing servers survive (I'll add a note to the review too),

thanks again, marios

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Baker (<email address hidden>) on branch: master
Review: https://review.openstack.org/283379
Reason: See https://review.openstack.org/#/c/283832/

[Steve Baker (steve-stevebaker)]

I'm proposing that https://review.openstack.org/#/c/283832/ be the fix for stable/liberty tripleo, and that tripleo-heat-templates specifies a user_data_update_policy when a heat release contains this change https://review.openstack.org/#/c/274149/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/274149
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=d4188127a14686f6d9844180b977cf5fa05aa024
Submitter: Jenkins
Branch: master

commit d4188127a14686f6d9844180b977cf5fa05aa024
Author: Steve Baker <email address hidden>
Date: Wed Feb 24 16:25:16 2016 +1300

    Add user_data_update_policy property to OS::Nova::Server

    This may be set to either 'REPLACE' (default) or 'IGNORE'.
    This allows template authors to choose the desired behaviour when the
    user_data property is changed.

    Co-Authored-By: Steve Hardy <email address hidden>

    Change-Id: I3239c7252a2c329330283b86181abd52aee9e967
    Closes-Bug: #1539541

[Marios Andreou (marios-b)]

Update - the landed fix above worked fine for the user-data replacement issue but as discussed at https://bugzilla.redhat.com/show_bug.cgi?id=1314429 the problem persists because of updates to other properties of the OS::Nova::Server resource.

Steve Baker has a new review @ https://review.openstack.org/#/c/288273/ "Prevent any property change from replacing OS::Nova::Server"

to prevent any property update from replacing the server (for now)

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/heat 6.0.0.0rc1 release candidate.