parameter_merge_strategies: merge behaves like overwrite

[John Fulton]

When using the environments/hyperconverged-ceph.yaml [1] to deploy an HCI (nova/ceph on one node) overcloud, compute nodes are deployed with working OSDs, but nova compute services are not configured. For example, nova.conf is empty, nova processes are not running on the compute nodes, and nova's systemd unit files show nova services to be disabled. The deploy completes successfully but nova doesn't work. Other services, e.g. glance and cinder (backed by Ceph) do work.

It's as if the parameter_merge_strategies merge strategy [1] is behaving like overwrite instead [2].

[1] https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/environments/hyperconverged-ceph.yaml#L10-L11
[2] http://docs.openstack.org/developer/heat/template_guide/environment.html#environment-merging

[Giulio Fidente (gfidente)]

I am debugging the POST request and it looks like tripleoclient is passing via heatclient parameter_merge_strategies as intended:

2016-10-21 08:43:27.954 10409 DEBUG heatclient.common.http [-] curl -g -i -X POST -H 'X-Auth-User: admin' -H 'X-Auth-Token: {SHA1}5b4a7dc7215ce4ce0f124f0f454a3d1672c9fc9b' -H 'X-Region-Name: regionOne' -H 'Accept: application/json' -H 'User-Agent: python-heatclient' -H 'Content-Type: application/json' -d '{"stack_name": "overcloud", "environment": {"parameter_defaults": {"MysqlMaxConnections": 8192, "ControllerCount": 3, ... }, "parameter_merge_strategies": {"ComputeServices": "merge"}, "resource_registry": {"OS::TripleO::Services::Timezone": "http://192.168.1.1:8080/v1/AUTH_b0ffd5e578ee44ebb9bfcc9a5425426a/overcloud/puppet/services/time/timezone.yaml", ...}}

the list of parameter_defaults and resource_registry is longer but parameter_merge_strategies seems to be there as wanted

[Rabi Mishra (rabi)]

So, the merge_strategies were never intended to merge schema_defaults with parameter_defaults or parameters.

Merge strategies can merge parameter_defaults in multiple env files or/and parameters in multiple env files. But not merge schema defaults with param_defaults or param_defaults with parameters.

Though I'm not very keen, if everyone agrees we may allow merging template schema defaults with parameter_defualts, which woul probably solve your usecase.

There is also a bug which results in duplicates, as we are merging(overwrite old implementation) in the client as well as in the server. We would fix that.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/390064

[Steven Hardy (shardy)]

IMO this is not a heat bug, we should fix this in tripleo-heat-templates

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/391064

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/390064
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=2407c8643689042f1ab9cd582af477bdc3a9860f
Submitter: Jenkins
Branch: master

commit 2407c8643689042f1ab9cd582af477bdc3a9860f
Author: rabi <email address hidden>
Date: Wed Nov 2 08:41:54 2016 +0530

    Ignore merged env if environment files are specified

    Ignore merged environment from the client, if there are
    'environment_files' in the request for merging in the server.
    This would avoid the duplicates when 'parameter_merge_strategies'
    are specified in the first env file.

    Change-Id: I4ee274d69e42a47c4b8fd5a25567fb2eb144a8cc
    Related-Bug: #1635409

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/394442

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/391064
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d297e02c0396e24b4dcb8ab8daadaee724277d61
Submitter: Jenkins
Branch: master

commit d297e02c0396e24b4dcb8ab8daadaee724277d61
Author: Steven Hardy <email address hidden>
Date: Thu Oct 27 09:38:43 2016 +0100

    Move per role Services defaults into environment file

    For parameter merge strategies to work we need to merge multiple environment
    files, which doesn't consider the defaults defined in the heat template.

    Moving where we define these defaults will enable the merge strategies
    applied when appending services to roles in environment files to work.

    Change-Id: I1ef1ad685c8a15308d051665c576a98b277f2496
    Closes-Bug: #1635409

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/394442
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a9ff4c53fb1def00d61c1f08a3ef81a1f173a27b
Submitter: Jenkins
Branch: stable/newton

commit a9ff4c53fb1def00d61c1f08a3ef81a1f173a27b
Author: Steven Hardy <email address hidden>
Date: Thu Oct 27 09:38:43 2016 +0100

    Move per role Services defaults into environment file

    For parameter merge strategies to work we need to merge multiple environment
    files, which doesn't consider the defaults defined in the heat template.

    Moving where we define these defaults will enable the merge strategies
    applied when appending services to roles in environment files to work.

    Change-Id: I1ef1ad685c8a15308d051665c576a98b277f2496
    Closes-Bug: #1635409
    (cherry picked from commit da7d9d97da28bf50b0cc6fb5110ac7eb44c70224)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 5.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 6.0.0.0b1 development milestone.

[Steven Hardy (shardy)]

The tripleo fix turned out to be insufficient, reopening

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/402396

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/402396
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2a7e0445187df045d61b6b34b6943a3c6a448308
Submitter: Jenkins
Branch: master

commit 2a7e0445187df045d61b6b34b6943a3c6a448308
Author: Giulio Fidente <email address hidden>
Date: Fri Nov 25 11:32:49 2016 +0100

    Provide full list of services for Compute role in HCI scenario

    Until bug #1635409 is fixed, we can provide the full list of
    services needed on the Compute role, plus CephOSD, in the
    hyperconverged-ceph environment file, preserving the user
    experience.

    Change-Id: I42409bc098c740759b378969526e13efaf002d3c
    Related-Bug: #1635409

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/403142

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/403142
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e2ab4915a969d0d6ae9e4c9b55aa9c1547cfb04c
Submitter: Jenkins
Branch: stable/newton

commit e2ab4915a969d0d6ae9e4c9b55aa9c1547cfb04c
Author: Giulio Fidente <email address hidden>
Date: Fri Nov 25 11:32:49 2016 +0100

    Provide full list of services for Compute role in HCI scenario

    Until bug #1635409 is fixed, we can provide the full list of
    services needed on the Compute role, plus CephOSD, in the
    hyperconverged-ceph environment file, preserving the user
    experience.

    Change-Id: I42409bc098c740759b378969526e13efaf002d3c
    Related-Bug: #1635409
    (cherry picked from commit 2a7e0445187df045d61b6b34b6943a3c6a448308)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 5.1.0 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/434848

[OpenStack Infra (hudson-openstack)]

Change abandoned by John Fulton (<email address hidden>) on branch: master
Review: https://review.openstack.org/389342
Reason: environments/hyperconverged-ceph.yaml went in a different direction and this patch no longer applies.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/434848
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1a719a037df585a45e57316a2a20d6282fe5ba8b
Submitter: Jenkins
Branch: master

commit 1a719a037df585a45e57316a2a20d6282fe5ba8b
Author: Giulio Fidente <email address hidden>
Date: Thu Feb 16 12:47:33 2017 +0100

    Align HCI environment file with list of services from roles_data

    Until we get bug #1635409 fixed we'll have to keep the two lists
    in sync.

    Change-Id: Ifd996bd4c95f901f242696b37e179073be6334d0
    Related-Bug: #1635409

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/438036

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/438036
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2358764c0656a06821d264aca04e72fabd7fe9be
Submitter: Jenkins
Branch: stable/ocata

commit 2358764c0656a06821d264aca04e72fabd7fe9be
Author: Giulio Fidente <email address hidden>
Date: Thu Feb 16 12:47:33 2017 +0100

    Align HCI environment file with list of services from roles_data

    Until we get bug #1635409 fixed we'll have to keep the two lists
    in sync.

    Change-Id: Ifd996bd4c95f901f242696b37e179073be6334d0
    Related-Bug: #1635409
    (cherry picked from commit 1a719a037df585a45e57316a2a20d6282fe5ba8b)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/444226

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/444226
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=0c1ed05bb556743126af273eac11daeb47bded98
Submitter: Jenkins
Branch: master

commit 0c1ed05bb556743126af273eac11daeb47bded98
Author: Steven Hardy <email address hidden>
Date: Fri Mar 10 11:15:26 2017 +0000

    UpdateCapabilitiesAction add purge_missing option

    Currently any paths not included in the input map are ignored,
    but for the CLI we have an exact list we want to set, which will
    be easier with this option.

    Change-Id: I7ebc8b657ff5ab9cc275b29b4d17053888a251ee
    Partial-Bug: #1635409

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/448209

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/449633

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/450262

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/450263

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/450264

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/449633
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=040e997867b964b9ea197cf0d2f1cab9eeb2b3d2
Submitter: Jenkins
Branch: master

commit 040e997867b964b9ea197cf0d2f1cab9eeb2b3d2
Author: Steven Hardy <email address hidden>
Date: Fri Mar 24 12:25:41 2017 +0000

    Move clients into class constructor

    Currently we reference the various clients in several functions, so it's
    cleaner if we just define them in the constructor vs passing them around
    between functions etc. In particular the swift client will be needed
    in several functions if we're to break up the plan-update logic related
    to handling environments effectively, so we may as well move all clients
    for consistency (when more validations are done via mistral we can probably
    remove some of these altogether e.g ref the validations-in-workflows
    blueprint).

    Change-Id: I8e3ca726ec6d873529127c2326f1b65dbb2bc22e
    Partial-Bug: #1635409

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/450708

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/450709

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/450262
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=ab98502a65fa04ff943f9f105dd27dff51e48ad3
Submitter: Jenkins
Branch: master

commit ab98502a65fa04ff943f9f105dd27dff51e48ad3
Author: Steven Hardy <email address hidden>
Date: Fri Mar 24 14:46:51 2017 +0000

    Add breakpoint cleanup env during plan processing

    Move this environment addition to an earlier function, so we can
    decouple the plan creation and environment processing from the actual
    heat_deploy.

    Change-Id: Icbeb2a3f6e819b213590bfb0e237d86ef49a3b1f
    Partial-Bug: #1635409

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/450263
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=91634d82d7d71c59b80e2b3e06ae778f976c1d82
Submitter: Jenkins
Branch: master

commit 91634d82d7d71c59b80e2b3e06ae778f976c1d82
Author: Steven Hardy <email address hidden>
Date: Fri Mar 24 14:54:15 2017 +0000

    Validate NtpServers during plan processing

    Moving this to an earlier function allows us to decouple the plan
    processing from the heat_deploy which deploys the plan. In future
    it would be good to move this to a mistral validation instead of
    a client-side special case.

    Change-Id: If9e9fceb85408221cebbbacb8908f73b42268c19
    Partial-Bug: #1635409

[Ben Nemec (bnemec)]

There are a lot of patches that have merged for this bug, does it need to stay open?

[Steven Hardy (shardy)]

@Ben - yes the heat part of this is fixed, but we still need changes to the plan management in tripleo, e.g to upload discrete environments so heat can do server side merging vs the current situation where tripleoclient merges all the -e options into one environment before passing it to heat.

I have some patches for this which I've revived since the heat parts of this landed so we can make progress again:

https://review.openstack.org/#/q/status:open+project:openstack/python-tripleoclient+branch:master+topic:bug/1635409

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/450264
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=ca8e84f0c34f45a714fbfdc318ad7caf9eddfeaf
Submitter: Jenkins
Branch: master

commit ca8e84f0c34f45a714fbfdc318ad7caf9eddfeaf
Author: Steven Hardy <email address hidden>
Date: Mon Mar 27 13:22:50 2017 +0100

    Write user parameters environment to swift

    Currently we only merge this into the local merged environment,
    but instead we should write it as a separate user-environment in
    swift, which will enable heat server side parameter merging to
    work in future. The _write_user_environment function is added to
    handle writing the file to the local tmpdir tht_root, and uploading
    the same data to the plan.

    Partial-Bug: #1635409
    Change-Id: I74bde48ef18c2e59233b7535578f394fbd1f7553

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/450708
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=9bb593b85985821af5099055437ba80655cd4a7e
Submitter: Jenkins
Branch: master

commit 9bb593b85985821af5099055437ba80655cd4a7e
Author: Steven Hardy <email address hidden>
Date: Mon Mar 27 17:29:09 2017 +0100

    Write rhel registration parameters env to swift

    Similar to the user parameters, we need to upload these as an
    enivironment to the plan container, so that in future we can enable
    passing a list of un-merged environments to heat.

    Change-Id: I1f28279da5e7d728a7ce8cd916bcf9031b9fe302
    Partial-Bug: #1635409

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/450709
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=39293a7b1727d6b321b27ecc55a89ba0e97e6840
Submitter: Jenkins
Branch: master

commit 39293a7b1727d6b321b27ecc55a89ba0e97e6840
Author: Steven Hardy <email address hidden>
Date: Mon Mar 27 18:36:14 2017 +0100

    Write breakpoint cleanup env to swift

    To be consistent with the other parameters/registration environment
    write this to swift, such that we can move to server side merging
    of environments in future.

    Change-Id: I11f31ec3a356b32de34da1b128b6ec776550b5c5
    Partial-Bug: #1635409

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/509760

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/509760
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=313faad5f13abf0fddd85d07706fded6ed02bf38
Submitter: Zuul
Branch: master

commit 313faad5f13abf0fddd85d07706fded6ed02bf38
Author: Steven Hardy <email address hidden>
Date: Tue Oct 31 11:26:09 2017 +0000

    Return Environment from validate_template

    This is to enable preview of the merged environment
    without merging the environment on the client side.

    Related-Bug: #1635409
    Change-Id: I7ec3af729a65164230153021f438bf226cc5e858

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/624625

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steven Hardy (<email address hidden>) on branch: master
Review: https://review.openstack.org/448209

[Alex Schultz (alex-schultz)]

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.opendev.org/624625
Reason: This review is > 90 days without comment, and failed Zuul the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results. For more details check policy https://specs.openstack.org/openstack/tripleo-specs/specs/policy/patch-abandonment.html