Custom role requires manual environment/files

[Steven Hardy]

There are some usability issues with the custom roles interfaces, because we don't currently template default mappings for various per-role things in the base overcloud-resource-registry-puppet.yaml file.

It's possible to work around these things by copying files from existing roles and making a bespoke additional environment (or hacking on overcloud-resource-registry-puppet.yaml by copying the relevant lines), but this isn't convenient from a user interface perspective.

Instead we should template as much of this as possible by default, such that just adding a role name and list of services to roles_data.yaml is enough to create a custom role.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/378735

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/378736

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/378737

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/378750

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/378735
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7e9fd59f545a09b9ffe3ce549d49b4b4dd465684
Submitter: Jenkins
Branch: master

commit 7e9fd59f545a09b9ffe3ce549d49b4b4dd465684
Author: Steven Hardy <email address hidden>
Date: Fri Sep 23 13:51:45 2016 +0100

    j2 template per-role things in default registry

    The default resource-registry file contains a bunch of per-role
    things which mean you need to cut/paste into a custom environment
    file for custom roles, even if you only want the defaults like the
    built-in roles. Using j2 we can template these just like in the
    overcloud.j2.yaml and other files.

    Change-Id: I52a9bffd043ca8fb0f05077c8a401a68def82926
    Partial-Bug: #1626976

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379452
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b0d660702e837d493068055d85a511ef39afb510
Submitter: Jenkins
Branch: master

commit b0d660702e837d493068055d85a511ef39afb510
Author: Carlos Camacho <email address hidden>
Date: Thu Sep 29 14:57:36 2016 +0200

    Move the rest of static roles resource registry entries to j2

    Moving the rest of the static based resource registry
    entries to j2, this allows to extend the content of the
    template to the roles_list.

    Also moved the templates to correspond with the role name.

    Partial-Bug: #1626976

    Change-Id: I1cbe101eb4ce5a89cba5f2cc45cace43d3380f22

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/380443

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/378736
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b3c73a7e9924037ab29aff547b3096c8f5fb1a62
Submitter: Jenkins
Branch: master

commit b3c73a7e9924037ab29aff547b3096c8f5fb1a62
Author: Steven Hardy <email address hidden>
Date: Fri Sep 23 15:39:33 2016 +0100

    Replace per role manifests with a common role manifest

    This removes the (nearly empty) per role manifests, and
    replaces them with a generic manifest, where we use str_replace
    to substitute the role name at runtime (or in some cases a
    subset of the name for backwards compatibility)

    Change-Id: I79da0f523189959b783bbcbb3b0f37be778e02fe
    Partial-Bug: #1626976

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/380443
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d636e3a19fbfe997648465d24bd2231ce86373db
Submitter: Jenkins
Branch: stable/newton

commit d636e3a19fbfe997648465d24bd2231ce86373db
Author: Carlos Camacho <email address hidden>
Date: Thu Sep 29 14:57:36 2016 +0200

    Move the rest of static roles resource registry entries to j2

    Moving the rest of the static based resource registry
    entries to j2, this allows to extend the content of the
    template to the roles_list.

    Also moved the templates to correspond with the role name.

    Partial-Bug: #1626976

    Change-Id: I1cbe101eb4ce5a89cba5f2cc45cace43d3380f22
    (cherry picked from commit b0d660702e837d493068055d85a511ef39afb510)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/381757

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/378750
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=fb38363bd58fc0f8c697c81de431f30faa4630a9
Submitter: Jenkins
Branch: master

commit fb38363bd58fc0f8c697c81de431f30faa4630a9
Author: Steven Hardy <email address hidden>
Date: Wed Sep 28 17:05:37 2016 +0100

    Modify j2 templating to allow role files generation

    Currently we only allow transformation of a single *.j2.yaml
    file into a *.yaml file - this adds the option to specify a
    *.role.j2.yaml file, which will be rendered once per role to create
    multiple files (one per role).

    Co-Authored-By: Carlos Camacho <email address hidden>

    Change-Id: I9f920e191344040a564214f3f9a1147b265e9ff3
    Partial-Bug: #1626976

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/381903

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steven Hardy (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/381903
Reason: Abandoning until the tht patch which depends-on this lands

[Dan Sneddon (dsneddon)]

A related bug covers NIC configuration templates for custom roles: https://bugs.launchpad.net/tripleo/+bug/1625558

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steven Hardy (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/381903
Reason: We need the tht patch to land before this is restored

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/381903
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=0e529ba7046577518788331ec1d1f7a951afe8a6
Submitter: Jenkins
Branch: stable/newton

commit 0e529ba7046577518788331ec1d1f7a951afe8a6
Author: Steven Hardy <email address hidden>
Date: Wed Sep 28 17:05:37 2016 +0100

    Modify j2 templating to allow role files generation

    Currently we only allow transformation of a single *.j2.yaml
    file into a *.yaml file - this adds the option to specify a
    *.role.j2.yaml file, which will be rendered once per role to create
    multiple files (one per role).

    Co-Authored-By: Carlos Camacho <email address hidden>

    Change-Id: I9f920e191344040a564214f3f9a1147b265e9ff3
    Partial-Bug: #1626976
    (cherry picked from commit fb38363bd58fc0f8c697c81de431f30faa4630a9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/378737
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=94fddffa6d3e98df227803cca0879855fe47e6d8
Submitter: Jenkins
Branch: master

commit 94fddffa6d3e98df227803cca0879855fe47e6d8
Author: Steven Hardy <email address hidden>
Date: Wed Sep 28 17:03:42 2016 +0100

    j2 template role config templates

    This means the user won't have to manually specify e.g the
    OS::TripleO::ACustomRoleConfig resource manually.

    Partial-Bug: 1626976
    Change-Id: I063571d4c5cbc2f295a7a044d81c27d703bd0e10
    Depends-On: I9f920e191344040a564214f3f9a1147b265e9ff3

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/381975
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=883addf267933395c580e0eab3efc379401c946c
Submitter: Jenkins
Branch: master

commit 883addf267933395c580e0eab3efc379401c946c
Author: Carlos Camacho <email address hidden>
Date: Tue Oct 4 18:28:39 2016 +0200

    Move the main template files for defalut services to new syntax generation

    When generating these templates, we should
    create them with the "-role" appended as they will
    be generated from a role.role.j2.yaml file.

    i.e. role.role.j2.yaml will generate <service>-role.yaml
         config.role.j2.yaml will generate <service>-config.yaml

    Partial-Bug: #1626976
    Change-Id: I614dc462fd7fc088b67634d489d8e7b68e7d4ab1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/382137

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/381757
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=766d9ad5f5d2f38f5107cdbb48c0d8ed4af1c78d
Submitter: Jenkins
Branch: stable/newton

commit 766d9ad5f5d2f38f5107cdbb48c0d8ed4af1c78d
Author: Steven Hardy <email address hidden>
Date: Fri Sep 23 15:39:33 2016 +0100

    Replace per role manifests with a common role manifest

    This removes the (nearly empty) per role manifests, and
    replaces them with a generic manifest, where we use str_replace
    to substitute the role name at runtime (or in some cases a
    subset of the name for backwards compatibility)

    Change-Id: I79da0f523189959b783bbcbb3b0f37be778e02fe
    Partial-Bug: #1626976
    (cherry picked from commit b3c73a7e9924037ab29aff547b3096c8f5fb1a62)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/382276

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/382276
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3a3f27ae34005d358e632884977668a8292f7719
Submitter: Jenkins
Branch: stable/newton

commit 3a3f27ae34005d358e632884977668a8292f7719
Author: Steven Hardy <email address hidden>
Date: Wed Sep 28 17:03:42 2016 +0100

    j2 template role config templates

    This means the user won't have to manually specify e.g the
    OS::TripleO::ACustomRoleConfig resource manually.

    Partial-Bug: 1626976
    Change-Id: I063571d4c5cbc2f295a7a044d81c27d703bd0e10
    Depends-On: I9f920e191344040a564214f3f9a1147b265e9ff3
    (cherry picked from commit 94fddffa6d3e98df227803cca0879855fe47e6d8)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/382137
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=465dcdfa5ea8d94b6a37036c3e34537e0da2037a
Submitter: Jenkins
Branch: stable/newton

commit 465dcdfa5ea8d94b6a37036c3e34537e0da2037a
Author: Carlos Camacho <email address hidden>
Date: Tue Oct 4 18:28:39 2016 +0200

    Move the main template files for defalut services to new syntax generation

    When generating these templates, we should
    create them with the "-role" appended as they will
    be generated from a role.role.j2.yaml file.

    i.e. role.role.j2.yaml will generate <service>-role.yaml
         config.role.j2.yaml will generate <service>-config.yaml

    Partial-Bug: #1626976
    Change-Id: I614dc462fd7fc088b67634d489d8e7b68e7d4ab1
    (cherry picked from commit 883addf267933395c580e0eab3efc379401c946c)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/382737

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/382738

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/381593
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=1621ccaf38ffeb0960051ee511d8cc3965997c0b
Submitter: Jenkins
Branch: master

commit 1621ccaf38ffeb0960051ee511d8cc3965997c0b
Author: Carlos Camacho <email address hidden>
Date: Tue Oct 4 12:03:41 2016 +0200

    Add support to create role main template file based in role.role.j2.yaml

    Wiring up the creation of a main role templates file
    based in role.role.j2.yaml

    Also added the validation for the exclusion list.

    Partial-Bug: #1626976

    Change-Id: I6d7247bbb8702eb0ab9bdf133b5ab1c6e8349d98

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/382738
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=fe308c8232a2004267a138d70f4a3fe3809ea745
Submitter: Jenkins
Branch: stable/newton

commit fe308c8232a2004267a138d70f4a3fe3809ea745
Author: Carlos Camacho <email address hidden>
Date: Tue Oct 4 12:03:41 2016 +0200

    Add support to create role main template file based in role.role.j2.yaml

    Wiring up the creation of a main role templates file
    based in role.role.j2.yaml

    Also added the validation for the exclusion list.

    Partial-Bug: #1626976

    Change-Id: I6d7247bbb8702eb0ab9bdf133b5ab1c6e8349d98

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/381587
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8c52f96753771edcc594e63a5a5fa00ecdb05040
Submitter: Jenkins
Branch: master

commit 8c52f96753771edcc594e63a5a5fa00ecdb05040
Author: Carlos Camacho <email address hidden>
Date: Tue Oct 4 11:50:33 2016 +0200

    Add generic template for custom roles.

    This submission creates a generic template
    file to deploy custom roles.

    Also adds a file to specify an exclusion role
    list in order to avoid not to generate the
    template for those roles.

    Partial-Bug: #1626976

    Depends-On: I6d7247bbb8702eb0ab9bdf133b5ab1c6e8349d98

    Change-Id: I3e11c089023b793a5063d9e1714527a3fe2b7458

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/382911

[Carlos Camacho (ccamacho)]

Hopefully, all work will be landed for Newton GA, this is a local test getting all submissions merged.

https://paste.fedoraproject.org/444864/47576058/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/382737
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=936f4c8d1536ca6f8a9615c32fe6b21b3b14c8db
Submitter: Jenkins
Branch: stable/newton

commit 936f4c8d1536ca6f8a9615c32fe6b21b3b14c8db
Author: Carlos Camacho <email address hidden>
Date: Tue Oct 4 11:50:33 2016 +0200

    Add generic template for custom roles.

    This submission creates a generic template
    file to deploy custom roles.

    Also adds a file to specify an exclusion role
    list in order to avoid not to generate the
    template for those roles.

    Partial-Bug: #1626976

    Depends-On: I6d7247bbb8702eb0ab9bdf133b5ab1c6e8349d98

    Change-Id: I3e11c089023b793a5063d9e1714527a3fe2b7458

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/382973

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/382911
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=aab8d64d71a1a4e97ea932a6df8a943037f40515
Submitter: Jenkins
Branch: master

commit aab8d64d71a1a4e97ea932a6df8a943037f40515
Author: Carlos Camacho <email address hidden>
Date: Thu Oct 6 10:51:31 2016 +0000

    Add the J2 exclude file exception (When file not found)

    When the file j2_excludes.yaml is not found in tht we need to default
    this exclude list to [].

    Change-Id: I23b84d4ee8c4d0bff4b4e38dc4731924a6272ac5
    Partial-Bug: #1626976

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/382217
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e32e2110b8058b265dfa8246532b45896b461427
Submitter: Jenkins
Branch: master

commit e32e2110b8058b265dfa8246532b45896b461427
Author: Carlos Camacho <email address hidden>
Date: Wed Oct 5 11:29:59 2016 +0200

    Add Select per-network hostnames for service_node_names to role.role.j2.yaml

    This will wire up the per-network hostnames in the generic role.

    Needs to land after https://review.openstack.org/#/c/378764

    Partial-Bug: #1626976

    Change-Id: I595f35cce03d9f416a1768aa5c349a1bb20b0e19

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/383521

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/382973
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=d61747032a55e07c181c78aa712f2a8559718627
Submitter: Jenkins
Branch: stable/newton

commit d61747032a55e07c181c78aa712f2a8559718627
Author: Carlos Camacho <email address hidden>
Date: Thu Oct 6 10:51:31 2016 +0000

    Add the J2 exclude file exception (When file not found)

    When the file j2_excludes.yaml is not found in tht we need to default
    this exclude list to [].

    Change-Id: I23b84d4ee8c4d0bff4b4e38dc4731924a6272ac5
    Partial-Bug: #1626976
    (cherry picked from commit aab8d64d71a1a4e97ea932a6df8a943037f40515)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/383521
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=db82aedc76079473ca74c47af431f4cabff24322
Submitter: Jenkins
Branch: stable/newton

commit db82aedc76079473ca74c47af431f4cabff24322
Author: Carlos Camacho <email address hidden>
Date: Wed Oct 5 11:29:59 2016 +0200

    Add Select per-network hostnames for service_node_names to role.role.j2.yaml

    This will wire up the per-network hostnames in the generic role.

    Needs to land after https://review.openstack.org/#/c/378764

    Partial-Bug: #1626976

    Change-Id: I595f35cce03d9f416a1768aa5c349a1bb20b0e19
    (cherry picked from commit e32e2110b8058b265dfa8246532b45896b461427)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/384321
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=ad2edc32e343fac048e795e1cd5fc5f094084095
Submitter: Jenkins
Branch: master

commit ad2edc32e343fac048e795e1cd5fc5f094084095
Author: Carlos Camacho <email address hidden>
Date: Mon Oct 10 09:08:39 2016 +0200

    Default the J2 excludes files to safe values.

    This submission add 3 validations when the J2
    excludes file is processed.

    1- j2_excludes.yaml does not exist.
    2- j2_excludes.yaml its empty.
    3- j2_excludes.yaml only contains 'name:'
       without any template.

    The last 3 conditions will be defaulted to {"name": []}

    Previously the inclusion of j2_excludes.yaml was mandatory.

    Partial-Bug: #1626976

    Change-Id: Iafb5f8defee4139c88f2c3291cb43b3b8a5edf30

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/385471

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/385471
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=a006d97808b9aca1098dbb239562887759fdc764
Submitter: Jenkins
Branch: stable/newton

commit a006d97808b9aca1098dbb239562887759fdc764
Author: Carlos Camacho <email address hidden>
Date: Mon Oct 10 09:08:39 2016 +0200

    Default the J2 excludes files to safe values.

    This submission add 3 validations when the J2
    excludes file is processed.

    1- j2_excludes.yaml does not exist.
    2- j2_excludes.yaml its empty.
    3- j2_excludes.yaml only contains 'name:'
       without any template.

    The last 3 conditions will be defaulted to {"name": []}

    Previously the inclusion of j2_excludes.yaml was mandatory.

    Partial-Bug: #1626976

    Change-Id: Iafb5f8defee4139c88f2c3291cb43b3b8a5edf30
    (cherry picked from commit ad2edc32e343fac048e795e1cd5fc5f094084095)