Include default config files in python wheel

[Jesse Pretorius]

The projects which deploy OpenStack from source or using python wheels currently have to either carry templates for api-paste, policy and rootwrap files or need to source them from git during deployment. This results in some rather complex mechanisms which could be radically simplified by simply ensuring that all the same files are included in the built wheel.

A precedence for this has already been set in neutron [1], glance [2] and designate [3] through the use of the data_files option in the files section of setup.cfg.

[1] https://github.com/openstack/neutron/blob/d3c393ff6b5fbd0bdaabc8ba678d755ebfba08f7/setup.cfg#L24-L39
[2] https://github.com/openstack/glance/blob/02cd5cba70a8465a951cb813a573d390887174b7/setup.cfg#L20-L21
[3] https://github.com/openstack/designate/blob/25eb143db04554d65efe2e5d60ad3afa6b51d73a/setup.cfg#L30-L37

This bug will be used for a cross-project implementation of patches to normalise the implementation across the OpenStack projects. Hopefully the result will be a consistent implementation across all the major projects.

A mailing list thread corresponding to this standard setting was begun:
http://lists.openstack.org/pipermail/openstack-dev/2017-September/122794.html

[Matt Riedemann (mriedem)]

Why are neutron and glance part of this bug if they already fixed this? Why aren't cinder/keystone affected?

[Jesse Pretorius (jesse-pretorius)]

@Matt I'll be patching both neutron and glance to include more files or to optimise the implementation. I will be adding more projects as I go through them - I ended up getting pulled into something else yesterday before completing this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/506145

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/506149

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/506155

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/506188

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/506191

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/506192

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/506198

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/506200

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/506203

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/506206

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/506210

[Michael Johnson (johnsom)]

Can you be more specific as to what is missing from Octavia? We already include the example config in our data_files:
https://github.com/openstack/octavia/blob/master/setup.cfg#L25-L38

[Jesse Pretorius (jesse-pretorius)]

@Michael I see that there are no sample api-paste.ini files and it seems that the default policy is in-code? I guess that Octavia does not implement any paste middleware? If so, yes - nothing needs to be done here. Apologies for the confusion.

[Michael Johnson (johnsom)]

Correct, our policy is in code and we don't use paste. Marking invalid.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/506559

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/506210
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=873c4fed836f8f4c16469ad16478b7bbbfb0daa6
Submitter: Jenkins
Branch: master

commit 873c4fed836f8f4c16469ad16478b7bbbfb0daa6
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 15:23:06 2017 +0100

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron, designate and glance as has
    been mentioned in the related bug report.

    Change-Id: Iae987bdae55b75143f192f9a642cf1ff564c62b6
    Closes-Bug: #1718356

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/509198

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/506198
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=110259750e15323a2cbc3c8fb29b9d51cc68cf7c
Submitter: Jenkins
Branch: master

commit 110259750e15323a2cbc3c8fb29b9d51cc68cf7c
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 15:11:46 2017 +0100

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron, designate and glance as has
    been mentioned in the related bug report.

    Change-Id: Ibeed85e148298e1aea7542b950bc24a7f0513de8
    Closes-Bug: #1718356

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/506149
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=0502d749a6ac43167cfcbdd73bd25c542035dbbb
Submitter: Jenkins
Branch: master

commit 0502d749a6ac43167cfcbdd73bd25c542035dbbb
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 13:52:43 2017 +0100

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron and the glance metadefs as has
    been mentioned in the related bug report.

    Change-Id: I505a2cdff3ec699dcb4a37106765ef889872d367
    Closes-Bug: #1718356

[OpenStack Infra (hudson-openstack)]

Change abandoned by zhangyangyang (<email address hidden>) on branch: master
Review: https://review.openstack.org/509198

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/506192
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=e486a50f77ad09e04a6cb20bfc40c798d9161535
Submitter: Zuul
Branch: master

commit e486a50f77ad09e04a6cb20bfc40c798d9161535
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 15:04:03 2017 +0100

    Include all rootwrap filters when building wheels

    The current method of specifying each rootwrap filter
    in the file list is prone to errors when adding or
    removing filters. Instead of relying on a manually
    maintained list this patch just includes all the files
    of the correct naming convention from the applicable
    folder. This is simpler and easier to maintain.

    Change-Id: I116efd3ff1799965bb46da785b2ad96c7f5b97c5
    Closes-Bug: #1718356

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 13.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/designate 6.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 10.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/sahara 8.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 6.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic 9.2.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/506145
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=b0b102b8dc5ae0327f0b772c6a9aafaa5c666f87
Submitter: Zuul
Branch: master

commit b0b102b8dc5ae0327f0b772c6a9aafaa5c666f87
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 13:37:50 2017 +0100

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron and the glance metadefs as has
    been mentioned in the related bug report.

    Change-Id: Ie0eb7f16bc975de0db80a2144fd8db9e09f6317f
    Closes-Bug: #1718356

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 16.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/congress 7.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/zun 1.0.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/541106

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/magnum 6.0.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/506203
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=39ae2ee3a9084e0b6945f0fbec3dd735dc0efa7d
Submitter: Zuul
Branch: master

commit 39ae2ee3a9084e0b6945f0fbec3dd735dc0efa7d
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 15:16:08 2017 +0100

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron, designate and glance as has
    been mentioned in the related bug report.

    Change-Id: I3bb03644674f016018a178a76cca9d12afe11c43
    Closes-Bug: #1718356

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/506142
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=45f1404c68553fd01f386522dd16841526c68dbf
Submitter: Zuul
Branch: master

commit 45f1404c68553fd01f386522dd16841526c68dbf
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 13:28:09 2017 +0100

    Include all rootwrap filters when building wheels

    The current method of specifying each rootwrap filter
    in the file list is prone to errors when adding or
    removing filters. Instead of relying on a manually
    maintained list this patch just includes all the files
    of the correct naming convention from the applicable
    folder. This is simpler and easier to maintain.

    Closes-Bug: #1718356
    Change-Id: I7f8c55f63d1c5a85a6a92062e918426f7d2d3c35

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/506191
Committed: https://git.openstack.org/cgit/openstack/barbican/commit/?id=5d162ebdda9071b9a691f3130228b51f4a808836
Submitter: Zuul
Branch: master

commit 5d162ebdda9071b9a691f3130228b51f4a808836
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 15:01:08 2017 +0100

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron and the glance metadefs as has
    been mentioned in the related bug report.

    Change-Id: Iacb17585a3235e95faa109ff11f1b039429ff770
    Closes-Bug: #1718356

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/553463
Committed: https://git.openstack.org/cgit/openstack/cyborg/commit/?id=ac6b70dc6a9c440250b739c9fed9a1b74d7642be
Submitter: Zuul
Branch: master

commit ac6b70dc6a9c440250b739c9fed9a1b74d7642be
Author: Nguyen Van Trung <email address hidden>
Date: Thu Mar 15 23:20:29 2018 +0700

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron and the glance metadefs as has
    been mentioned in the related bug report.

    Change-Id: I466f235fec7be024f654739a31365724eaf24097
    Closes-Bug: #1718356

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/558240

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/558240
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=8367e442304a16687be602d4a23762d3408b9010
Submitter: Zuul
Branch: master

commit 8367e442304a16687be602d4a23762d3408b9010
Author: Jesse Pretorius <email address hidden>
Date: Mon Apr 2 17:51:18 2018 +0100

    Use the venv contents instead of a git source for templates

    When deploying the base templates for api-paste, policy files
    and other files which are included in the service git source,
    we now use the venv files instead of requiring access to a
    git source and a complex set of lookups and variable
    implementations.

    This is simpler and more cross-series, and works from Queens
    due to the related bug's patches.

    Change-Id: I6a4e2514e66b15b2ae227e62b6dc9ae1a50a4fbd
    Related-Bug: #1718356

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/senlin 6.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/trove 10.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/murano 6.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cyborg 1.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/barbican 7.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/554819
Committed: https://git.openstack.org/cgit/openstack/karbor/commit/?id=c20e61d04221695bfd316417254649733e7ae692
Submitter: Zuul
Branch: master

commit c20e61d04221695bfd316417254649733e7ae692
Author: Nguyen Van Trung <email address hidden>
Date: Wed Mar 21 14:40:21 2018 +0700

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron and the glance metadefs as has
    been mentioned in the related bug report.

    Change-Id: I81b5c00ace7b6d5fab4f982f403149a87b3afbd0
    Closes-Bug: #1718356

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/579944
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_cinder/commit/?id=83c07a4cb9960ce605a5c3fb5469f2870641e7d0
Submitter: Zuul
Branch: master

commit 83c07a4cb9960ce605a5c3fb5469f2870641e7d0
Author: Mohammed Naser <email address hidden>
Date: Tue Jul 3 14:32:39 2018 -0400

    Use rootwrap configuration files from the virtualenv

    We currently vendor the rootwrap configuration file however we
    really should be using the one that ships from with the Python
    package.

    This patch changes the behaviour so that we ensure that all the
    upstream bundled rootwrap files will always be in sync
    (addition and deletion) with the content on the disk, so
    we do not have to vendor it anymore.

    Related-Bug: #1718356
    Change-Id: Id08eca3a0cbde541a8d669bf64d5077a9b2a9085

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/582123

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/582123
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_cinder/commit/?id=5dd4abe61adcdcc3950afd3950619c4c98b8b809
Submitter: Zuul
Branch: master

commit 5dd4abe61adcdcc3950afd3950619c4c98b8b809
Author: Jesse Pretorius <email address hidden>
Date: Thu Jul 12 10:21:44 2018 +0100

    Use rsync instead of synchronize module for remote copy

    The synchronize module with delegation introduces complexity
    with permissions which we really, really do not need to have
    to deal with for a remote sync. So instead, we just use rsync
    because it's simpler to use and understand.

    Change-Id: I6fae5af8d59ea9499cc90a35f1d2f5176fbf9d04
    Related-Bug: #1718356

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/karbor 1.1.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/619485

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/619571

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/619485
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3466832b9972002e1c9eccde9ab2392dd7363e3a
Submitter: Zuul
Branch: stable/queens

commit 3466832b9972002e1c9eccde9ab2392dd7363e3a
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 13:28:09 2017 +0100

    Include all rootwrap filters when building wheels

    The current method of specifying each rootwrap filter
    in the file list is prone to errors when adding or
    removing filters. Instead of relying on a manually
    maintained list this patch just includes all the files
    of the correct naming convention from the applicable
    folder. This is simpler and easier to maintain.

    Closes-Bug: #1718356
    Change-Id: I7f8c55f63d1c5a85a6a92062e918426f7d2d3c35
    (cherry picked from commit 45f1404c68553fd01f386522dd16841526c68dbf)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/619571
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=888cbc2970e3c06956f1fae403c07743371cae7c
Submitter: Zuul
Branch: stable/pike

commit 888cbc2970e3c06956f1fae403c07743371cae7c
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 13:28:09 2017 +0100

    Include all rootwrap filters when building wheels

    The current method of specifying each rootwrap filter
    in the file list is prone to errors when adding or
    removing filters. Instead of relying on a manually
    maintained list this patch just includes all the files
    of the correct naming convention from the applicable
    folder. This is simpler and easier to maintain.

    Closes-Bug: #1718356
    Change-Id: I7f8c55f63d1c5a85a6a92062e918426f7d2d3c35
    (cherry picked from commit 45f1404c68553fd01f386522dd16841526c68dbf)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.6 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ghanshyam <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/karbor/+/541106
Reason: karbor is retired now http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018643.html