Include default config files in python wheel

Bug #1718356 reported by Jesse Pretorius on 2017-09-20
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Barbican
Fix Released
Undecided
Jesse Pretorius
Cinder
Undecided
Jesse Pretorius
Cyborg
Fix Released
Undecided
zhangyangyang
Designate
Undecided
Nguyen Van Trung
Fuxi
Undecided
Unassigned
Glance
Undecided
Jesse Pretorius
Ironic
Fix Released
Wishlist
Jesse Pretorius
Karbor
Undecided
Spencer Yu
Magnum
In Progress
Undecided
Jesse Pretorius
OpenStack Compute (nova)
Undecided
Jesse Pretorius
OpenStack DBaaS (Trove)
Undecided
Jesse Pretorius
OpenStack Heat
In Progress
Undecided
Jesse Pretorius
OpenStack Identity (keystone)
Undecided
Jesse Pretorius
Sahara
Fix Released
Undecided
Jesse Pretorius
Zun
Undecided
Rajat Sharma
kuryr-libnetwork
Undecided
Unassigned
neutron
Undecided
Jesse Pretorius
octavia
Invalid
Undecided
Jesse Pretorius
openstack-ansible
Medium
Jesse Pretorius

Bug Description

The projects which deploy OpenStack from source or using python wheels currently have to either carry templates for api-paste, policy and rootwrap files or need to source them from git during deployment. This results in some rather complex mechanisms which could be radically simplified by simply ensuring that all the same files are included in the built wheel.

A precedence for this has already been set in neutron [1], glance [2] and designate [3] through the use of the data_files option in the files section of setup.cfg.

[1] https://github.com/openstack/neutron/blob/d3c393ff6b5fbd0bdaabc8ba678d755ebfba08f7/setup.cfg#L24-L39
[2] https://github.com/openstack/glance/blob/02cd5cba70a8465a951cb813a573d390887174b7/setup.cfg#L20-L21
[3] https://github.com/openstack/designate/blob/25eb143db04554d65efe2e5d60ad3afa6b51d73a/setup.cfg#L30-L37

This bug will be used for a cross-project implementation of patches to normalise the implementation across the OpenStack projects. Hopefully the result will be a consistent implementation across all the major projects.

A mailing list thread corresponding to this standard setting was begun:
http://lists.openstack.org/pipermail/openstack-dev/2017-September/122794.html

Changed in neutron:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Changed in glance:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Changed in openstack-ansible:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Changed in nova:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Matt Riedemann (mriedem) wrote :

Why are neutron and glance part of this bug if they already fixed this? Why aren't cinder/keystone affected?

@Matt I'll be patching both neutron and glance to include more files or to optimise the implementation. I will be adding more projects as I go through them - I ended up getting pulled into something else yesterday before completing this.

Changed in cinder:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Changed in keystone:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Changed in barbican:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Changed in designate:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Changed in heat:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Changed in ironic:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Changed in octavia:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Changed in magnum:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Changed in trove:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Changed in sahara:
assignee: nobody → Jesse Pretorius (jesse-pretorius)
Changed in neutron:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/506145

Changed in glance:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/506149

Changed in keystone:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/506155

Changed in cinder:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/506188

Changed in nova:
status: New → In Progress
Changed in barbican:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/506191

Changed in designate:
status: New → In Progress
description: updated

Fix proposed to branch: master
Review: https://review.openstack.org/506198

Changed in ironic:
status: New → In Progress
Changed in magnum:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/506200

Changed in trove:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/506203

Changed in heat:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/506206

Changed in sahara:
status: New → In Progress
Michael Johnson (johnsom) wrote :

Can you be more specific as to what is missing from Octavia? We already include the example config in our data_files:
https://github.com/openstack/octavia/blob/master/setup.cfg#L25-L38

Changed in octavia:
status: New → Incomplete

@Michael I see that there are no sample api-paste.ini files and it seems that the default policy is in-code? I guess that Octavia does not implement any paste middleware? If so, yes - nothing needs to be done here. Apologies for the confusion.

Michael Johnson (johnsom) wrote :

Correct, our policy is in code and we don't use paste. Marking invalid.

Changed in octavia:
status: Incomplete → Invalid
Rajat Sharma (tajar29) on 2017-09-22
Changed in zun:
assignee: nobody → Rajat Sharma (tajar29)
Changed in kuryr-libnetwork:
assignee: nobody → Rajat Sharma (tajar29)
Changed in fuxi:
assignee: nobody → Rajat Sharma (tajar29)

Fix proposed to branch: master
Review: https://review.openstack.org/506559

Changed in zun:
status: New → In Progress

Reviewed: https://review.openstack.org/506210
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=873c4fed836f8f4c16469ad16478b7bbbfb0daa6
Submitter: Jenkins
Branch: master

commit 873c4fed836f8f4c16469ad16478b7bbbfb0daa6
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 15:23:06 2017 +0100

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron, designate and glance as has
    been mentioned in the related bug report.

    Change-Id: Iae987bdae55b75143f192f9a642cf1ff564c62b6
    Closes-Bug: #1718356

Changed in sahara:
status: In Progress → Fix Released
Changed in designate:
assignee: Jesse Pretorius (jesse-pretorius) → Nguyen Van Trung (trungnv)
Dmitry Tantsur (divius) on 2017-10-02
Changed in ironic:
importance: Undecided → Wishlist
Changed in cyborg:
assignee: nobody → zhangyangyang (zhangyangyang)
status: New → In Progress
Changed in openstack-ansible:
status: New → Confirmed
description: updated

Reviewed: https://review.openstack.org/506198
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=110259750e15323a2cbc3c8fb29b9d51cc68cf7c
Submitter: Jenkins
Branch: master

commit 110259750e15323a2cbc3c8fb29b9d51cc68cf7c
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 15:11:46 2017 +0100

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron, designate and glance as has
    been mentioned in the related bug report.

    Change-Id: Ibeed85e148298e1aea7542b950bc24a7f0513de8
    Closes-Bug: #1718356

Changed in ironic:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/506149
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=0502d749a6ac43167cfcbdd73bd25c542035dbbb
Submitter: Jenkins
Branch: master

commit 0502d749a6ac43167cfcbdd73bd25c542035dbbb
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 13:52:43 2017 +0100

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron and the glance metadefs as has
    been mentioned in the related bug report.

    Change-Id: I505a2cdff3ec699dcb4a37106765ef889872d367
    Closes-Bug: #1718356

Changed in keystone:
status: In Progress → Fix Released

Change abandoned by zhangyangyang (<email address hidden>) on branch: master
Review: https://review.openstack.org/509198

Reviewed: https://review.openstack.org/506192
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=e486a50f77ad09e04a6cb20bfc40c798d9161535
Submitter: Zuul
Branch: master

commit e486a50f77ad09e04a6cb20bfc40c798d9161535
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 15:04:03 2017 +0100

    Include all rootwrap filters when building wheels

    The current method of specifying each rootwrap filter
    in the file list is prone to errors when adding or
    removing filters. Instead of relying on a manually
    maintained list this patch just includes all the files
    of the correct naming convention from the applicable
    folder. This is simpler and easier to maintain.

    Change-Id: I116efd3ff1799965bb46da785b2ad96c7f5b97c5
    Closes-Bug: #1718356

Changed in designate:
status: In Progress → Fix Released

This issue was fixed in the openstack/cinder 12.0.0.0b1 development milestone.

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

This issue was fixed in the openstack/keystone 13.0.0.0b1 development milestone.

This issue was fixed in the openstack/designate 6.0.0.0b1 development milestone.

This issue was fixed in the openstack/heat 10.0.0.0b1 development milestone.

This issue was fixed in the openstack/sahara 8.0.0.0b1 development milestone.

This issue was fixed in the openstack/manila 6.0.0.0b1 development milestone.

This issue was fixed in the openstack/ironic 9.2.0 release.

Reviewed: https://review.openstack.org/506145
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=b0b102b8dc5ae0327f0b772c6a9aafaa5c666f87
Submitter: Zuul
Branch: master

commit b0b102b8dc5ae0327f0b772c6a9aafaa5c666f87
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 13:37:50 2017 +0100

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron and the glance metadefs as has
    been mentioned in the related bug report.

    Change-Id: Ie0eb7f16bc975de0db80a2144fd8db9e09f6317f
    Closes-Bug: #1718356

Changed in glance:
status: In Progress → Fix Released

This issue was fixed in the openstack/glance 16.0.0.0b2 development milestone.

This issue was fixed in the openstack/congress 7.0.0.0b2 development milestone.

Changed in cinder:
status: In Progress → Fix Released

This issue was fixed in the openstack/zun 1.0.0 release.

Spencer Yu (yushb) on 2018-02-06
Changed in karbor:
assignee: nobody → Spencer Yu (yushb)

Fix proposed to branch: master
Review: https://review.openstack.org/541106

Changed in karbor:
status: New → In Progress

This issue was fixed in the openstack/magnum 6.0.0 release.

Reviewed: https://review.openstack.org/506203
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=39ae2ee3a9084e0b6945f0fbec3dd735dc0efa7d
Submitter: Zuul
Branch: master

commit 39ae2ee3a9084e0b6945f0fbec3dd735dc0efa7d
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 15:16:08 2017 +0100

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron, designate and glance as has
    been mentioned in the related bug report.

    Change-Id: I3bb03644674f016018a178a76cca9d12afe11c43
    Closes-Bug: #1718356

Changed in trove:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/506142
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=45f1404c68553fd01f386522dd16841526c68dbf
Submitter: Zuul
Branch: master

commit 45f1404c68553fd01f386522dd16841526c68dbf
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 13:28:09 2017 +0100

    Include all rootwrap filters when building wheels

    The current method of specifying each rootwrap filter
    in the file list is prone to errors when adding or
    removing filters. Instead of relying on a manually
    maintained list this patch just includes all the files
    of the correct naming convention from the applicable
    folder. This is simpler and easier to maintain.

    Closes-Bug: #1718356
    Change-Id: I7f8c55f63d1c5a85a6a92062e918426f7d2d3c35

Changed in neutron:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/506191
Committed: https://git.openstack.org/cgit/openstack/barbican/commit/?id=5d162ebdda9071b9a691f3130228b51f4a808836
Submitter: Zuul
Branch: master

commit 5d162ebdda9071b9a691f3130228b51f4a808836
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 15:01:08 2017 +0100

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron and the glance metadefs as has
    been mentioned in the related bug report.

    Change-Id: Iacb17585a3235e95faa109ff11f1b039429ff770
    Closes-Bug: #1718356

Changed in barbican:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/553463
Committed: https://git.openstack.org/cgit/openstack/cyborg/commit/?id=ac6b70dc6a9c440250b739c9fed9a1b74d7642be
Submitter: Zuul
Branch: master

commit ac6b70dc6a9c440250b739c9fed9a1b74d7642be
Author: Nguyen Van Trung <email address hidden>
Date: Thu Mar 15 23:20:29 2018 +0700

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron and the glance metadefs as has
    been mentioned in the related bug report.

    Change-Id: I466f235fec7be024f654739a31365724eaf24097
    Closes-Bug: #1718356

Changed in cyborg:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/558240
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=8367e442304a16687be602d4a23762d3408b9010
Submitter: Zuul
Branch: master

commit 8367e442304a16687be602d4a23762d3408b9010
Author: Jesse Pretorius <email address hidden>
Date: Mon Apr 2 17:51:18 2018 +0100

    Use the venv contents instead of a git source for templates

    When deploying the base templates for api-paste, policy files
    and other files which are included in the service git source,
    we now use the venv files instead of requiring access to a
    git source and a complex set of lookups and variable
    implementations.

    This is simpler and more cross-series, and works from Queens
    due to the related bug's patches.

    Change-Id: I6a4e2514e66b15b2ae227e62b6dc9ae1a50a4fbd
    Related-Bug: #1718356

This issue was fixed in the openstack/senlin 6.0.0.0b1 development milestone.

This issue was fixed in the openstack/neutron 13.0.0.0b1 development milestone.

This issue was fixed in the openstack/trove 10.0.0.0b1 development milestone.

This issue was fixed in the openstack/murano 6.0.0.0b1 development milestone.

This issue was fixed in the openstack/cyborg 1.0.0.0b1 development milestone.

This issue was fixed in the openstack/barbican 7.0.0.0b1 development milestone.

Reviewed: https://review.openstack.org/554819
Committed: https://git.openstack.org/cgit/openstack/karbor/commit/?id=c20e61d04221695bfd316417254649733e7ae692
Submitter: Zuul
Branch: master

commit c20e61d04221695bfd316417254649733e7ae692
Author: Nguyen Van Trung <email address hidden>
Date: Wed Mar 21 14:40:21 2018 +0700

    Add default configuration files to data_files

    In order to make it simpler to use the default
    configuration files when deploying services
    from source, the files are added to pbr's
    data_files section so that the files are
    included in the built wheels and therefore
    deployed with the code. Packaging and deployment
    tools can then more easily use the default files
    if they wish to.

    This pattern is already established with similar
    files for neutron and the glance metadefs as has
    been mentioned in the related bug report.

    Change-Id: I81b5c00ace7b6d5fab4f982f403149a87b3afbd0
    Closes-Bug: #1718356

Changed in karbor:
status: In Progress → Fix Released
tags: added: neutron-proactive-backport-potential
tags: added: neutron-easy-proactive-backport-potential
Rajat Sharma (tajar29) on 2018-05-31
Changed in fuxi:
assignee: Rajat Sharma (tajar29) → nobody
tags: removed: neutron-easy-proactive-backport-potential neutron-proactive-backport-potential

Reviewed: https://review.openstack.org/579944
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_cinder/commit/?id=83c07a4cb9960ce605a5c3fb5469f2870641e7d0
Submitter: Zuul
Branch: master

commit 83c07a4cb9960ce605a5c3fb5469f2870641e7d0
Author: Mohammed Naser <email address hidden>
Date: Tue Jul 3 14:32:39 2018 -0400

    Use rootwrap configuration files from the virtualenv

    We currently vendor the rootwrap configuration file however we
    really should be using the one that ships from with the Python
    package.

    This patch changes the behaviour so that we ensure that all the
    upstream bundled rootwrap files will always be in sync
    (addition and deletion) with the content on the disk, so
    we do not have to vendor it anymore.

    Related-Bug: #1718356
    Change-Id: Id08eca3a0cbde541a8d669bf64d5077a9b2a9085

hongbin (hongbin034) on 2018-07-05
Changed in zun:
status: In Progress → Fix Released
Rajat Sharma (tajar29) on 2018-07-06
Changed in kuryr-libnetwork:
assignee: Rajat Sharma (tajar29) → nobody

Reviewed: https://review.openstack.org/582123
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_cinder/commit/?id=5dd4abe61adcdcc3950afd3950619c4c98b8b809
Submitter: Zuul
Branch: master

commit 5dd4abe61adcdcc3950afd3950619c4c98b8b809
Author: Jesse Pretorius <email address hidden>
Date: Thu Jul 12 10:21:44 2018 +0100

    Use rsync instead of synchronize module for remote copy

    The synchronize module with delegation introduces complexity
    with permissions which we really, really do not need to have
    to deal with for a remote sync. So instead, we just use rsync
    because it's simpler to use and understand.

    Change-Id: I6fae5af8d59ea9499cc90a35f1d2f5176fbf9d04
    Related-Bug: #1718356

This issue was fixed in the openstack/karbor 1.1.0 release.

Reviewed: https://review.openstack.org/619485
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3466832b9972002e1c9eccde9ab2392dd7363e3a
Submitter: Zuul
Branch: stable/queens

commit 3466832b9972002e1c9eccde9ab2392dd7363e3a
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 13:28:09 2017 +0100

    Include all rootwrap filters when building wheels

    The current method of specifying each rootwrap filter
    in the file list is prone to errors when adding or
    removing filters. Instead of relying on a manually
    maintained list this patch just includes all the files
    of the correct naming convention from the applicable
    folder. This is simpler and easier to maintain.

    Closes-Bug: #1718356
    Change-Id: I7f8c55f63d1c5a85a6a92062e918426f7d2d3c35
    (cherry picked from commit 45f1404c68553fd01f386522dd16841526c68dbf)

tags: added: in-stable-queens

Reviewed: https://review.openstack.org/619571
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=888cbc2970e3c06956f1fae403c07743371cae7c
Submitter: Zuul
Branch: stable/pike

commit 888cbc2970e3c06956f1fae403c07743371cae7c
Author: Jesse Pretorius <email address hidden>
Date: Thu Sep 21 13:28:09 2017 +0100

    Include all rootwrap filters when building wheels

    The current method of specifying each rootwrap filter
    in the file list is prone to errors when adding or
    removing filters. Instead of relying on a manually
    maintained list this patch just includes all the files
    of the correct naming convention from the applicable
    folder. This is simpler and easier to maintain.

    Closes-Bug: #1718356
    Change-Id: I7f8c55f63d1c5a85a6a92062e918426f7d2d3c35
    (cherry picked from commit 45f1404c68553fd01f386522dd16841526c68dbf)

tags: added: in-stable-pike
Changed in openstack-ansible:
status: Confirmed → In Progress
importance: Undecided → Medium

This issue was fixed in the openstack/neutron 11.0.7 release.

This issue was fixed in the openstack/neutron 12.0.6 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers