Adding additional Python wheels to the repo

[git-harry]

The repo-build playbook uses the py_pkg lookup plugin to gather a list of Python packages that need to exist on the repo. The py_pkgs filter is used to search the playbooks directory for requirements files or YAML files in defaults or vars subdirectories. If, for example, a cinder backend requires an additional library the overrides file needs updating in /etc/openstack_deploy and a requirements file needs dropping in the playbooks directory. This feels like a clunky method for including additional packages and not very obvious, if there's another way let me know.

I think the lookup plugin should be modified so that at the very least it picks up variables that have been overridden in /etc/openstack_deploy.

The question of adding in additional Python packages relates to using alternative cinder backends. Has the project got any thoughts on using including all the applicable libraries by default?

[Kevin Carter (kevin-carter)]

All of the libs that are found within the global-requirements.txt within https://github.com/openstack/requirements are built within the repo server environment. Within the build process the lookup plugin is used to see what packages are being used throughout the stack, that data is then compared to the global requirements and a difference is generated. All of the requirements files are built using one of two pip build tasks that leverage a constraints file to provide boundaries for the build process.

The output of a build can be found here:
http://rpc-repo.rackspace.com/os-releases/12.0.0/requirements_local_filtered.txt
http://rpc-repo.rackspace.com/os-releases/12.0.0/requirements_global_requirements.txt
http://rpc-repo.rackspace.com/os-releases/12.0.0/requirements_constraints.txt

Within that build process an absolute requirements file is generated which will allow a deployer to create all of the base requirements as needed at anytime:
http://rpc-repo.rackspace.com/os-releases/12.0.0/requirements_absolute_requirements.txt

All of that leads me to the question of, what alternative lib, thats not part of upstream, are we not already building?

As an aside, you could add additional deps to the global-requirement-pins.txt which would load additional packages / pins into the build / constraint process.
* https://github.com/openstack/openstack-ansible/blob/master/global-requirement-pins.txt

That said it may be a good idea to scan /etc/openstack_deploy (as you've mentioned) for additional packages / pins. The plugin already loads in using a list and should be a trivial matter to extend however its never been tested with more than one source. If that's something that you wouldn't mind taking on I'd be for supporting that as an extension for further flexibility.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/241483
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=2f35e36e54efb7b0e767d0888782f86696ef84b9
Submitter: Jenkins
Branch: master

commit 2f35e36e54efb7b0e767d0888782f86696ef84b9
Author: Kevin Carter <email address hidden>
Date: Tue Nov 3 23:39:21 2015 -0600

    Updated the py_pkgs lookup plugin for multi source

    The py_pkgs lookup plugin will now handle multiple sources. This change
    was needed to enable the system to allow for overrides of repo_package
    resources and general python packaging as found throughout the stack.
    The module has precedence as it loads / searches for packages and he item
    loaded last is the one that has the most precedence.

    The repo-build play has been updated to now search for compatible packages
    in the main playbook directory, the root ansible role directory and the
    user_variables file.

    Documentation has been added regarding this capability and how to override
    and set items to meet the needs of the deployment.

    Closes-Bug: #1510575
    Implements: blueprint independent-role-repositories
    Change-Id: Ib7cda14db18c0ca58bb5ac495d1c201812cf0f48
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/252956
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=3555a5d62e4d525ce35afe9930339d1bd99aabac
Submitter: Jenkins
Branch: liberty

commit 3555a5d62e4d525ce35afe9930339d1bd99aabac
Author: Kevin Carter <email address hidden>
Date: Tue Nov 3 23:39:21 2015 -0600

    Updated the py_pkgs lookup plugin for multi source

    The py_pkgs lookup plugin will now handle multiple sources. This change
    was needed to enable the system to allow for overrides of repo_package
    resources and general python packaging as found throughout the stack.
    The module has precedence as it loads / searches for packages and he item
    loaded last is the one that has the most precedence.

    The repo-build play has been updated to now search for compatible packages
    in the main playbook directory, the root ansible role directory and the
    user_variables file.

    Documentation has been added regarding this capability and how to override
    and set items to meet the needs of the deployment.

    Closes-Bug: #1510575
    Implements: blueprint independent-role-repositories
    Change-Id: Ib7cda14db18c0ca58bb5ac495d1c201812cf0f48
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit 2f35e36e54efb7b0e767d0888782f86696ef84b9)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 12.0.8 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 12.0.9 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 13.0.0 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 13.0.0 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 12.0.11 release.