Upper constraints are not applied correctly in repo_build

[Jimmy McCrory]

The task setting applying upper constraints against release required python packages is not working correctly.
https://github.com/openstack/openstack-ansible-repo_build/blob/master/tasks/repo_set_facts.yml#L52

The 'pip_constraint_update' filter merges two lists of Python packags. The package versions in the second list provided to the filter overrides package versions from the first list.
https://github.com/openstack/openstack-ansible-plugins/blob/master/filter/osa-filters.py#L245

'_upper_constraints' and 'local_requirement_normalized' will need to be swapped so that upper-constraints's versions are overriding as expected.

Here's a gist showing the current upper-constraints from openstack/requirements for Mitaka, the local package requirements from the py_pkgs lookup plugin, and the resulting merged constraints for both the current state and when the lists are swapped.
https://gist.github.com/jimmymccrory/52a288a1b494475456466aaf1fd49897

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/346319
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-repo_build/commit/?id=373978f3c6c4bdf612c6bc53f8c38229dfbabd2b
Submitter: Jenkins
Branch: master

commit 373978f3c6c4bdf612c6bc53f8c38229dfbabd2b
Author: Jimmy McCrory <email address hidden>
Date: Fri Jul 22 23:44:10 2016 -0700

    Fix setting of upper_constraints fact

    The pip_constraint_update filter overrides pip versions from the second
    provided list to the first. Update the 'Set upper constraints' task so
    that upper-constraints correctly take priority over the local
    requirements packages versions.

    A test has been included to validate that the role built package
    constraints file is correctly applying the upper-constraints from the
    openstack requirements repo.

    Closes-Bug: 1605846
    Change-Id: I53864a34f3ba56a84ed658fa3952aba3c509e0b1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/346386

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: liberty
Review: https://review.openstack.org/346388

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/346386
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-repo_build/commit/?id=40fec4ad3537825055127c1d06a081857d82a6ff
Submitter: Jenkins
Branch: stable/mitaka

commit 40fec4ad3537825055127c1d06a081857d82a6ff
Author: Jimmy McCrory <email address hidden>
Date: Fri Jul 22 23:44:10 2016 -0700

    Fix setting of upper_constraints fact

    The pip_constraint_update filter overrides pip versions from the second
    provided list to the first. Update the 'Set upper constraints' task so
    that upper-constraints correctly take priority over the local
    requirements packages versions.

    A test has been included to validate that the role built package
    constraints file is correctly applying the upper-constraints from the
    openstack requirements repo.

    Closes-Bug: 1605846
    Change-Id: I53864a34f3ba56a84ed658fa3952aba3c509e0b1
    (cherry picked from commit 373978f3c6c4bdf612c6bc53f8c38229dfbabd2b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/346388
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=7055f795095d8e19c052371259ca77bf42c3095f
Submitter: Jenkins
Branch: liberty

commit 7055f795095d8e19c052371259ca77bf42c3095f
Author: Jimmy McCrory <email address hidden>
Date: Sat Jul 23 14:40:56 2016 +0100

    Fix setting of upper_constraints fact

    The pip_constraint_update filter overrides pip versions from the second
    provided list to the first. Update the 'Set upper constraints' task so
    that upper-constraints correctly take priority over the local
    requirements packages versions.

    A test has been included to validate that the role built package
    constraints file is correctly applying the upper-constraints from the
    openstack requirements repo.

    Notes:
    - This is a manual backport of https://review.openstack.org/346319
    - This patch also includes a version increment to 12.2.0 to express
      the significance of this fix.

    Closes-Bug: 1605846
    Change-Id: I53864a34f3ba56a84ed658fa3952aba3c509e0b1

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-repo_build 14.0.0.0b2 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-repo_build 13.3.0 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 12.2.0 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/350352

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/350352
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-repo_build/commit/?id=38d7faffa96aa2b03e718320f2dc281cc12029b8
Submitter: Jenkins
Branch: master

commit 38d7faffa96aa2b03e718320f2dc281cc12029b8
Author: Jesse Pretorius <email address hidden>
Date: Wed Aug 3 00:15:56 2016 +0100

    Fix repo constraints construction and package installation

    The current repo build process has the following issues:

    1. The constraints consruction doesn't properly handle two constraints
       which use the same name, but have different version specs. eg:
       ovs===2.5.0;python_version=='2.7'
       ovs===2.6.0.dev2;python_version=='3.4'
       This is a problem in both the constraints construction, and
       in the testing.

    2. The pip packages installed on the repo server in order to
       construct the repo do not respect the global pins or the upper
       constraint overrides.

    3. The constraints are selected based on the requirements specified.
       This makes the process unnecessarily complex and hard to test.

    4. The global pins are not applied to the constructed constraints,
       so the wrong packages get built and installed.

    5. The tests aren't thoroughly checking that the constraints are
       correct.

    This patch corrects all of these issues and hopefully makes the
    mechanism for constructing constraints more readily apparent, and
    therefore easier to maintain.

    Change-Id: If52d073d43081468e2faf2cd063c4b211c29994f
    Related-Bug: #1605846
    Closes-Bug: #1609056

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/350576

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: liberty
Review: https://review.openstack.org/350605

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/350612

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/350654
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-repo_build/commit/?id=ca864e298dc9e1486a09ed37b1b438c14df50248
Submitter: Jenkins
Branch: master

commit ca864e298dc9e1486a09ed37b1b438c14df50248
Author: Jesse Pretorius <email address hidden>
Date: Wed Aug 3 16:40:51 2016 +0100

    Correct global_pins depth

    In https://review.openstack.org/350352 the repo_build process
    implemented the test vars and changes to the constraints build
    which caters properly for global pins, but the data set used
    was incorrectly mapped from the py_pkgs lookup output.

    This patch fixes the depth, tests and template to use the
    correct depth.

    Related-Bug: #1605846
    Closes-Bug: #1609056
    Change-Id: Ic8959fd27b4012d569aef1b9ddd1d7be40ed49e0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/350612
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-plugins/commit/?id=450d972f27a34a5ac0024a24fbc5b377c70aac10
Submitter: Jenkins
Branch: stable/mitaka

commit 450d972f27a34a5ac0024a24fbc5b377c70aac10
Author: Jesse Pretorius <email address hidden>
Date: Wed Aug 3 15:24:31 2016 +0100

    Backport py_pkgs changes from Newton

    A minimal backport of the Newton changes to the py_pkgs lookup
    plugin in order to support https://review.openstack.org/350576
    does not appear to be possible.

    This is therefore a complete backport of all changes.

    Related-Bug: #1605846
    Related-Bug: #1609056
    Needed-By: If52d073d43081468e2faf2cd063c4b211c29994f
    Change-Id: Ic473eb9de26a2a3b92e234851b8f731c0255976c

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/352095

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/350576
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-repo_build/commit/?id=ae56b67cc07ca5a1e2dd03c74b5049b5942f41cf
Submitter: Jenkins
Branch: stable/mitaka

commit ae56b67cc07ca5a1e2dd03c74b5049b5942f41cf
Author: Jesse Pretorius <email address hidden>
Date: Wed Aug 3 00:15:56 2016 +0100

    Fix repo constraints construction and package installation

    The current repo build process has the following issues:

    1. The constraints consruction doesn't properly handle two constraints
       which use the same name, but have different version specs. eg:
       ovs===2.5.0;python_version=='2.7'
       ovs===2.6.0.dev2;python_version=='3.4'
       This is a problem in the constraints construction.

    2. The pip packages installed on the repo server in order to
       construct the repo do not respect the global pins or the upper
       constraint overrides.

    3. The constraints are selected based on the requirements specified.
       This makes the process unnecessarily complex.

    4. The global pins are not applied to the constructed constraints,
       so the wrong packages get built and installed.

    This patch corrects all of these issues and hopefully makes the
    mechanism for constructing constraints more readily apparent, and
    therefore easier to maintain.

    Conflicts:
    >------tasks/repo_pre_build.yml
    >------tests/test-repo-build.yml
    >------tests/test-vars.yml

    This backport also includes corrections to the template from
    https://review.openstack.org/350654

    Depends-On: Ic473eb9de26a2a3b92e234851b8f731c0255976c
    Change-Id: If52d073d43081468e2faf2cd063c4b211c29994f
    Related-Bug: #1605846
    Closes-Bug: #1609056
    (cherry picked from commit 38d7faffa96aa2b03e718320f2dc281cc12029b8)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/352095
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=4273df8562d5892fc73f0be3be91f0a44bff2292
Submitter: Jenkins
Branch: stable/mitaka

commit 4273df8562d5892fc73f0be3be91f0a44bff2292
Author: Jesse Pretorius <email address hidden>
Date: Sun Aug 7 11:16:48 2016 -0500

    Update role SHA's to pull in upper-constraints fix

    This patch updates the role SHA's to ensure that the upper-constraints
    fixes are included in the next release.

    It also includes an update to the lxc_hosts role to change the default
    apt sources used due to issues with the Rackspace mirror.

    Related-Bug: #1605846
    Closes-Bug: #1609056
    Depends-On: Ic473eb9de26a2a3b92e234851b8f731c0255976c
    Depends-On: I294e1f8842fff1ad643db64b506b44f29204f8d0
    Depends-On: Ic473eb9de26a2a3b92e234851b8f731c0255976c
    Change-Id: If95779e1e626dfb9140b77fcf4ea6b00028c73d2

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/350605
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=4c620fee1f509d15009fdc18c3ad53cac7bd5f69
Submitter: Jenkins
Branch: liberty

commit 4c620fee1f509d15009fdc18c3ad53cac7bd5f69
Author: Jesse Pretorius <email address hidden>
Date: Wed Aug 3 14:40:41 2016 +0100

    Fix repo constraints construction and package installation

    The current repo build process has the following issues:

    1. The constraints consruction doesn't properly handle two constraints
       which use the same name, but have different version specs. eg:
       ovs===2.5.0;python_version=='2.7'
       ovs===2.6.0.dev2;python_version=='3.4'
       This is a problem in the constraints construction.

    2. The pip packages installed on the repo server in order to
       construct the repo do not respect the global pins or the upper
       constraint overrides.

    3. The constraints are selected based on the requirements specified.
       This makes the process unnecessarily complex.

    4. The global pins are not applied to the constructed constraints,
       so the wrong packages get built and installed.

    This patch corrects all of these issues and hopefully makes the
    mechanism for constructing constraints more readily apparent, and
    therefore easier to maintain.

    This backport combines selective changes implemented in the
    following reviews:

    - Add logic to NOT build "proprietary" packages
      https://review.openstack.org/300505

    - Updated py_pkgs to index requirement files
      https://review.openstack.org/345448

    - Fix py_pkgs lookup to not include optional packages
      https://review.openstack.org/347834

    - Update py_pkgs to itemise global pins
      https://review.openstack.org/347930

    - Fix repo constraints construction and package installation
      https://review.openstack.org/350352

    - Correct global_pins depth
      https://review.openstack.org/350654

    Change-Id: If52d073d43081468e2faf2cd063c4b211c29994f
    Related-Bug: #1605846
    Closes-Bug: #1609056