Upgrade from 10.1.15 to r11.0.0 Fails Due to oslo.utils Downgrade

[Sean Carlisle]

Hey Team,

It looks like upgrades may be broken when moving from 10.1.15 to r11.0.0. Once the upgrade script got to the setup-infrastructure.yml playbook, it failed when attempting to install the python-glanceclient and python-novaclient pip packages in the utility containers, and so the whole playbook bombs out. Running the command manually reveals the problem:

root@617562-infra01_utility_container-52da13b5:/# /usr/local/bin/pip install python-novaclient
Ignoring indexes: https://pypi.python.org/simple
Requirement already up-to-date: python-novaclient in /usr/local/lib/python2.7/dist-packages
Requirement already up-to-date: oslo.i18n<1.6.0,>=1.5.0 in /usr/local/lib/python2.7/dist-packages (from python-novaclient)
Requirement already up-to-date: argparse in /usr/local/lib/python2.7/dist-packages (from python-novaclient)
Requirement already up-to-date: oslo.serialization<1.5.0,>=1.4.0 in /usr/local/lib/python2.7/dist-packages (from python-novaclient)
Requirement already up-to-date: requests!=2.4.0,>=2.2.0 in /usr/local/lib/python2.7/dist-packages (from python-novaclient)
Requirement already up-to-date: pbr!=0.7,<1.0,>=0.6 in /usr/local/lib/python2.7/dist-packages (from python-novaclient)
Requirement already up-to-date: oslo.utils<1.5.0,>=1.4.0 in /usr/local/lib/python2.7/dist-packages (from python-novaclient)
Requirement already up-to-date: iso8601>=0.1.9 in /usr/local/lib/python2.7/dist-packages (from python-novaclient)
Requirement already up-to-date: python-keystoneclient<1.4.0,>=1.2.0 in /usr/local/lib/python2.7/dist-packages (from python-novaclient)
Requirement already up-to-date: Babel>=1.3 in /usr/local/lib/python2.7/dist-packages (from python-novaclient)
Requirement already up-to-date: six>=1.9.0 in /usr/local/lib/python2.7/dist-packages (from python-novaclient)
Requirement already up-to-date: PrettyTable<0.8,>=0.7 in /usr/local/lib/python2.7/dist-packages (from python-novaclient)
Requirement already up-to-date: simplejson>=2.2.0 in /usr/local/lib/python2.7/dist-packages (from python-novaclient)
Requirement already up-to-date: msgpack-python>=0.4.0 in /usr/local/lib/python2.7/dist-packages (from oslo.serialization<1.5.0,>=1.4.0->python-novaclient)
Requirement already up-to-date: pytz>=2013.6 in /usr/local/lib/python2.7/dist-packages (from oslo.serialization<1.5.0,>=1.4.0->python-novaclient)
Requirement already up-to-date: pip in /usr/local/lib/python2.7/dist-packages (from pbr!=0.7,<1.0,>=0.6->python-novaclient)
Collecting netaddr<=0.7.13,>=0.7.12 (from oslo.utils<1.5.0,>=1.4.0->python-novaclient)
  Could not find a version that satisfies the requirement netaddr<=0.7.13,>=0.7.12 (from oslo.utils<1.5.0,>=1.4.0->python-novaclient) (from versions: 0.7.15)
No matching distribution found for netaddr<=0.7.13,>=0.7.12 (from oslo.utils<1.5.0,>=1.4.0->python-novaclient)

oslo.utils was upgraded to 1.4.1 in 10.1.15 and netaddr 0.7.13 is a requirement:

oslo.utils-1.4.1.dist-info/metadata.json:{"metadata_version": "2.0", "name": "oslo.utils", "version": "1.4.1", "classifiers": ["Environment :: OpenStack", "Intended Audience :: Information Technology", "Intended Audience :: System Administrators", "License :: OSI Approved :: Apache Software License", "Operating System :: POSIX :: Linux", "Programming Language :: Python", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.3"], "run_requires": [{"requires": ["pbr (!=0.7,<1.0,>=0.6)", "Babel (<=1.3,>=1.3)", "six (<=1.9.0,>=1.7.0)", "iso8601 (<=0.1.10,>=0.1.9)", "oslo.i18n (<=1.3.1,>=1.0.0)", "netaddr (<=0.7.13,>=0.7.12)"

It looks like oslo.utils is downgraded back to 1.4.0 for r11.0.0:

http://mirror.rackspace.com/rackspaceprivatecloud/os-releases/11.2.1/

Since pip does not downgrade packages, it keeps oslo.utils at 1.4.1 instead of downgrading to 1.4.0, breaking the requirement for netaddr<=0.7.13,>=0.7.12.

Steps to Reproduce:

* Install Greenfield 10.1.15
* Follow the upgrade guide found here: http://docs.rackspace.com/rpc/api/v11/bk-rpc-releasenotes/content/ch-upgrade.html

Can we utilize the "--force-reinstall" flag for all pip install commands to prevent this in the future since the pip wheels keep changing?

Thanks,

Sean

[Ian Cordasco (icordasc)]

Pretty sure this needs to be filed against https://github.com/rcbops/rpc-openstack

r11.0.0 is not a tag or ref that exists in openstack-ansible. Want me to copy this over for you?

[Jordan Callicoat (jcallicoat)]

Looks like this is a concrete example of LP:1489251 for OSA.

[Jesse Pretorius (jesse-pretorius)]

rpc-openstack r11.0.0 embeds openstack-ansible 11.2.1 and this issue applies to openstack-ansible upgrades too. The issue is not isolated.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/229383

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229383
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=2c19010ff9a936c5d0ad3d72e2015308c105ff0f
Submitter: Jenkins
Branch: master

commit 2c19010ff9a936c5d0ad3d72e2015308c105ff0f
Author: Jesse Pretorius <email address hidden>
Date: Wed Sep 30 12:48:13 2015 +0100

    Allow pip extra arguments to be passed to playbooks

    This patch adds the variable 'pip_install_options' which is passed to the pip
    install module as extra arguments in order to allow the use of options like
    '--force-reinstall' when executing playbooks.

    eg: openstack-ansible -e pip_install_options="--force-reinstall" \
            setup-openstack.yml

    This is required due to constant upstream changes in dependencies which
    result in python wheel version upgrades and downgrades between tagged
    versions of openstack-ansible.

    The intention is that this can be used whenever a deployer switches between
    tags for both upgrades and downgrades.

    DocImpact
    Closes-Bug: #1489251
    Closes-Bug: #1499451
    Related-Bug: #1501114
    Change-Id: I996185e009a4c4af4f23798619bdbd0d490360c9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: kilo
Review: https://review.openstack.org/229803

[Jesse Pretorius (jesse-pretorius)]

Marked invalid for Liberty as this doesn't directly apply to Liberty. I have added this bug as a reference for https://blueprints.launchpad.net/openstack-ansible/+spec/liberty-upgrade-path to try to ensure that this is taken into consideration for the Liberty upgrade testing.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229542
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=f5e3b1171f41ff10b60725c7c30d13a18f3156ca
Submitter: Jenkins
Branch: kilo

commit f5e3b1171f41ff10b60725c7c30d13a18f3156ca
Author: Jesse Pretorius <email address hidden>
Date: Wed Sep 30 12:48:13 2015 +0100

    Allow pip extra arguments to be passed to playbooks

    This patch adds the variable 'pip_install_options' which is passed to the pip
    install module as extra arguments in order to allow the use of options like
    '--force-reinstall' when executing playbooks.

    eg: openstack-ansible -e pip_install_options="--force-reinstall" \
            setup-openstack.yml

    This is required due to constant upstream changes in dependencies which
    result in python wheel version upgrades and downgrades between tagged
    versions of openstack-ansible.

    The intention is that this can be used whenever a deployer switches between
    tags for both upgrades and downgrades.

    DocImpact
    Closes-Bug: #1489251
    Closes-Bug: #1499451
    Related-Bug: #1501114
    Change-Id: I996185e009a4c4af4f23798619bdbd0d490360c9
    (cherry picked from commit 2c19010ff9a936c5d0ad3d72e2015308c105ff0f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229803
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=70656eb25bd678b541ff7e3a58a586701c79f12f
Submitter: Jenkins
Branch: kilo

commit 70656eb25bd678b541ff7e3a58a586701c79f12f
Author: Jesse Pretorius <email address hidden>
Date: Thu Oct 1 10:50:34 2015 +0100

    Ensure that all pip wheels are reinstalled during major upgrades

    This patch implements the '--force-reinstall' for all pip install tasks during
    the upgrade process.

    This is required due to upstream python wheel version changes which include
    downgrades from previously implemented versions.

    Closes-Bug: #1501114
    Change-Id: I7682f3a7f3c8481b21fb00b71cfd384f15dc5eeb

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.11 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 11.2.12 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.14 release.