Issues with upgrade due to older openstackclient in Keystone container

[James Denton]

Upgrading from r11.1.3 to 12.0.8 using the documentation found at:

http://docs.openstack.org/developer/openstack-ansible/liberty/upgrade-guide/manual-upgrade.html

During the 'os-keystone-install' play, the following task appeared to hang:

TASK: [os_keystone | Ensure service tenant] ***********************************
skipping: [infra03_keystone_container-d9ab093b]
skipping: [infra01_keystone_container-76b4424f]

I logged into the infra02_keystone_container and found I was unable to use the openstack client:

root@infra02_keystone_container-a210fbf5:~# openstack project list
Traceback (most recent call last):
  File "/usr/local/bin/openstack", line 7, in <module>
    from openstackclient.shell import main
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", line 28, in <module>
    from openstackclient.common import clientmanager
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/common/clientmanager.py", line 25, in <module>
    from openstackclient.api import auth
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/api/auth.py", line 25, in <module>
    from openstackclient.i18n import _
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/i18n.py", line 16, in <module>
    from oslo import i18n
ImportError: cannot import name i18n

It appeared the Kilo openstack client was still installed, but Oslo had been upgraded:

root@infra02_keystone_container-a210fbf5:~# pip freeze | grep oslo
oslo.concurrency==1.8.2
oslo.config==3.1.0
oslo.context==0.2.0
oslo.db==1.7.3
oslo.i18n==3.1.0
oslo.log==1.0.0
oslo.messaging==1.8.3
oslo.middleware==1.0.0
oslo.policy==0.3.2
oslo.serialization==2.1.0
oslo.utils==3.2.0

The openstack client could be used from the utility container, but it had not been updated.

Used pip to upgrade the client, and that appeared to resolve the issue upon subsequent reruns of the playbook:

root@infra02_keystone_container-a210fbf5:# pip install python-openstackclient
Ignoring indexes: https://pypi.python.org/simple
Collecting python-openstackclient
  Downloading http://172.30.224.72:8181/os-releases/12.0.8/python_openstackclient-2.0.0-py2.py3-none-any.whl (342kB)
    100% |################################| 344kB 69.6MB/s
Collecting python-novaclient!=2.33.0,>=2.29.0 (from python-openstackclient)
  Downloading http://172.30.224.72:8181/os-releases/12.0.8/python_novaclient-2.35.0-py2.py3-none-any.whl (330kB)
    100% |################################| 331kB 67.2MB/s
Requirement already up-to-date: stevedore>=1.5.0 in /usr/local/lib/python2.7/dist-packages (from python-openstackclient)
Requirement already up-to-date: oslo.config>=2.7.0 in /usr/local/lib/python2.7/dist-packages (from python-openstackclient)
Requirement already up-to-date: oslo.i18n>=1.5.0 in /usr/local/lib/python2.7/dist-packages (from python-openstackclient)
Requirement already up-to-date: python-keystoneclient!=1.8.0,>=1.6.0 in /usr/local/lib/python2.7/dist-packages (from python-openstackclient)
Collecting python-cinderclient>=1.3.1 (from python-openstackclient)
  Downloading http://172.30.224.72:8181/os-releases/12.0.8/python_cinderclient-1.5.0-py2.py3-none-any.whl (235kB)
    100% |################################| 237kB 39.5MB/s
Requirement already up-to-date: keystoneauth1>=1.0.0 in /usr/local/lib/python2.7/dist-packages (from python-openstackclient)
Requirement already up-to-date: requests>=2.8.1 in /usr/local/lib/python2.7/dist-packages (from python-openstackclient)
Collecting python-neutronclient>=2.6.0 (from python-openstackclient)
  Downloading http://172.30.224.72:8181/os-releases/12.0.8/python_neutronclient-3.1.0-py2.py3-none-any.whl (229kB)
    100% |################################| 229kB 68.3MB/s
Collecting python-glanceclient>=0.18.0 (from python-openstackclient)
  Downloading http://172.30.224.72:8181/os-releases/12.0.8/python_glanceclient-1.2.0-py2.py3-none-any.whl (176kB)
    100% |################################| 180kB 24.5MB/s
Requirement already up-to-date: oslo.utils>=2.8.0 in /usr/local/lib/python2.7/dist-packages (from python-openstackclient)
Requirement already up-to-date: Babel>=1.3 in /usr/local/lib/python2.7/dist-packages (from python-openstackclient)
Requirement already up-to-date: pbr>=1.6 in /usr/local/lib/python2.7/dist-packages (from python-openstackclient)
Requirement already up-to-date: six>=1.9.0 in /usr/local/lib/python2.7/dist-packages (from python-openstackclient)
Requirement already up-to-date: os-client-config!=1.6.2,>=1.4.0 in /usr/local/lib/python2.7/dist-packages (from python-openstackclient)
Collecting cliff>=1.15.0 (from python-openstackclient)
  Downloading http://172.30.224.72:8181/os-releases/12.0.8/cliff-1.15.0-py2-none-any.whl
Requirement already up-to-date: oslo.serialization>=1.10.0 in /usr/local/lib/python2.7/dist-packages (from python-novaclient!=2.33.0,>=2.29.0->python-openstackclient)
Requirement already up-to-date: argparse in /usr/local/lib/python2.7/dist-packages (from python-novaclient!=2.33.0,>=2.29.0->python-openstackclient)
Requirement already up-to-date: iso8601>=0.1.9 in /usr/local/lib/python2.7/dist-packages (from python-novaclient!=2.33.0,>=2.29.0->python-openstackclient)
Requirement already up-to-date: PrettyTable<0.8,>=0.7 in /usr/local/lib/python2.7/dist-packages (from python-novaclient!=2.33.0,>=2.29.0->python-openstackclient)
Requirement already up-to-date: simplejson>=2.2.0 in /usr/local/lib/python2.7/dist-packages (from python-novaclient!=2.33.0,>=2.29.0->python-openstackclient)
Requirement already up-to-date: netaddr!=0.7.16,>=0.7.12 in /usr/local/lib/python2.7/dist-packages (from oslo.config>=2.7.0->python-openstackclient)
Requirement already up-to-date: debtcollector>=0.3.0 in /usr/local/lib/python2.7/dist-packages (from python-keystoneclient!=1.8.0,>=1.6.0->python-openstackclient)
Requirement already up-to-date: warlock<2,>=1.0.1 in /usr/local/lib/python2.7/dist-packages (from python-glanceclient>=0.18.0->python-openstackclient)
Requirement already up-to-date: monotonic>=0.3 in /usr/local/lib/python2.7/dist-packages (from oslo.utils>=2.8.0->python-openstackclient)
Requirement already up-to-date: netifaces>=0.10.4 in /usr/local/lib/python2.7/dist-packages (from oslo.utils>=2.8.0->python-openstackclient)
Requirement already up-to-date: pytz>=2013.6 in /usr/local/lib/python2.7/dist-packages (from oslo.utils>=2.8.0->python-openstackclient)
Requirement already up-to-date: PyYAML>=3.1.0 in /usr/local/lib/python2.7/dist-packages (from os-client-config!=1.6.2,>=1.4.0->python-openstackclient)
Requirement already up-to-date: appdirs>=1.3.0 in /usr/local/lib/python2.7/dist-packages (from os-client-config!=1.6.2,>=1.4.0->python-openstackclient)
Requirement already up-to-date: requestsexceptions>=1.1.1 in /usr/local/lib/python2.7/dist-packages (from os-client-config!=1.6.2,>=1.4.0->python-openstackclient)
Requirement already up-to-date: cmd2>=0.6.7 in /usr/local/lib/python2.7/dist-packages (from cliff>=1.15.0->python-openstackclient)
Requirement already up-to-date: pyparsing>=2.0.1 in /usr/local/lib/python2.7/dist-packages (from cliff>=1.15.0->python-openstackclient)
Collecting unicodecsv>=0.8.0 (from cliff>=1.15.0->python-openstackclient)
  Downloading http://172.30.224.72:8181/os-releases/12.0.8/unicodecsv-0.14.1-py2-none-any.whl
Requirement already up-to-date: msgpack-python>=0.4.0 in /usr/local/lib/python2.7/dist-packages (from oslo.serialization>=1.10.0->python-novaclient!=2.33.0,>=2.29.0->python-openstackclient)
Requirement already up-to-date: wrapt>=1.7.0 in /usr/local/lib/python2.7/dist-packages (from debtcollector>=0.3.0->python-keystoneclient!=1.8.0,>=1.6.0->python-openstackclient)
Requirement already up-to-date: jsonschema<3,>=0.7 in /usr/local/lib/python2.7/dist-packages (from warlock<2,>=1.0.1->python-glanceclient>=0.18.0->python-openstackclient)
Requirement already up-to-date: jsonpatch<2,>=0.10 in /usr/local/lib/python2.7/dist-packages (from warlock<2,>=1.0.1->python-glanceclient>=0.18.0->python-openstackclient)
Requirement already up-to-date: functools32 in /usr/local/lib/python2.7/dist-packages (from jsonschema<3,>=0.7->warlock<2,>=1.0.1->python-glanceclient>=0.18.0->python-openstackclient)
Requirement already up-to-date: jsonpointer>=1.9 in /usr/local/lib/python2.7/dist-packages (from jsonpatch<2,>=0.10->warlock<2,>=1.0.1->python-glanceclient>=0.18.0->python-openstackclient)
Installing collected packages: python-novaclient, python-cinderclient, unicodecsv, cliff, python-neutronclient, python-glanceclient, python-openstackclient
  Found existing installation: python-novaclient 2.23.3
    Uninstalling python-novaclient-2.23.3:
      Successfully uninstalled python-novaclient-2.23.3
  Found existing installation: python-cinderclient 1.1.2
    Uninstalling python-cinderclient-1.1.2:
      Successfully uninstalled python-cinderclient-1.1.2
  Found existing installation: cliff 1.10.1
    Uninstalling cliff-1.10.1:
      Successfully uninstalled cliff-1.10.1
  Found existing installation: python-neutronclient 2.4.0
    Uninstalling python-neutronclient-2.4.0:
      Successfully uninstalled python-neutronclient-2.4.0
  Found existing installation: python-glanceclient 0.17.3
    Uninstalling python-glanceclient-0.17.3:
      Successfully uninstalled python-glanceclient-0.17.3
  Found existing installation: python-openstackclient 1.0.4
    Uninstalling python-openstackclient-1.0.4:
      Successfully uninstalled python-openstackclient-1.0.4
Successfully installed cliff-1.15.0 python-cinderclient-1.5.0 python-glanceclient-1.2.0 python-neutronclient-3.1.0 python-novaclient-2.35.0 python-openstackclient-2.0.0 unicodecsv-0.14.1

[Jesse Pretorius (jesse-pretorius)]

This is likely to be an issue for anything not installed into a venv, so the upgrade process should implement a flag to either force a reinstall or force an upgrade of any pip package installs.

[Nolan Brubaker (nolan-brubaker)]

Note that the packages are reinstalled when using the upgrade script; the manual process needs to have a note about the flag added in.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: liberty
Review: https://review.openstack.org/299008

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/299008
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=9b04c517e48ed6420552179c79ac84a3dc831414
Submitter: Jenkins
Branch: liberty

commit 9b04c517e48ed6420552179c79ac84a3dc831414
Author: Steve Lewis <email address hidden>
Date: Tue Mar 29 15:07:27 2016 -0700

    Add release note for services in venvs

    There is a known issue and an educational concern for operators who
    work in OSA-based environments upgraded to Liberty from a prior
    release.

    Change-Id: Id63d28b357674dd4b1c821feba9f94ba1e8c81e6
    Related-Bug: #1563377

[Jesse Pretorius (jesse-pretorius)]

Milestone will be re-added if/when a patch is submitted.

[James Denton (james-denton)]

Outdated and not an issue anymore.