os-neutron install fails from Kilo to Liberty

Bug #1577799 reported by Bjoern
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Medium
Cam J. Loader

Bug Description

TASK: [os_neutron | Install requires pip packages] ****************************
changed: [aio1_neutron_server_container-4ff7cca0] => (item=virtualenv)
changed: [aio1_neutron_agents_container-24399eca] => (item=virtualenv)
changed: [aio1] => (item=virtualenv)
changed: [aio1_neutron_server_container-4ff7cca0] => (item=virtualenv-tools)
changed: [aio1_neutron_agents_container-24399eca] => (item=virtualenv-tools)
changed: [aio1] => (item=virtualenv-tools)
changed: [aio1_neutron_server_container-4ff7cca0] => (item=python-keystoneclient)
changed: [aio1_neutron_agents_container-24399eca] => (item=python-keystoneclient)
changed: [aio1_neutron_server_container-4ff7cca0] => (item=httplib2)
changed: [aio1_neutron_agents_container-24399eca] => (item=httplib2)
failed: [aio1] => (item=python-keystoneclient) => {"attempts": 5, "cmd": "/usr/local/bin/pip install --force-reinstall python-keystoneclient", "failed": true, "item": "python-keystoneclient"}
msg: Task failed as maximum retries was encountered
changed: [aio1] => (item=httplib2)

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
os_neutron | Install requires pip packages ----------------------------- 38.62s
galera_client | Update apt sources ------------------------------------- 12.96s
Wait for container ssh -------------------------------------------------- 5.36s
galera_client | Install pip packages ------------------------------------ 4.09s
pip_install | Install PIP ----------------------------------------------- 3.23s
os_neutron | Ensure Rabbitmq vhost -------------------------------------- 2.30s
galera_client | Install galera package ---------------------------------- 2.02s
os_neutron | Ensure rabbitmq user --------------------------------------- 1.63s
Sort the rabbitmq servers ----------------------------------------------- 1.59s
Use the lxc-openstack aa profile ---------------------------------------- 1.55s
           to retry, use: --limit @/root/os-neutron-install.retry

aio1 : ok=38 changed=3 unreachable=0 failed=1
aio1_neutron_agents_container-24399eca : ok=44 changed=6 unreachable=0 failed=0
aio1_neutron_server_container-4ff7cca0 : ok=39 changed=5 unreachable=0 failed=0

Not sure why pbr would cause this issue:

  Found existing installation: pbr 0.11.1
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 726, in install
    requirement.uninstall(auto_confirm=True)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 699, in uninstall
    for path in pip.wheel.uninstallation_paths(dist):
  File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 533, in unique
    for item in fn(*args, **kw):
  File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 552, in uninstallation_paths
    r = csv.reader(FakeFile(dist.get_metadata_lines('RECORD')))
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 1624, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 1616, in get_metadata
    return self._get(self._fn(self.egg_info, name))
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 1727, in _get
    with open(path, 'rb') as stream:
IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/pbr-0.11.1.dist-info/RECORD'

Tags: in-liberty
Revision history for this message
Jesse Pretorius (jesse-pretorius) wrote :

Please add the output for this command executed on the host:

/usr/local/bin/pip install --force-reinstall --verbose python-keystoneclient

Revision history for this message
Bjoern (bjoern-t) wrote :

I'm redoing it, after verifying that nova/neutron kilo plays run successfully. After that I'll update this ticket

Changed in openstack-ansible:
status: New → Incomplete
Revision history for this message
Bjoern (bjoern-t) wrote :
Download full text (351.4 KiB)

I'm not sure how useful this is :

/usr/local/bin/pip install --force-reinstall --verbose python-keystoneclient
DEPRECATION: --allow-all-external has been deprecated and will be removed in the future. Due to changes in the repository protocol, it no longer has any effect.
Ignoring indexes: https://pypi.python.org/simple
Collecting python-keystoneclient
  1 location(s) to search for versions of python-keystoneclient:
  * http://172.29.236.100:8181/os-releases/12.0.13/
  Skipping link http://172.29.236.100:8181/os-releases/12.0.13/ (from -f); unsupported archive format: .13
  Getting page http://172.29.236.100:8181/os-releases/12.0.13/
  Starting new HTTP connection (1): 172.29.236.100
  "GET /os-releases/12.0.13/ HTTP/1.1" 200 None
  Analyzing links from page http://172.29.236.100:8181/os-releases/12.0.13/
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/MANIFEST.in (from http://172.29.236.100:8181/os-releases/12.0.13/); unsupported archive format: .in
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/aioeventlet-0.4-py2-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-keystoneclient)
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/alembic-0.8.3-py2.py3-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-keystoneclient)
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/amqp-1.4.8-py2.py3-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-keystoneclient)
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/ansible-1.9.6-py2-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-keystoneclient)
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/ansible_lint-2.3.9-py2-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-keystoneclient)
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/anyjson-0.3.3-py2-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-keystoneclient)
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/aodh-1.1.3.dev1-py2-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-keystoneclient)
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/apache_libcloud-0.15.1-py2.py3-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-keystoneclient)
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/appdirs-1.4.0-py2.py3-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-keystoneclient)
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/argcomplete-1.0.0-py2.py3-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-keystoneclient)
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/argparse-1.4.0-py2.py3-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-k...

Changed in openstack-ansible:
status: Incomplete → New
Revision history for this message
Bjoern (bjoern-t) wrote :
Download full text (300.7 KiB)

Diff on the pips before upgrade and to the moment where it stopped:

@@ -1,10 +1,11 @@
 aioeventlet==0.4
 alembic==0.8.0
 amqp==1.4.9
-ansible==1.9.3
+ansible==1.9.4
+ansible-lint==2.3.5
 anyjson==0.3.3
 apache-libcloud==0.15.1
-Babel==2.2.0
+Babel==2.1.1
 Beaver==33.3.0
 boto==2.39.0
 ceilometermiddleware==0.1.0
@@ -18,6 +19,7 @@
 conf-d==0.0.4
 configobj==5.0.6
 cryptography==1.2.2
+debtcollector==1.1.0
 decorator==4.0.9
 distro-info==0.12
 dnspython==1.12.0
@@ -51,9 +53,10 @@
 lxml==3.5.0
 Mako==1.0.3
 MarkupSafe==0.23
+monotonic==0.4
 mosquitto==1.2.3
 msgpack-pure==0.1.3
-msgpack-python==0.4.7
+msgpack-python==0.4.6
 MySQL-python==1.2.5
 netaddr==0.7.18
 netifaces==0.10.4
@@ -62,16 +65,16 @@
 numpy==1.10.4
 oauth==1.0.1
 oslo.concurrency==1.8.2
-oslo.config==1.9.3
+oslo.config==2.4.0
 oslo.context==0.2.0
 oslo.db==1.7.3
-oslo.i18n==1.5.0
+oslo.i18n==3.1.0
 oslo.log==1.0.0
 oslo.messaging==1.8.3
 oslo.middleware==1.0.0
 oslo.rootwrap==1.6.0
-oslo.serialization==1.4.0
-oslo.utils==1.4.2
+oslo.serialization==2.1.0
+oslo.utils==3.2.0
 oslo.vmware==0.8.0.post15
 paramiko==1.16.0
 Paste==2.0.2
@@ -97,7 +100,7 @@
 python-daemon==1.6.1
 python-editor==0.5
 python-glanceclient==0.17.3
-python-keystoneclient==1.3.4
+python-keystoneclient==1.7.2
 python-memcached==1.57
 python-mimeparse==1.5.1
 python-neutronclient==2.4.0
@@ -109,7 +112,7 @@
 PyYAML==3.11
 redis==2.10.5
 repoze.lru==0.6
-requests==2.7.0
+requests==2.8.1
 retrying==1.3.3
 rfc3986==0.3.1
 Routes==2.2
@@ -119,7 +122,7 @@
 SQLAlchemy==0.9.10
 sqlalchemy-migrate==0.9.7
 sqlparse==0.1.18
-stevedore==1.3.0
+stevedore==1.10.0
 suds==0.4
 swift==2.3.1.dev11
 tablib==0.11.1
@@ -131,8 +134,10 @@
 trollius==2.0
 unittest2==1.1.0
 urllib3==1.14
+virtualenv==13.1.2
+virtualenv-tools==1.0
 warlock==1.2.0
 WebOb==1.5.1
 websockify==0.6.1
-wheel==0.26.0
+wrapt==1.10.6
 xattr==0.7.8

Here the recent output:

Ignoring indexes: https://pypi.python.org/simple
Collecting python-keystoneclient
  1 location(s) to search for versions of python-keystoneclient:
  * http://172.29.236.100:8181/os-releases/12.0.13/
  Skipping link http://172.29.236.100:8181/os-releases/12.0.13/ (from -f); unsupported archive format: .13
  Getting page http://172.29.236.100:8181/os-releases/12.0.13/
  Starting new HTTP connection (1): 172.29.236.100
  "GET /os-releases/12.0.13/ HTTP/1.1" 200 None
  Analyzing links from page http://172.29.236.100:8181/os-releases/12.0.13/
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/MANIFEST.in (from http://172.29.236.100:8181/os-releases/12.0.13/); unsupported archive format: .in
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/aioeventlet-0.4-py2-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-keystoneclient)
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/alembic-0.8.3-py2.py3-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-keystoneclient)
    Skipping link http://172.29.236.100:8181/os-releases/12.0.13/amqp-1.4.8-py2.py3-none-any.whl (from http://172.29.236.100:8181/os-releases/12.0.13/); wrong project name (not python-keystoneclien...

Ian Cordasco (icordasc)
Changed in openstack-ansible:
assignee: nobody → Cameron Loader (4-cameron)
Revision history for this message
Cam J. Loader (cjloader) wrote :

The repo build should fix the problem because I just ran the upgrade and it worked just fine. I think just redoing the repo should fix this.

Changed in openstack-ansible:
status: New → Incomplete
Revision history for this message
Bjoern (bjoern-t) wrote :

This is a similar situation as https://bugs.launchpad.net/openstack-ansible/+bug/1585243 where I would vote to rebuild the repo container on every upgrade but then we have to do it inside the upgrade playbooks

Changed in openstack-ansible:
status: Incomplete → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible (liberty)

Fix proposed to branch: liberty
Review: https://review.openstack.org/328485

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible (liberty)

Reviewed: https://review.openstack.org/328485
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=368b7e2b0ef5abf792373f232087d0a4e595a5b9
Submitter: Jenkins
Branch: liberty

commit 368b7e2b0ef5abf792373f232087d0a4e595a5b9
Author: Cameron Loader <email address hidden>
Date: Fri Jun 10 15:21:05 2016 -0500

    Destroy repo containers before upgrade

    During the upgrade from Kilo to Liberty, this change deletes the
    repo containers and recreates them to fix an issue with dependencies.

    Change-Id: Iec9e306b41d39c09a6ef440861370d113e3cb3e5
    Closes-bug: 1577799
    Closes-bug: 1585243

tags: added: in-liberty
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/openstack-ansible 12.0.16

This issue was fixed in the openstack/openstack-ansible 12.0.16 release.

Changed in openstack-ansible:
status: In Progress → Fix Released
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.