Kilo upgrade to Liberty died on repo build process

Bug #1585243 reported by Bjoern
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
High
Cam J. Loader

Bug Description

Currently the upgrade process died from a 9abc2c85cf62a21d80dfa2d8f6a70022c0a346fc to d4e54bff5b602711c8e7bbbe6e800349fb4d5f22 OSA SHA

TASK: [repo_build | Create OpenStack-Ansible requirement wheels] **************
failed: [12345-infra02_repo_container-4ffc3fd9] => {"changed": true, "cmd": "pip wheel --timeout 120 --wheel-dir /tmp/openstack-wheel-output --allow-all-external --find-links /tmp/openstack-wheel-output --
constraint /var/www/repo/os-releases/12.0.13/requirements_constraints.txt --index-url https://rpc-repo.rackspace.com/pools --trusted-host rpc-repo.rackspace.com --extra-index-url https://pypi.python.org/sim
ple --trusted-host pypi.python.org --build /tmp/openstack-builder --log /var/log/repo/repo_builder.log --requirement /var/www/repo/os-releases/12.0.13/requirements_local_filtered.txt", "delta": "0:00:02.103
216", "end": "2016-05-24 09:49:18.055923", "rc": 1, "start": "2016-05-24 09:49:15.952707", "warnings": []}
stderr: DEPRECATION: --allow-all-external has been deprecated and will be removed in the future. Due to changes in the repository protocol, it no longer has any effect.
Ignoring dnspython3: markers u"python_version=='3.4'" don't match your environment
Note: checking out '62b18d667064506b3f8f62653fc0b8662b41ebf5'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

  Could not find a tag or branch '62b18d667064506b3f8f62653fc0b8662b41ebf5', assuming commit.
Command "python setup.py egg_info" failed with error code 1 in /tmp/openstack-builder/aodh/
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
stdout: Collecting aodh from git+file:///var/www/repo/openstackgit/aodh@62b18d667064506b3f8f62653fc0b8662b41ebf5#egg=aodh (from -c /var/www/repo/os-releases/12.0.13/requirements_constraints.txt (line 2))
  Cloning file:///var/www/repo/openstackgit/aodh (to 62b18d667064506b3f8f62653fc0b8662b41ebf5) to /tmp/openstack-builder/aodh
    Complete output from command python setup.py egg_info:
    ERROR:root:Error parsing
    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/pbr/core.py", line 109, in pbr
        attrs = util.cfg_to_args(path)
      File "/usr/local/lib/python2.7/dist-packages/pbr/util.py", line 261, in cfg_to_args
        wrap_commands(kwargs)
      File "/usr/local/lib/python2.7/dist-packages/pbr/util.py", line 482, in wrap_commands
        for cmd, _ in dist.get_command_list():
      File "/usr/local/lib/python2.7/dist-packages/setuptools/dist.py", line 447, in get_command_list
        cmdclass = ep.resolve()
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2235, in resolve
        module = __import__(self.module_name, fromlist=['__name__'], level=0)
      File "/usr/local/lib/python2.7/dist-packages/pbr/testr_command.py", line 47, in <module>
        from testrepository import commands
    ImportError: No module named testrepository
    error in setup command: Error parsing /tmp/openstack-builder/aodh/setup.cfg: ImportError: No module named testrepository

    ----------------------------------------

FATAL: all hosts have already failed -- aborting

# git status
HEAD detached at d4e54bf

Tags: in-liberty
Revision history for this message
Ian Cordasco (icordasc) wrote :

Can you provide the value of `setuptools` inside the repo container in question?

Revision history for this message
Ian Cordasco (icordasc) wrote :

s/value/version/

Revision history for this message
Bjoern (bjoern-t) wrote :

Well I rebuild the container already so I only have the playbook logs at this time. But I'm pretty sure I will see that again, running another upgrade today

Ian Cordasco (icordasc)
Changed in openstack-ansible:
assignee: nobody → Xin (zxuuzx)
Revision history for this message
Jesse Pretorius (jesse-pretorius) wrote :

It seems to me that cruft is being left behind during the upgrade process - most likely bits of pip/setuptools/wheel that are in system packages and aren't being cleaned up properly. This is why the container delete and rebuild works.

Revision history for this message
Bjoern (bjoern-t) wrote :

I ran into those kind of issues on multiple upgrades since upgrading from Juno to Kilo etc.
At some point I prefer that we delete the repo containers as part of the upgrades and rebuild them freshly. But that also can cause issues in the case where 3rd party pips are build onto repo containers to include them into the release tree. I would vote for a rebuild approach including how to side load 3rd party pip packages like needed for cinder drivers etc.

Revision history for this message
Jean-Philippe Evrard (jean-philippe-evrard) wrote :

Assigned to Cameron Loader due to this commit: https://review.openstack.org/#/c/328485/3

Changed in openstack-ansible:
assignee: Xin (zxuuzx) → Cameron Loader (4-cameron)
Changed in openstack-ansible:
status: New → Confirmed
Changed in openstack-ansible:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible (liberty)

Reviewed: https://review.openstack.org/328485
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=368b7e2b0ef5abf792373f232087d0a4e595a5b9
Submitter: Jenkins
Branch: liberty

commit 368b7e2b0ef5abf792373f232087d0a4e595a5b9
Author: Cameron Loader <email address hidden>
Date: Fri Jun 10 15:21:05 2016 -0500

    Destroy repo containers before upgrade

    During the upgrade from Kilo to Liberty, this change deletes the
    repo containers and recreates them to fix an issue with dependencies.

    Change-Id: Iec9e306b41d39c09a6ef440861370d113e3cb3e5
    Closes-bug: 1577799
    Closes-bug: 1585243

tags: added: in-liberty
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/openstack-ansible 12.0.16

This issue was fixed in the openstack/openstack-ansible 12.0.16 release.

Changed in openstack-ansible:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.