stable/juno gate is busted on Oslo releases on 4/21

Bug #1446847 reported by Matt Riedemann
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack-Gate
Fix Committed
Undecided
Unassigned
cliff
Fix Released
Critical
Doug Hellmann

Bug Description

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRVJST1I6IG9wZW5zdGFja2NsaWVudC5zaGVsbCBFeGNlcHRpb24gcmFpc2VkOiAoc3RldmVkb3JlIDEuMi4wICgvdXNyL2xvY2FsL2xpYi9weXRob24yLjcvZGlzdC1wYWNrYWdlcyksIFJlcXVpcmVtZW50LnBhcnNlKCdzdGV2ZWRvcmU+PTEuMy4wJyksIHNldChbJ2NsaWZmJ10pKVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI5NjQ4MjU4ODkyfQ==

http://logs.openstack.org/89/165189/1/gate/gate-tempest-dsvm-full/c896f3b/logs/devstacklog.txt.gz#_2015-04-21_20_20_26_378

ERROR: openstackclient.shell Exception raised: (stevedore 1.2.0 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('stevedore>=1.3.0'), set(['cliff']))

stevedore 1.4.0 and cliff 1.11.0 were released today.

Known direct causes:
 - python-openstack-client has an uncapped python-novaclient, which then violates the constraint on python-novaclient from global requirements as python-openstack-client is installed from pip.
 - keystonemiddleware has an uncapped requirement on oslo.config which has an uncapped requirement on stevedore, also installed from pip [so no requirements file override in place]

Both would be addressed if we installed lower versions of python-novaclient and oslo.config in advance

Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
Matt Riedemann (mriedem) wrote :

openstackclient 1.1.0 was released today too.

on stable/juno that should be capped:

https://github.com/openstack/requirements/blob/stable/juno/global-requirements.txt#L117

python-openstackclient>=0.3.0,<=1.0.1

Revision history for this message
Matt Riedemann (mriedem) wrote :

from pip-freeze on stable/juno:

python-openstackclient==1.0.1

stevedore==1.2.0

cliff==1.11.0

That cliff version is outside the cap on stable/juno: cliff>=1.7.0,<=1.9.0 # Apache-2.0

Revision history for this message
Doug Hellmann (doug-hellmann) wrote :

Related change in cliff: https://review.openstack.org/176089
Related change in stable/juno cinder: https://review.openstack.org/176082

Revision history for this message
Robert Collins (lifeless) wrote :

pip install 'keystonemiddleware>=1.0.0,<1.4.0' is called directly.
Collecting stevedore>=1.3.0 (from oslo.config>=1.4.0->keystonemiddleware<1.4.0,>=1.0.0)

Then *after* that we call _sync_requirements_file which applies
'stevedore': 'stevedore>=1.0.0,<=1.2.0'

Then we install keystone:
install -e /opt/stack/new/keystone
Which transitively is ok with stevedore
Requirement already satisfied (use --upgrade to upgrade): stevedore>=1.1.0 in /usr/local/lib/python2.7/dist-packages (from python-keystoneclient<1.2.0,>=0.10.0->keystone==2014.2.4.dev2)

And then we lower it - invalidating previously made pip choices (https://github.com/pypa/pip/issues/2687)
pip install -e /opt/stack/new/glance
...
Collecting stevedore<=1.2.0,>=1.0.0 (from glance==2014.2.4.dev1)
2015-04-21 20:16:34.076 | Downloading http://pypi.region-b.geo-1.openstack.org/packages/py2.py3/s/stevedore/stevedore-1.2.0-py2.py3-none-any.whl

Now, we're installing keystonemiddleware from a release (good), but that release strictly requires a newer version of stevedore than Juno is willing to accept.

One possible fix : use an older keystonemiddleware.
Another possible fix, loosen the global requirements for juno to accept newer stevedores - since its API compatible (1.2 ~= 1.4) we should be able to do that.

Revision history for this message
Robert Collins (lifeless) wrote :

The wrong cliff version turns up here:
pip install -e /opt/stack/new/cinder
..
Collecting python-barbicanclient!=3.0.0,<3.0.2,>=2.1.0 (from cinder==2014.2.4.dev7)
...
Collecting cliff>=1.7.0 (from python-barbicanclient!=3.0.0,<3.0.2,>=2.1.0->cinder==2014.2.4.dev7)

Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
Doug Hellmann (doug-hellmann) wrote :

I'm not sure why https://review.openstack.org/#/c/176113 didn't show up as related to this bug.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote :

I also started an etherpad this evening with more notes about the blockages and what we're investigating: https://etherpad.openstack.org/p/april-21-gate-wedge

Changed in python-cliff:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Doug Hellmann (doug-hellmann)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to cliff (master)

Reviewed: https://review.openstack.org/176089
Committed: https://git.openstack.org/cgit/openstack/cliff/commit/?id=a280892bcedcc83a77d16bfc56d072cd2fc90230
Submitter: Jenkins
Branch: master

commit a280892bcedcc83a77d16bfc56d072cd2fc90230
Author: Doug Hellmann <email address hidden>
Date: Tue Apr 21 21:34:41 2015 +0000

    Do not check requirements when loading plugins

    Turn off the setuptools requirements check when loading plugins to avoid
    errors with code that works but where the package metadata makes
    setuptools think it might not.

    Change-Id: I0b15d3a8d311f0ad988c16758cec32280c83eb7e
    Related-bug: #1446847

Changed in python-cliff:
status: In Progress → Fix Committed
milestone: none → next-liberty
status: Fix Committed → Fix Released
Revision history for this message
Robert Collins (lifeless) wrote : Re: stable/juno gate is busted on stevedore/cliff releases on 4/21

Collecting python-novaclient>=2.18.0 (from python-openstackclient<=1.0.1,>=0.3.0)
which pulls in 2.24 which brings in the oslo.utils conflict

description: updated
description: updated
Revision history for this message
Alan Pevec (apevec) wrote :

ContextualVersionConflict: (oslo.i18n 1.3.1 (/home/jenkins/workspace/periodic-glance-python27-juno/.tox/py27/lib/python2.7/site-packages), Requirement.parse('oslo.i18n>=1.5.0'), set(['oslo.utils', 'pycadf']))

summary: - stable/juno gate is busted on stevedore/cliff releases on 4/21
+ stable/juno gate is busted on Oslo releases on 4/21
Revision history for this message
Brant Knudson (blk-u) wrote :

We haven't been able to create a keystonemiddleware with updated caps because of the uncapped oslo.messaging in the stable/juno pycadf. For this we need a stable/juno pycadf branch and a release of pycadf that caps oslo.messaging.

Revision history for this message
Brant Knudson (blk-u) wrote :

Here's the (failing) review by proposal bot to update keystonemiddleware caps: https://review.openstack.org/#/c/173123/2

Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

I haven't seen that failure for a long time. I guess we can close the bug.

Changed in openstack-gate:
status: New → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.