python-ceilometerclient 1.0.12 has uncapped oslo.utils which breaks unit test jobs on stable/juno

[Matt Riedemann]

For example:

http://logs.openstack.org/05/206705/3/check/gate-horizon-python27/3e84064/console.html#_2015-08-22_19_19_15_770

The details are in this IRC discussion:

(4:37:03 PM) mriedem: tonyb: http://logs.openstack.org/05/206705/3/check/gate-horizon-python27/3e84064/console.html#_2015-08-22_19_19_15_770 is actually a keystoneclient thing it looks like
(4:38:36 PM) tonyb: mriedem: Hmmm, okay. It's keystone that's failing but I thought the version of oslo that git pulled in was because of pythonn-ceilometer
(4:38:48 PM) tonyb: mriedem: let me check the pip logs
(4:39:02 PM) mriedem: well, stable/juno should be using python-ceilometerclient 1.0.12
(4:39:07 PM) mriedem: since g-r is capped at <1.0.13
(4:39:15 PM) mriedem: and keystoneclient is uncapped in ceilometerclient 1.0.12
(4:39:15 PM) mriedem: https://github.com/openstack/python-ceilometerclient/blob/1.0.12/requirements.txt#L9
(4:39:35 PM) mriedem: python-ceilometerclient==1.0.12
(4:39:45 PM) mriedem: python-keystoneclient==1.1.1
(4:39:58 PM) mriedem: which is ok given keystoneclient and g-r
(4:39:58 PM) mriedem: https://github.com/openstack/requirements/blob/stable/juno/global-requirements.txt#L114
(4:40:03 PM) mriedem: python-keystoneclient>=0.10.0,<1.2.0
(4:40:23 PM) mriedem: oslo.utils==2.3.0 is not ok
(4:40:28 PM) mriedem: https://github.com/openstack/requirements/blob/stable/juno/global-requirements.txt#L68
(4:40:57 PM) tonyb: okay
(4:41:01 PM) mriedem: keystoneclient 1.1.1 has capped oslo.utils https://github.com/openstack/python-keystoneclient/blob/1.1.1/requirements.txt#L14
(4:41:11 PM) mriedem: you're right though
(4:41:17 PM) mriedem: ceilometerclient 1.0.12 has uncapped oslo.utils
(4:41:17 PM) mriedem: https://github.com/openstack/python-ceilometerclient/blob/1.0.12/requirements.txt#L7
(4:41:23 PM) mriedem: m'fer
(4:41:49 PM) mriedem: and g-r stable/juno caps ceilometerclient at <1.0.13, and g-r stable/kilo has ceilometerclient>=1.0.13
(4:41:54 PM) mriedem: so we are stuck :(
(4:42:07 PM) tonyb: Yeah so doesnt that mean g-r merge in ceilometerclient and then release
(4:42:09 PM) mriedem: should have done a ceilometerclient 1.1 for kilo
(4:42:23 PM) mriedem: tonyb: but release it as what?
(4:42:32 PM) mriedem: 1.0.13 is taken
(4:43:32 PM) tonyb: So I was thinking that we change juno g-r to be <1.1.0,!=1.0.13,!=1.0.14 and release 1.0.15 (and then black list 1.0.15 in kilo)
(4:43:44 PM) tonyb: it's gross but wil it work?
(4:44:19 PM) tonyb: I may have the versions slightly wrong but that's my idea
(4:44:45 PM) mriedem: well,
(4:45:01 PM) mriedem: i'd think you'd actually have to raise kilo ceilometerclient to >=1.1,<1.2
(4:45:38 PM) mriedem: oh god, even g-r on master has ceilometerclient>=1.0.13
(4:45:39 PM) mriedem: https://github.com/openstack/requirements/blob/master/global-requirements.txt#L148
(4:46:26 PM) mriedem: seems we should raise the minimum required version of python-ceilometerclient on master before we forget that's overlapping with stable/kilo
(4:46:32 PM) tonyb: *sigh*
(4:46:56 PM) mriedem: so "<1.1.0,!=1.0.13,!=1.0.14 and release 1.0.15" on stable/juno g-r sounds right
(4:47:02 PM) mriedem: we want to reserve 1.0.x for juno
(4:47:41 PM) tonyb: mriedem: okay.
(4:48:47 PM) mriedem: i'm not sure how to unfudge kilo right now
(4:48:58 PM) mriedem: besides >=1.1,<1.2
(4:49:12 PM) mriedem: but we'd have to raise minimum ceilometerclient on master first
(4:49:14 PM) mriedem: to >=1.2
(4:50:15 PM) mriedem: tonyb: the question would be if there is anything in ceilometerclient 1.1 that doesn't work with stable/kilo g-r
(4:51:07 PM) mriedem: tonyb: oslo.i18n would be a problem
(4:51:11 PM) mriedem: in kilo for ceilometerclient 1.1
(4:51:15 PM) tonyb: mriedem: Yeah. I guess we (I) need to write this all down as best I can and get some ceilometer eyes on it.
(4:51:31 PM) mriedem: i went in the ceilometer channel and asked for gordc but he's not around
(4:51:38 PM) mriedem: dumped my concern anyway
(4:51:40 PM) tonyb: mriedem: Yeah I'm working on new releases of a few oslo libraries
(4:51:56 PM) mriedem: so maybe for kilo, we'd need to sync g-r stable/kilo to ceilometerclient 1.1.0 and release that as 1.1.1
(4:52:17 PM) mriedem: and make ceilometerclient >=1.1.1,<1.2
(4:52:34 PM) mriedem: and then master ceilometerclient is >=1.2
(4:53:47 PM) tonyb: mriedem: I think that makes sense (IIUC)
(4:54:23 PM) tonyb: mriedem: If you see/hear me talk about the tangled web of onions this is what I'm talking about.
(4:55:00 PM) tonyb: mriedem: which is why I considered just not fixing it, but I've been encouraged to make this work. so .....
(4:55:42 PM) mriedem: note that the latest release from stable/kilo ceilometerclient was 1.0.14
(4:55:57 PM) mriedem: but making g-r stable/kilo require >=1.1.1 would ignore 1.0.14
(4:56:08 PM) mriedem: and we'd block 1.0.13 and 1.0.14 from g-r stable/juno as you said
(4:56:45 PM) tonyb: mriedem: Yeah. I kinda thought we'd have to do alternating blacklists in juno and kilo (so add !=1.0.15 to kilo) but you're idea is neater
(4:57:32 PM) mriedem: well, the simple big picture idea in my mind is reserve 1.0.x for juno, 1.1.x for kilo and >=1.2 for master
(4:57:55 PM) mriedem: basically, the series of 1.0.x relases in ceilometerclient screwed this up, there should have been minor version bumps in there
(4:58:12 PM) tonyb: mriedem: Yeah I couldn't think of a way to do that but you have (assuming it works) :)
(4:58:14 PM) mriedem: let me open a bug for python-ceilometerclient and try to post this there

[Matt Riedemann (mriedem)]

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiZnJvbSBvc2xvLnV0aWxzIGltcG9ydCB0aW1ldXRpbHNcIiBBTkQgdGFnczpcImNvbnNvbGVcIiBBTkQgYnVpbGRfYnJhbmNoOlwic3RhYmxlL2p1bm9cIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQ0MTkyMTA0Njc1MywibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

[Matt Riedemann (mriedem)]

proposed fix:

1. g-r master (liberty): python-ceilometerclient>=1.2

2. g-r stable/kilo: python-ceilometerclient>=1.1.1,<1.2 (release a sync of stable/kilo g-r to stable/kilo python-ceilometerclient as 1.1.1)

3. g-r stable/juno: python-ceilometerclient<1.1.0,!=1.0.13,!=1.0.14 and release 1.0.15 with a sync of stable/juno g-r

The point is, leave 1.0.x for juno, 1.1.x for kilo and >=1.2 for liberty

[Matt Riedemann (mriedem)]

Note that the latest release from stable/kilo was 1.0.14:

https://review.openstack.org/#/c/175883/

There have been g-r syncs for stable/kilo since then though, which should be released as 1.1.x.

[Matt Riedemann (mriedem)]

Here is the g-r change for master: https://review.openstack.org/#/c/222386/

[Matt Riedemann (mriedem)]

gordon chung Sep 10 9:17 PM
Patch Set 1:
ok yeah, i just confirmed, 1.1.0 ~= 1.0.14 minus requirements.
if the requirements for 1.1.0 work for stable/kilo, we're all good.

[Matt Riedemann (mriedem)]

python-ceilometerclient 1.1.1 release request for stable/kilo:

https://review.openstack.org/#/c/223684/

[Matt Riedemann (mriedem)]

stable/kilo g-r cap change: https://review.openstack.org/#/c/223734/

[Matt Riedemann (mriedem)]

The stable/juno g-r sync https://review.openstack.org/#/c/173126/ is failing because of the existing caps on ceilometerclient in stable/juno:

http://logs.openstack.org/26/173126/4/check/gate-tempest-dsvm-neutron-src-python-ceilometerclient/3ebd171/logs/screen-s-proxy.txt.gz

pkg_resources.ContextualVersionConflict: (python-ceilometerclient 1.0.13.dev2 (/opt/stack/new/python-ceilometerclient), Requirement.parse('python-ceilometerclient<1.0.13,>=1.0.6'), set(['ceilometer']))

This is because ceilometer on stable/juno has capped ceilometerclient:

https://github.com/openstack/ceilometer/blob/stable/juno/requirements.txt#L32

python-ceilometerclient<1.0.13,>=1.0.6

So our plan for g-r stable/juno was to merge the sync patch for ceilometerclient https://review.openstack.org/#/c/173126/ and release that as 1.0.15, and then update g-r on stable/juno to be:

python-ceilometerclient>=1.0.6,!=1.0.13,!=1.0.14,<1.1.0

But we can't land that g-r sync change due to this chicken/egg issue.

So how to move forward?

One idea is to make the g-r change we plan to make anyway, which is python-ceilometerclient>=1.0.6,!=1.0.13,!=1.0.14,<1.1.0. That would then sync to ceilometer on stable/juno, but I'm guessing that sync will fail because of this bug we're trying to fix here, in 1.0.12 since that has uncapped oslo.utils.

We could force merge https://review.openstack.org/#/c/173126/ and release that as 1.0.15 and then move ahead with our g-r change of python-ceilometerclient>=1.0.6,!=1.0.13,!=1.0.14,<1.1.0 - when that syncs to ceilometer it should be fine since 1.0.15 will be available and within that range.

[Matt Riedemann (mriedem)]

Note that we should merge the stable/kilo g-r change https://review.openstack.org/#/c/223734/ before we release ceilometerclient 1.0.15.

[Matt Riedemann (mriedem)]

Another idea, we could maybe also block ceilometerclient 1.0.12 from stable/juno g-r since that's the problem version. That leaves us with 1.0.11 for ceilometerclient in stable/juno, and that didn't have oslo.utils:

https://github.com/openstack/python-ceilometerclient/blob/1.0.11/requirements.txt

So maybe that's something that can get us around this. Given this bug, we should probably block 1.0.12 anyway.

[Matt Riedemann (mriedem)]

This is the stable/juno g-r change: https://review.openstack.org/#/c/224807/

[Matt Riedemann (mriedem)]

stable/juno ceilometer g-r sync: https://review.openstack.org/#/c/224947/

[Matt Riedemann (mriedem)]

We now have the g-r stable/juno sync to python-ceilometerclient merged:

https://review.openstack.org/#/c/173126/

We can release that as 1.0.15.

[Matt Riedemann (mriedem)]

1.0.15 release request: https://review.openstack.org/#/c/225339/

[Matt Riedemann (mriedem)]

1.0.15 was released on 9/23:

https://pypi.python.org/pypi/python-ceilometerclient/1.0.15