grenade: swift-proxy-server fails to start in juno due to python-neutronclient 2.3.12 caps

[Matt Riedemann]

Looks like this is killing grenade jobs in kilo:

http://logs.openstack.org/57/210557/1/check/gate-grenade-dsvm/51b2e81/logs/grenade.sh.txt.gz#_2015-08-07_19_16_15_899

2015-08-07 19:15:30.407 | venv create: /opt/stack/new/tempest/.tox/venv
2015-08-07 19:15:32.619 | venv installdeps: -r/opt/stack/new/tempest/requirements.txt, -r/opt/stack/new/tempest/test-requirements.txt
2015-08-07 19:15:59.505 | venv develop-inst: /opt/stack/new/tempest
2015-08-07 19:16:12.503 | venv installed: -f file:///opt/stack/new/.wheelhouse,anyjson==0.3.3,appdirs==1.4.0,Babel==2.0,boto==2.38.0,cffi==1.1.2,cliff==1.13.0,cmd2==0.6.8,coverage==3.7.1,cryptography==0.9.3,debtcollector==0.7.0,docutils==0.12,ecdsa==0.13,enum34==1.0.4,extras==0.0.3,fasteners==0.12.0,fixtures==1.3.1,flake8==2.2.4,funcsigs==0.4,functools32==3.2.3.post2,hacking==0.10.2,httplib2==0.9.1,idna==2.0,ipaddress==1.0.14,iso8601==0.1.10,Jinja2==2.8,jsonschema==2.5.1,linecache2==1.0.0,MarkupSafe==0.23,mccabe==0.2.1,mock==1.3.0,monotonic==0.3,mox==0.5.3,mox3==0.9.0,msgpack-python==0.4.6,netaddr==0.7.15,netifaces==0.10.4,os-client-config==1.6.1,os-testr==0.3.0,oslo.concurrency==2.4.0,oslo.config==2.2.0,oslo.context==0.5.0,oslo.i18n==2.3.0,oslo.log==1.8.0,oslo.serialization==1.8.0,oslo.utils==2.2.0,oslosphinx==3.1.0,oslotest==1.10.0,paramiko==1.15.2,pbr==1.4.0,pep8==1.5.7,posix-ipc==1.0.0,prettytable==0.7.2,pyasn1==0.1.8,pycparser==2.14,pycrypto==2.6.1,pyflakes==0.8.1,Pygments==2.0.2,pyOpenSSL==0.15.1,pyparsing==2.0.3,python-mimeparse==0.1.4,python-subunit==1.1.0,pytz==2015.4,PyYAML==3.11,requests==2.7.0,retrying==1.3.3,six==1.9.0,Sphinx==1.2.3,stevedore==1.7.0,-e git+https://git.openstack.org/openstack/tempest@c09fd74805c44059713f2ffd2485908e741f9e00#egg=tempest-master,tempest-lib==0.7.0,testrepository==0.0.20,testscenarios==0.5.0,testtools==1.8.0,traceback2==1.4.0,unittest2==1.1.0,wheel==0.24.0,wrapt==1.10.5
2015-08-07 19:16:12.504 | venv runtests: PYTHONHASHSEED='3882246887'
2015-08-07 19:16:12.504 | venv runtests: commands[0] | verify-tempest-config -uro ../../../../tmp/tmp.9RokwHMVCb
2015-08-07 19:16:15.893 | Running config verification...
2015-08-07 19:16:15.899 | ERROR: InvocationError: '/opt/stack/new/tempest/.tox/venv/bin/verify-tempest-config -uro ../../../../tmp/tmp.9RokwHMVCb'

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRVJST1I6IEludm9jYXRpb25FcnJvcjogJy9vcHQvc3RhY2svbmV3L3RlbXBlc3QvLnRveC92ZW52L2Jpbi92ZXJpZnktdGVtcGVzdC1jb25maWcgLXVyb1wiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDM4OTc3MTc3NDY1fQ==

[Matt Riedemann (mriedem)]

Here we go:

http://logs.openstack.org/57/210557/1/check/gate-grenade-dsvm/51b2e81/logs/tempest.txt.gz#_2015-08-07_19_16_15_847

2015-08-07 19:16:15.847 21819 CRITICAL tempest [-] error: [Errno 111] Connection refused
2015-08-07 19:16:15.847 21819 ERROR tempest Traceback (most recent call last):
2015-08-07 19:16:15.847 21819 ERROR tempest File ".tox/venv/bin/verify-tempest-config", line 10, in <module>
2015-08-07 19:16:15.847 21819 ERROR tempest sys.exit(main())
2015-08-07 19:16:15.847 21819 ERROR tempest File "/opt/stack/new/tempest/tempest/cmd/verify_tempest_config.py", line 353, in main
2015-08-07 19:16:15.847 21819 ERROR tempest results = verify_extensions(os, service, results)
2015-08-07 19:16:15.847 21819 ERROR tempest File "/opt/stack/new/tempest/tempest/cmd/verify_tempest_config.py", line 171, in verify_extensions
2015-08-07 19:16:15.847 21819 ERROR tempest __, resp = extensions_client.list_extensions()
2015-08-07 19:16:15.847 21819 ERROR tempest File "/opt/stack/new/tempest/tempest/services/object_storage/account_client.py", line 155, in list_extensions
2015-08-07 19:16:15.847 21819 ERROR tempest resp, body = self.get('info')
2015-08-07 19:16:15.847 21819 ERROR tempest File "/opt/stack/new/tempest/.tox/venv/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 271, in get
2015-08-07 19:16:15.847 21819 ERROR tempest return self.request('GET', url, extra_headers, headers)
2015-08-07 19:16:15.847 21819 ERROR tempest File "/opt/stack/new/tempest/.tox/venv/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 630, in request
2015-08-07 19:16:15.847 21819 ERROR tempest headers=headers, body=body)
2015-08-07 19:16:15.847 21819 ERROR tempest File "/opt/stack/new/tempest/.tox/venv/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 530, in _request
2015-08-07 19:16:15.847 21819 ERROR tempest req_url, method, headers=req_headers, body=req_body)
2015-08-07 19:16:15.847 21819 ERROR tempest File "/opt/stack/new/tempest/.tox/venv/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 561, in raw_request
2015-08-07 19:16:15.847 21819 ERROR tempest headers=headers, body=body)
2015-08-07 19:16:15.847 21819 ERROR tempest File "/opt/stack/new/tempest/.tox/venv/local/lib/python2.7/site-packages/tempest_lib/common/http.py", line 25, in request
2015-08-07 19:16:15.847 21819 ERROR tempest return super(ClosingHttp, self).request(*args, **new_kwargs)
2015-08-07 19:16:15.847 21819 ERROR tempest File "/opt/stack/new/tempest/.tox/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1608, in request
2015-08-07 19:16:15.847 21819 ERROR tempest (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
2015-08-07 19:16:15.847 21819 ERROR tempest File "/opt/stack/new/tempest/.tox/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1350, in _request
2015-08-07 19:16:15.847 21819 ERROR tempest (response, content) = self._conn_request(conn, request_uri, method, body, headers)
2015-08-07 19:16:15.847 21819 ERROR tempest File "/opt/s...

[Matt Riedemann (mriedem)]

Looks like tempest fails trying to talk to swift on the kilo side:

http://logs.openstack.org/57/210557/1/check/gate-grenade-dsvm/51b2e81/logs/new/screen-s-proxy.txt.gz

[Matt Riedemann (mriedem)]

Traceback (most recent call last):
  File "/opt/stack/new/swift/bin/swift-proxy-server", line 23, in <module>
    sys.exit(run_wsgi(conf_file, 'proxy-server', **options))
  File "/opt/stack/new/swift/swift/common/wsgi.py", line 473, in run_wsgi
    loadapp(conf_path, global_conf=global_conf)
  File "/opt/stack/new/swift/swift/common/wsgi.py", line 382, in loadapp
    ctx = loadcontext(loadwsgi.APP, conf_file, global_conf=global_conf)
  File "/opt/stack/new/swift/swift/common/wsgi.py", line 366, in loadcontext
    global_conf=global_conf)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
    global_conf=global_conf)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
    return loader.get_context(object_type, name, global_conf)
  File "/opt/stack/new/swift/swift/common/wsgi.py", line 62, in get_context
    object_type, name=name, global_conf=global_conf)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 450, in get_context
    global_additions=global_additions)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 562, in _pipeline_app_context
    for name in pipeline[:-1]]
  File "/opt/stack/new/swift/swift/common/wsgi.py", line 62, in get_context
    object_type, name=name, global_conf=global_conf)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 454, in get_context
    section)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 476, in _context_from_use
    object_type, name=use, global_conf=global_conf)
  File "/opt/stack/new/swift/swift/common/wsgi.py", line 62, in get_context
    object_type, name=name, global_conf=global_conf)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 406, in get_context
    global_conf=global_conf)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
    global_conf=global_conf)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg
    return loader.get_context(object_type, name, global_conf)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 620, in get_context
    object_type, name=name)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 640, in find_egg_entry_point
    pkg_resources.require(self.spec)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 952, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 844, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (oslo.serialization 1.4.0 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('oslo.serialization<=1.2.0,>=1.0.0'), set(['python-neutronclient']))

[Matt Riedemann (mriedem)]

python-neutronclient 2.3.12 was released on 8/4 and we start seeing this failure after that:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwicGtnX3Jlc291cmNlcy5Db250ZXh0dWFsVmVyc2lvbkNvbmZsaWN0OiAob3Nsby5zZXJpYWxpemF0aW9uIDEuNC4wICgvdXNyL2xvY2FsL2xpYi9weXRob24yLjcvZGlzdC1wYWNrYWdlcyksIFJlcXVpcmVtZW50LnBhcnNlKCdvc2xvLnNlcmlhbGl6YXRpb248PTEuMi4wLD49MS4wLjAnKSwgc2V0KFsncHl0aG9uLW5ldXRyb25jbGllbnQnXSkpXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0Mzg5Nzc3NzAwODB9

[Matt Riedemann (mriedem)]

This is what introduced the failure:

https://github.com/openstack/python-neutronclient/commit/3f03a3bdb87782930eaf236f726f9bed7229e441

[Matt Riedemann (mriedem)]

OK, the problem is that neutronclient doesn't get updated on the new kilo side of grenade past 2.3.12 because it satisfies the requirement:

https://github.com/openstack/requirements/blob/stable/kilo/global-requirements.txt#L132

python-neutronclient>=2.3.11,<2.5.0

But since neutronclient 2.3.12 caps things for juno, we can't use it on kilo due to the conflict and then kaboom.

[Adam Gandelman (gandelman-a)]

I think this is a requirements bug not necessarily a neutronclient bug.

python-neutronclient has an overlapping range between juno and kilo:

stable/juno: python-neutronclient>=2.3.6,<2.4.0
stable/kilo: python-neutronclient>=2.3.11,<2.5.0

In this case, keystonemiddleware is upgraded to pulls in a newer oslo.serialization, but that offends stable/juno's oslo.serialization>=1.0.0,<=1.2.0 (which python-neutronclient 2.3.12 correctly includes in its requirements.txt)

TBH I dont' really remember the consensus on the overlaps. I thought we added those specifically to ensure grenade upgrades work with the client libraries held back, but looking at the current state of g-r it looks they are not consistent across all of the client libs.

[Matt Riedemann (mriedem)]

Proposed workaround here: https://review.openstack.org/#/c/210870/

[Ihar Hrachyshka (ihar-hrachyshka)]

Is it fixed now?

[Akihiro Motoki (amotoki)]

neutronclient requirement is changed to >=2.4.0 inhttps://review.openstack.org/#/c/211174/

[Akihiro Motoki (amotoki)]

the correct link is https://review.openstack.org/#/c/211174/