OpenStack Keystone Charm

tox -e pep8 not working since requests version conflict with charm-tools

Bug #1836119 reported by Joe Guo on 2019-07-10

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Keystone Charm	Fix Released	Undecided	Joe Guo

Bug Description

In `charm-keystone/test-requirements.txt`:

charm-tools>=2.4.4
requests>=2.18.4

In `charm-tools/requirements.txt`:

requests<=2.20.1

When we run `tox -e pep8` in `charm-keystone`, top level requirement `requests>=2.18.4` was used, and latest requests(2.22.0) was installed.

When `charm-tools` was called by `tox`, it failed with such error:

```
pep8 run-test: commands[1] | charm-proof
Traceback (most recent call last):
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (requests 2.22.0 (/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages), Requirement.parse('requests<=2.20.1'), {'charm-tools'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/bin/charm-proof", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3241, in <module>
    @_call_aside
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3225, in _call_aside
    f(*args, **kwargs)
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'requests<=2.20.1' distribution was not found and is required by charm-tools
ERROR: InvocationError for command /home/guoqiao/git/charm-keystone/.tox/pep8/bin/charm-proof (exited with code 1)
_____________________________ summary _____________________________
ERROR: pep8: commands failed

```

ENV: ubuntu 18.04

Suggested fix:
- remove requests requirement in `charm-keystone/test-requirements.txt` and let `charm-tools` decide
- perhaps update `charm-tools/requirements.txt` to use latest `requests` if there is no special reason to limit it to old version

Tags:

Joe Guo (guoqiao) on 2019-07-10

Changed in charm-keystone:
assignee:	nobody → guoqiao (guoqiao)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-11: Fix proposed to charm-keystone (master)

Fix proposed to branch: master
Review: https://review.opendev.org/670216

Changed in charm-keystone:
status:	New → In Progress

Revision history for this message

David Ames (thedac) wrote on 2019-07-17:

Joe,

I have been unable to re-create this problem. Can you provide the output of the failure without your change?

Revision history for this message

Joe Guo (guoqiao) wrote on 2019-07-17:

Download full text (6.4 KiB)

hi David,

This is because after I created this bug, I went to github/charm-tools and created related issue/PR immediately:

https://github.com/juju/charm-tools/issues/535
https://github.com/juju/charm-tools/pull/536

The PR was merged quickly and released to charm-tools 2.6.1.

To reproduce this issue, if you edit `charm-keystone/test-requirements.txt`:

charm-tools>=2.4.4 --> charm-tools==2.6.0

And test again:

rm -r .tox/pep8
tox -e pep8

This is my output:

guoqiao@t480 ~/git/charm-keystone [9:11:14]
> $ rm -r .tox/pep8 [±master ●]

guoqiao@t480 ~/git/charm-keystone [9:11:19]
> $ tox -e pep8 [±master ●]
pep8 create: /home/guoqiao/git/charm-keystone/.tox/pep8
pep8 installdeps: -r/home/guoqiao/git/charm-keystone/requirements.txt, -r/home/guoqiao/git/charm-keystone/test-requirements.txt
pep8 installed: appdirs==1.4.3,asn1crypto==0.24.0,async-generator==1.10,attrs==19.1.0,Babel==2.7.0,bcrypt==3.1.7,blessings==1.6,certifi==2019.6.16,cffi==1.12.3,chardet==3.0.4,charm-tools==2.6.0,Cheetah3==3.2.3,cliff==2.15.0,cmd2==0.9.14,colander==1.7.0,colorama==0.4.1,coverage==4.5.3,cryptography==2.7,debtcollector==1.21.0,decorator==4.4.0,dict2colander==0.2,distro==1.4.0,dnspython==1.16.0,dogpile.cache==0.7.1,entrypoints==0.3,extras==1.0.0,fixtures==3.0.0,flake8==2.4.1,future==0.17.1,httplib2==0.13.0,hvac==0.6.4,idna==2.8,importlib-metadata==0.18,iso8601==0.1.12,Jinja2==2.10.1,jmespath==0.9.4,jsonpatch==1.23,jsonpointer==2.0,jsonschema==2.5.1,juju==0.11.7,juju-wait==2.7.0,jujubundlelib==0.5.6,keyring==19.0.2,keystoneauth1==3.15.0,launchpadlib==1.10.7,lazr.restfulclient==0.14.2,lazr.uri==1.0.3,libcharmstore==0.0.9,linecache2==1.0.0,macaroonbakery==1.2.3,MarkupSafe==1.1.1,mccabe==0.3.1,mock==3.0.5,msgpack==0.6.1,munch==2.3.2,netaddr==0.7.19,netifaces==0.10.9,oauthlib==3.0.2,openstacksdk==0.31.1,os-client-config==1.32.0,os-service-types==1.7.0,osc-lib==1.13.0,oslo.config==6.11.0,oslo.context==2.22.1,oslo.i18n==3.23.1,oslo.log==3.44.0,oslo.serialization==2.29.1,oslo.utils==3.41.0,otherstuf==1.1.0,paramiko==2.6.0,parse==1.12.0,path.py==12.0.1,pathspec==0.3.4,pbr==1.8.1,pep8==1.7.1,prettytable==0.7.2,protobuf==3.9.0,psutil==1.2.1,pyasn1==0.4.5,pycparser==2.19,pyflakes==0.8.1,pyinotify==0.9.6,pymacaroons==0.13.0,PyNaCl==1.3.0,pyOpenSSL==19.0.0,pyparsing==2.4.0,pyperclip==1.7.0,pyRFC3339==1.1,python-cinderclient==4.2.1,python-dateutil==2.8.0,python-glanceclient==2.16.0,python-keystoneclient==3.20.0,python-mimeparse==1.6.0,python-neutronclient==6.12.0,python-novaclient==14.2.0,python-octaviaclient==1.9.0,python-openstackclient==3.19.0,python-subunit==1.3.0,python-swiftclient==3.8.0,pytz==2019.1,PyYAML==3.13,requests==2.22.0,...

hi David,

This is because after I created this bug, I went to github/charm-tools and created related issue/PR immediately:

https://github.com/juju/charm-tools/issues/535
https://github.com/juju/charm-tools/pull/536

The PR was merged quickly and released to charm-tools 2.6.1.

To reproduce this issue, if you edit `charm-keystone/test-requirements.txt`:

charm-tools>=2.4.4 --> charm-tools==2.6.0

And test again:
    
    rm -r .tox/pep8
    tox -e pep8

This is my output:

guoqiao@t480 ~/git/charm-keystone                                                                                             [9:11:14] 
> $ rm -r .tox/pep8                                                                                                         [±master ●]
                                                                                                                                        
guoqiao@t480 ~/git/charm-keystone                                                                                             [9:11:19] 
> $ tox -e pep8                                                                                                             [±master ●]
pep8 create: /home/guoqiao/git/charm-keystone/.tox/pep8
pep8 installdeps: -r/home/guoqiao/git/charm-keystone/requirements.txt, -r/home/guoqiao/git/charm-keystone/test-requirements.txt
pep8 installed: appdirs==1.4.3,asn1crypto==0.24.0,async-generator==1.10,attrs==19.1.0,Babel==2.7.0,bcrypt==3.1.7,blessings==1.6,certifi==2019.6.16,cffi==1.12.3,chardet==3.0.4,charm-tools==2.6.0,Cheetah3==3.2.3,cliff==2.15.0,cmd2==0.9.14,colander==1.7.0,colorama==0.4.1,coverage==4.5.3,cryptography==2.7,debtcollector==1.21.0,decorator==4.4.0,dict2colander==0.2,distro==1.4.0,dnspython==1.16.0,dogpile.cache==0.7.1,entrypoints==0.3,extras==1.0.0,fixtures==3.0.0,flake8==2.4.1,future==0.17.1,httplib2==0.13.0,hvac==0.6.4,idna==2.8,importlib-metadata==0.18,iso8601==0.1.12,Jinja2==2.10.1,jmespath==0.9.4,jsonpatch==1.23,jsonpointer==2.0,jsonschema==2.5.1,juju==0.11.7,juju-wait==2.7.0,jujubundlelib==0.5.6,keyring==19.0.2,keystoneauth1==3.15.0,launchpadlib==1.10.7,lazr.restfulclient==0.14.2,lazr.uri==1.0.3,libcharmstore==0.0.9,linecache2==1.0.0,macaroonbakery==1.2.3,MarkupSafe==1.1.1,mccabe==0.3.1,mock==3.0.5,msgpack==0.6.1,munch==2.3.2,netaddr==0.7.19,netifaces==0.10.9,oauthlib==3.0.2,openstacksdk==0.31.1,os-client-config==1.32.0,os-service-types==1.7.0,osc-lib==1.13.0,oslo.config==6.11.0,oslo.context==2.22.1,oslo.i18n==3.23.1,oslo.log==3.44.0,oslo.serialization==2.29.1,oslo.utils==3.41.0,otherstuf==1.1.0,paramiko==2.6.0,parse==1.12.0,path.py==12.0.1,pathspec==0.3.4,pbr==1.8.1,pep8==1.7.1,prettytable==0.7.2,protobuf==3.9.0,psutil==1.2.1,pyasn1==0.4.5,pycparser==2.19,pyflakes==0.8.1,pyinotify==0.9.6,pymacaroons==0.13.0,PyNaCl==1.3.0,pyOpenSSL==19.0.0,pyparsing==2.4.0,pyperclip==1.7.0,pyRFC3339==1.1,python-cinderclient==4.2.1,python-dateutil==2.8.0,python-glanceclient==2.16.0,python-keystoneclient==3.20.0,python-mimeparse==1.6.0,python-neutronclient==6.12.0,python-novaclient==14.2.0,python-octaviaclient==1.9.0,python-openstackclient==3.19.0,python-subunit==1.3.0,python-swiftclient==3.8.0,pytz==2019.1,PyYAML==3.13,requests==2.22.0,requestsexceptions==1.4.0,rfc3986==1.3.2,ruamel.yaml==0.15.100,SecretStorage==2.3.1,simplejson==3.16.0,six==1.12.0,stestr==2.3.1,stevedore==1.30.1,stuf==0.9.16,tenacity==5.0.4,testresources==2.0.1,testtools==2.3.0,theblues==0.5.1,traceback2==1.4.0,translationstring==1.3,unittest2==1.1.0,urllib3==1.25.3,vergit==1.0.2,virtualenv==16.6.2,voluptuous==0.11.5,wadllib==1.3.3,warlock==1.3.3,wcwidth==0.1.7,websockets==7.0,wrapt==1.11.2,zaza==0.0.2.dev1,zaza.openstack==0.0.1.dev1,zipp==0.5.2
pep8 run-test-pre: PYTHONHASHSEED='0'
pep8 run-test: commands[0] | flake8 hooks unit_tests tests actions lib
/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/flake8/engine.py:33: PkgResourcesDeprecationWarning: Parameters to load are deprecated.  Call .resolve and .require separately.
  checker = entry.load(require=False)
pep8 run-test: commands[1] | charm-proof
Traceback (most recent call last):
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (requests 2.22.0 (/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages), Requirement.parse('requests<=2.20.1'), {'charm-tools'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/bin/charm-proof", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3241, in <module>
    @_call_aside
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3225, in _call_aside
    f(*args, **kwargs)
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/home/guoqiao/git/charm-keystone/.tox/pep8/lib/python3.6/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'requests<=2.20.1' distribution was not found and is required by charm-tools
ERROR: InvocationError for command /home/guoqiao/git/charm-keystone/.tox/pep8/bin/charm-proof (exited with code 1)
_______________________________________________________________ summary ________________________________________________________________
ERROR:   pep8: commands failed

Revision history for this message

Joe Guo (guoqiao) wrote on 2019-07-17:

The root reason for this issue was on charm-tools side, which limited requests to old version.
When I encountered it, I was not sure how long will it take for charm-tools to fix it, or can I fix.
To stop it being broken, I decided to fix both repos in parallel.

Since charm-tools have been fixed and released quickly, so this issue is gone, and yes, this bug is unnecessary any more. But it still avoids potential version conflict in future. Either accept or deny it has good reason.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-18: Change abandoned on charm-keystone (master)

Change abandoned by David Ames (<email address hidden>) on branch: master
Review: https://review.opendev.org/670216
Reason: Fixed in upstream charm-tools