In compute node, KeyError: u'glance+https' is encountered

[Jun Hong Li]

Using https configuration, when booting a VM in Hyper-V compute node, the following KeyError: u'glance+https' problem is encountered:

2014-11-24 11:00:42.934 2100 TRACE nova.compute.manager [instance: bbf4a13d-ce69-426c-82bd-7f1aabc26006] pool_cls = pool_classes_by_scheme[scheme]
2014-11-24 11:00:42.934 2100 TRACE nova.compute.manager [instance: bbf4a13d-ce69-426c-82bd-7f1aabc26006] KeyError: u'glance+https'

[Jun Hong Li (junhongl)]

The problem is that we register the "glance+https" schema into requests.packages.urllib3.poolmanager, however in http.py, the session will use urllib3.poolmanager to get the connection, which leads to this KeyError problem

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/137748

[Andrea Rosa (andrea-rosa-m)]

I am seeing this issue even with a KVM compute node. Please can you amend the title to make the bug description more general?
Thanks

[Erno Kuvaja (jokke)]

"""
The problem is that we register the "glance+https" schema into requests.packages.urllib3.poolmanager, however in http.py, the session will use urllib3.poolmanager to get the connection, which leads to this KeyError problem
"""

Are you sure about this? The Session on http.py is coming from requests. Also our testing is using requests from pypi which has the embedded urllib3. If the statement above would be true, our tests should fail, no?

[Ian Cordasco (icordasc)]

So our testing only ever uses requests as installed from PyPI which means we don't run our tests with system requests/urllib3 (like a lot of people deploying OpenStack use). Here's the real problem and the real way of fixing it is in a CR I'm submitting.

On Fedora and Debian, there are symlinks so the API of importing requests.packages.urllib3 "Just Works". This means that when we import urllib3 from there, we're adding a new entry to sys.modules. Both Fedora and Debian, however, both rewrite imports extensively and likely didn't stop even after adding in the symlinks. What this means is that sys.modules now has an entry for urllib3 (which is what requests is using internally) and requests.packages.urllib3 (which is what we end up using regardless). This means that even if we register glance+https, we will never see it made available.

See https://review.openstack.org/#/c/140151/ for the better solution and reasoning in the commit message.

[Jun Hong Li (junhongl)]

jokke,

I checked the codes, the glanceclient.https.HTTPSAdapter inherits adapters.HTTPAdapter, and in adapters.HTTPAdapter.__init__, it calls init_poolmanager to initialize the poolmanager, where it hard codes to use urllib3.poolmanager.

Therefore, i think, maybe we can override the init_poolmanager to avoid this bug. Your idea?

[Ian Cordasco (icordasc)]

Jun, the truth is that no solution on our end will entirely fix this. See what we're doing upstream in requests https://github.com/kennethreitz/requests/pull/2375

[Jun Hong Li (junhongl)]

Ian,

So you propose to change the order of import? I'm fine with that, but I think it actually works as import poolmanager from urllib3 directly, because as far as i know, the "requests" python package relies on "urllib3". And we used the requests.session to send the request, which implies the "urllib3" should have been installed, right?

I agree that the better solution is the fix in "requests" itself as you mentioned.

[Ian Cordasco (icordasc)]

Jun, the package is severely modified when packaged by downstream vendors. So if you do pip install requests, you won't have urllib3 installed unless you install it because we vendor it. If you do apt-get install python-requests (or the yum equivalent) you'll have far more packages installed because the distributions un-vendor urllib3 and our other dependency

[Erno Kuvaja (jokke)]

Jun,
as Ian mentioned that's packaging issue, not exactly requests/Glance issue. As we get urllib3 via requests (when it's packaged correctly) we do not have that hard dependency you're describing. I would prefer a) not depend on urllib3 directly as using vanilla requests it's not needed (this brings just extra dependency management overhead) b) not need to hack the requests either hoping that something does not break on future changes or when the package vendors decide to change something on the fly again.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jun Hong Li (<email address hidden>) on branch: master
Review: https://review.openstack.org/137748
Reason: It looks more like a packaging problem.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140151
Committed: https://git.openstack.org/cgit/openstack/python-glanceclient/commit/?id=7ee96cbe390b2492f8d837c93f33a8f5bebdb388
Submitter: Jenkins
Branch: master

commit 7ee96cbe390b2492f8d837c93f33a8f5bebdb388
Author: Ian Cordasco <email address hidden>
Date: Mon Dec 8 15:28:04 2014 -0600

    Register our own ConnectionPool without globals

    Currently, on systems like Fedora and Debian, it is possible to import
    urllib3 as well as requests.packages.urllib3. They functionally point to
    the same code but sys.modules considers them to be separate items. When
    downstream packagers unvendor urllib3 from requests, they also change
    all the imports inside of the package. So if the code imports urllib3
    from requests.packages.urllib3 and modifies globals in a submodule, that
    will not be visible to requests since it has been rewritten to use
    urllib3 (not requests.packages.urllib3). By handling this logic
    ourselves, we can issue a release until upstream packages and requests
    can fix this and cut a new release.

    Change-Id: Ic77ce8a06d9d148a899b4b8695990fca8fdaefc5
    Closes-bug: 1396550