[SRU] Requests to https server can yield WantReadError

[Stuart McLaren]

If the glance client is instantiated when the socket module has been monkey patched (eg when inside nova/cinder) requests to the server can yield a WantReadError because the socket has been set to non-blocking but this is not being handled correctly.

--- Ubuntu SRU Justification ---

[Impact]

When the socket module has been monkey patched, glanceclient
will throw WantReadError exceptions when communicating with a HTTPS glance endpoint. This patching happens in the case of Nova (at least currently in Grizzly), and its attempt to use the glanceclient library for communicating with a SSL secured remote glance endpoint ends in error.

[Test Case]

Setup a glance server configured for serving via HTTPS and set its Keystone endpoint accordingly. Observe image-related API requests to Nova failing and WantReadError exceptions errors/traceabacks logged to the relevant nova API log.

[Regression Potential]

Should be minimal. The upstream fix has gone through code review and CI testing before merging.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/24916

[Adam Gandelman (gandelman-a)]

Just started hitting this today, specifically in communication between nova-api-os-compute + glance.

[Sina Sadeghi (sina-sa)]

Yes I also ran into this bug this morning.

[Stuart McLaren (stuart-mclaren)]

Adam, Sina,

I've proposed a patch here:

https://review.openstack.org/#/c/24916/

If possible, it would be great to know if this works for you.

[Sina Sadeghi (sina-sa)]

Hi Stuart,

I deployed your patch earlier today and it has let me move forward in testing the grizzly upgrade path.

Thanks very much.

[Stuart McLaren (stuart-mclaren)]

Hi Sina,

Great -- many thanks for letting me know.

[Sina Sadeghi (sina-sa)]

Still not in Ubuntu Cloud Archive.

[Sina Sadeghi (sina-sa)]

Guys, I just upgraded to the latest grizzly today and it overwrote this patch, which I had assumed would simply make it into stable. Now because of this, glance with SSL no longer works. Please mark this bug as critical?

[Adam Gandelman (gandelman-a)]

Sina-

The propose patch did not get accepted for merging into python-glanceclient and has since been abandoned/expired. The issue is still unresolved upstream so you should not expect a fix to make its way out via Ubuntu or the Ubuntu Cloud Archive until the bug has been addressed in the upstream project.

It seems there was disagreement as to whether this should be fixed the client library or the core project itself (glance, nova, etc). If it ends up being resolved in nova, you can expect a fix to make its way out to respective packages via an Ubuntu SRU based on the upstream stable/grizzly branch. If a fix makes its way into the client library, we should consider back-porting a patch to the current client version in 13.04 and grizzly Cloud Archive pocket, since client libraries have no stable upstream branches.

[Stuart McLaren (stuart-mclaren)]

All,

Mark and I had a great chat at the summit and quickly agreed a way forward (tweak https://review.openstack.org/#/c/24916/ to only use eventlet if the socket has actually been monkey patched).

I hope to push up a new patch real soon.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/24916
Committed: http://github.com/openstack/python-glanceclient/commit/2f33f5f283d921ce1db595a89c36eefb8b8fcd29
Submitter: Jenkins
Branch: master

commit 2f33f5f283d921ce1db595a89c36eefb8b8fcd29
Author: Stuart McLaren <email address hidden>
Date: Wed Mar 20 18:00:39 2013 +0000

    Prevent WantReadError when using https

    If the glance client is instantiated when the socket module has been
    monkey patched requests to the server can yield a WantReadError because
    the socket has been set to non-blocking but this is not being handled
    correctly. When this is the case use eventlet's GreenConnection which
    handles non-blocking sockets correctly.

    Also, for now, add a required getsockopt method to GreenConnection.
    This can be removed once the eventlet fix
    (https://bitbucket.org/eventlet/eventlet/commits/609f230) lands.

    Fixes bug 1157864.

    Change-Id: I187b69f75b8bcfe16facd41e69b1cd0490dae605

[Adam Gandelman (gandelman-a)]

Fixed in upload to saucy, python-glanceclient 1:0.9.0.13.gaddd3c3-0ubuntu1

[Dave Walker (davewalker)]

Hello Stuart, or anyone else affected,

Accepted python-glanceclient into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-glanceclient/1:0.9.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Stuart McLaren (stuart-mclaren)]

Hi Dave,

Just verified both monkey patched and non-monkey patched cases -- package looks good to me.

$ dpkg -l |grep glanceclient
ii python-glanceclient 1:0.9.0-0ubuntu1.1 Client library for Openstack glance server.

[Adam Conrad (adconrad)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-glanceclient - 1:0.9.0-0ubuntu1.1

---------------
python-glanceclient (1:0.9.0-0ubuntu1.1) raring-proposed; urgency=low

  * debian/patches/prevent-wantreaderror-https.patch: Prevent WantReadError
    on HTTPS connections when socket has been monkey patched. (LP: #1157864)
 -- Adam Gandelman <email address hidden> Mon, 29 Apr 2013 14:29:07 -0700