[SRU] Requests to https server can yield WantReadError

Bug #1157864 reported by Stuart McLaren on 2013-03-20
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
python-glanceclient
Critical
Stuart McLaren
ubuntu-cloud-archive
Undecided
Unassigned
python-glanceclient (Ubuntu)
Critical
Unassigned
Raring
Critical
Adam Gandelman

Bug Description

If the glance client is instantiated when the socket module has been monkey patched (eg when inside nova/cinder) requests to the server can yield a WantReadError because the socket has been set to non-blocking but this is not being handled correctly.

--- Ubuntu SRU Justification ---

[Impact]

When the socket module has been monkey patched, glanceclient
will throw WantReadError exceptions when communicating with a HTTPS glance endpoint. This patching happens in the case of Nova (at least currently in Grizzly), and its attempt to use the glanceclient library for communicating with a SSL secured remote glance endpoint ends in error.

[Test Case]

Setup a glance server configured for serving via HTTPS and set its Keystone endpoint accordingly. Observe image-related API requests to Nova failing and WantReadError exceptions errors/traceabacks logged to the relevant nova API log.

[Regression Potential]

Should be minimal. The upstream fix has gone through code review and CI testing before merging.

Fix proposed to branch: master
Review: https://review.openstack.org/24916

Changed in python-glanceclient:
assignee: nobody → Stuart McLaren (stuart-mclaren)
status: New → In Progress

Just started hitting this today, specifically in communication between nova-api-os-compute + glance.

Changed in python-glanceclient (Ubuntu):
status: New → Confirmed
importance: Undecided → High
Sina Sadeghi (sina-sa) wrote :

Yes I also ran into this bug this morning.

tags: added: cloud-archive
Stuart McLaren (stuart-mclaren) wrote :

Adam, Sina,

I've proposed a patch here:

https://review.openstack.org/#/c/24916/

If possible, it would be great to know if this works for you.

Sina Sadeghi (sina-sa) wrote :

Hi Stuart,

I deployed your patch earlier today and it has let me move forward in testing the grizzly upgrade path.

Thanks very much.

Stuart McLaren (stuart-mclaren) wrote :

Hi Sina,

Great -- many thanks for letting me know.

Chuck Short (zulcss) on 2013-04-04
Changed in python-glanceclient (Ubuntu):
status: Confirmed → In Progress
Sina Sadeghi (sina-sa) wrote :

Still not in Ubuntu Cloud Archive.

Changed in python-glanceclient:
status: In Progress → Confirmed
Changed in python-glanceclient (Ubuntu):
status: In Progress → Confirmed
Sina Sadeghi (sina-sa) wrote :

Guys, I just upgraded to the latest grizzly today and it overwrote this patch, which I had assumed would simply make it into stable. Now because of this, glance with SSL no longer works. Please mark this bug as critical?

Adam Gandelman (gandelman-a) wrote :

Sina-

The propose patch did not get accepted for merging into python-glanceclient and has since been abandoned/expired. The issue is still unresolved upstream so you should not expect a fix to make its way out via Ubuntu or the Ubuntu Cloud Archive until the bug has been addressed in the upstream project.

It seems there was disagreement as to whether this should be fixed the client library or the core project itself (glance, nova, etc). If it ends up being resolved in nova, you can expect a fix to make its way out to respective packages via an Ubuntu SRU based on the upstream stable/grizzly branch. If a fix makes its way into the client library, we should consider back-porting a patch to the current client version in 13.04 and grizzly Cloud Archive pocket, since client libraries have no stable upstream branches.

Changed in python-glanceclient (Ubuntu):
importance: High → Critical
Sina Sadeghi (sina-sa) on 2013-04-15
description: updated
tags: added: ops
Changed in python-glanceclient:
importance: Undecided → Critical

All,

Mark and I had a great chat at the summit and quickly agreed a way forward (tweak https://review.openstack.org/#/c/24916/ to only use eventlet if the socket has actually been monkey patched).

I hope to push up a new patch real soon.

Changed in python-glanceclient:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/24916
Committed: http://github.com/openstack/python-glanceclient/commit/2f33f5f283d921ce1db595a89c36eefb8b8fcd29
Submitter: Jenkins
Branch: master

commit 2f33f5f283d921ce1db595a89c36eefb8b8fcd29
Author: Stuart McLaren <email address hidden>
Date: Wed Mar 20 18:00:39 2013 +0000

    Prevent WantReadError when using https

    If the glance client is instantiated when the socket module has been
    monkey patched requests to the server can yield a WantReadError because
    the socket has been set to non-blocking but this is not being handled
    correctly. When this is the case use eventlet's GreenConnection which
    handles non-blocking sockets correctly.

    Also, for now, add a required getsockopt method to GreenConnection.
    This can be removed once the eventlet fix
    (https://bitbucket.org/eventlet/eventlet/commits/609f230) lands.

    Fixes bug 1157864.

    Change-Id: I187b69f75b8bcfe16facd41e69b1cd0490dae605

Changed in python-glanceclient:
status: In Progress → Fix Committed
Changed in cloud-archive:
status: New → Confirmed

Fixed in upload to saucy, python-glanceclient 1:0.9.0.13.gaddd3c3-0ubuntu1

Changed in python-glanceclient (Ubuntu):
status: Confirmed → Fix Released
Changed in python-glanceclient (Ubuntu Raring):
status: New → Confirmed
importance: Undecided → Critical
summary: - Requests to https server can yield WantReadError
+ [SRU] Requests to https server can yield WantReadError
description: updated
Changed in python-glanceclient (Ubuntu Raring):
assignee: nobody → Adam Gandelman (gandelman-a)
status: Confirmed → In Progress

Hello Stuart, or anyone else affected,

Accepted python-glanceclient into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-glanceclient/1:0.9.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-glanceclient (Ubuntu Raring):
status: In Progress → Fix Committed
tags: added: verification-needed

Hi Dave,

Just verified both monkey patched and non-monkey patched cases -- package looks good to me.

$ dpkg -l |grep glanceclient
ii python-glanceclient 1:0.9.0-0ubuntu1.1 Client library for Openstack glance server.

tags: added: verification-done
removed: verification-needed

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-glanceclient - 1:0.9.0-0ubuntu1.1

---------------
python-glanceclient (1:0.9.0-0ubuntu1.1) raring-proposed; urgency=low

  * debian/patches/prevent-wantreaderror-https.patch: Prevent WantReadError
    on HTTPS connections when socket has been monkey patched. (LP: #1157864)
 -- Adam Gandelman <email address hidden> Mon, 29 Apr 2013 14:29:07 -0700

Changed in python-glanceclient (Ubuntu Raring):
status: Fix Committed → Fix Released
James Page (james-page) on 2013-05-22
Changed in cloud-archive:
status: Confirmed → Fix Committed
Changed in cloud-archive:
status: Fix Committed → Fix Released
Tom Fifield (fifieldt) on 2014-05-27
Changed in python-glanceclient:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers