https: client can cause nova/cinder to leak sockets for 'get' 'show' 'delete' 'update'

[Stuart McLaren]

    Other OpenStack services which instantiate a 'https' glanceclient using
    ssl_compression=False and insecure=False (eg Nova, Cinder) are leaking
    sockets due to glanceclient not closing the connection to the Glance
    server.

    This could happen for a sub-set of calls, eg 'show', 'delete', 'update'.

    netstat -nopd would show the sockets would hang around forever:

    ... 127.0.0.1:9292 ESTABLISHED 9552/python off (0.00/0/0)

    urllib's ConnectionPool relies on the garbage collector to tear down
    sockets which are no longer in use. The 'verify_callback' function used to
    validate SSL certs was holding a reference to the VerifiedHTTPSConnection
    instance which prevented the sockets being torn down.

------------------

to reproduce, set up devstack with nova talking to glance over https (must be performing full cert verification) and
perform a nova operation such as:

 $ nova image-meta 53854ea3-23ed-4682-abf7-8415f2d6b7d9 set foo=bar

you will see connections from nova to glance which have no timeout (off):

 $ netstat -nopd | grep 9292

 tcp 0 0 127.0.0.1:34204 127.0.0.1:9292 ESTABLISHED 9552/python off (0.00/0/0)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/156975

[Stuart McLaren (stuart-mclaren)]

Note: eventually nova would no longer be able to service requests ('too many open file descriptors').

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/156975
Committed: https://git.openstack.org/cgit/openstack/python-glanceclient/commit/?id=ef9fd9fca05f8da8325ccaa6632e34d1321130bf
Submitter: Jenkins
Branch: master

commit ef9fd9fca05f8da8325ccaa6632e34d1321130bf
Author: Stuart McLaren <email address hidden>
Date: Tue Feb 17 17:36:56 2015 +0000

    https: Prevent leaking sockets for some operations

    Other OpenStack services which instantiate a 'https' glanceclient using
    ssl_compression=False and insecure=False (eg Nova, Cinder) are leaking
    sockets due to glanceclient not closing the connection to the Glance
    server.

    This could happen for a sub-set of calls, eg 'show', 'delete', 'update'.

    netstat -nopd would show the sockets would hang around forever:

    ... 127.0.0.1:9292 ESTABLISHED 9552/python off (0.00/0/0)

    urllib's ConnectionPool relies on the garbage collector to tear down
    sockets which are no longer in use. The 'verify_callback' function used to
    validate SSL certs was holding a reference to the VerifiedHTTPSConnection
    instance which prevented the sockets being torn down.

    Change-Id: Idb3e68151c48ed623ab89d05d88ea48465429838
    Closes-bug: 1423165

[Dr. Jens Harbott (j-harbott)]

Thanks for the fast release. Sadly this doesn't help us for our icehouse deployment, as there the client version is pinned to 0.14.2. Is there a possibility to do a 0.14.3 release that would include this bugfix?

[Dr. Jens Harbott (j-harbott)]

Nova stable/juno is still affected by this issue, since the fix is not available there currently due to the version cap on python-glanceclient.

[Dr. Jens Harbott (j-harbott)]

Cinder stable/juno is still affected by this issue, since the fix is not available there currently due to the version cap on python-glanceclient.

[Dr. Jens Harbott (j-harbott)]

Davanum: Can you explain why you consider this bug invalid for nova? It causes nova-api-os-compute daemon to run out of file descriptors after some time, leading to failing services with very obscure errors. Same for cinder-volume and other services, though at a lower rate at least in our setup.

[Ian Cordasco (icordasc)]

Jens, likely because glanceclient has been released with the fix and the problem isn't actually in Nova.

[Dr. Jens Harbott (j-harbott)]

O.k., but as I tried to explain above, the fixed glanceclient cannot to be used for nova in stable/juno due to version caps. The cap also cannot be easily lifted due to the Oslo namespace changes. And in the end it still is most of my nova daemons dying regularly, so for me there still is some unsolved bug here. If one could add python-glanceclient with branch stable/juno as affected and set that to Unresolved, I'd be fine with that, but as long a there is not such branch, this IMHO is a bug in nova stable/juno.

[Joe Gordon (jogo)]

I agree this is a bug, but its not a nova bug per se.

It sounds like this is a glanceclient issue / issue in https://github.com/openstack/requirements/tree/stable/juno

It sounds like the right answer here is to create a stable/juno glanceclient branch

[John Griffith (john-griffith)]

Going to close it for Cinder as well, as I don't know of a way to fix a broken glanceclient from the consumer end.

If you're interested however I did throw together a patched version of 0.14.2 here:
https://github.com/j-griffith/python-glanceclient/tree/stable/icehouse

Maybe you or somebody else could test it out, and we could convince the glance folks to push a branch for it; or people that need it can maybe just use it.

Thanks