swiftclient doesn't cleanup session it creates if one is not provided

[Alex Schultz]

If a session object/http connection is not provided to the swift client, the HTTPClient defined in swiftclient.client will create a session object. This session object leaks open connections because it is not properly closed when the object is no longer needed. This leads to a ResourceWarning about an unclosed socket:

sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 49772), raddr=('192.168.24.2', 13808)>

Reproducer script
$ cat swift.py
#!/usr/bin/python3 -Wd
from swiftclient import client
sc = client.head_object(url='https://192.168.24.2:13808/v1/AUTH_d25c0720341c499f981d4ddbcd2b6354',
                       container='tripleo-validations',
                       name='512e.yaml',
                       token='<my magical token>')
print(sc)

Here is the output
$ ./swift.py
/usr/lib64/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3.6/site-packages/repoze: missing __init__
  _warnings.warn(msg.format(portions[0]), ImportWarning)
/usr/lib64/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3.6/site-packages/paste: missing __init__
  _warnings.warn(msg.format(portions[0]), ImportWarning)
/usr/lib/python3.6/site-packages/pytz/__init__.py:499: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/share/zoneinfo/zone.tab' mode='r' encoding='UTF-8'>
  for l in open(os.path.join(_tzinfo_dir, 'zone.tab'))
/usr/lib/python3.6/site-packages/eventlet/patcher.py:1: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
{'content-length': '342', 'accept-ranges': 'bytes', 'last-modified': 'Thu, 01 Aug 2019 16:23:19 GMT', 'etag': 'bf113fde97403b7faf0c563cb79111e2', 'x-timestamp': '1564676598.16610', 'content-type': 'application/octet-stream', 'x-trans-id': 'tx399233867c224d0ea0317-005d44453d', 'x-openstack-request-id': 'tx399233867c224d0ea0317-005d44453d', 'date': 'Fri, 02 Aug 2019 14:14:21 GMT'}
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 51638), raddr=('192.168.24.2', 13808)>

[Alex Schultz (alex-schultz)]

https://review.opendev.org/#/c/674320/

[clayg (clay-gerrard)]

I'm not seeing the same warning, it's a bit of a strange issue [1].

What version of swiftclient & requests are you using? Do you know if the issue is limited to SSL?

1. https://github.com/psf/requests/issues/3912

[Alex Schultz (alex-schultz)]

$ rpm -qa | grep -E '(swift|requests)'
python3-swiftclient-3.8.0-0.20190718063408.47d5f44.el8ost.noarch
python3-requests-2.20.0-1.el8.noarch
puppet-swift-15.2.0-0.20190724162913.75bdc81.el8ost.noarch
python3-requestsexceptions-1.4.0-0.20190423131341.d7ac0ff.el8ost.noarch

[Alex Schultz (alex-schultz)]

I'll try w/o ssl but if you see the patch, the swift client isn't properly closing the session object when it creates one.

[Alex Schultz (alex-schultz)]

It is not ssl specific.

$ ./swift.py
/usr/lib64/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3.6/site-packages/repoze: missing __init__
  _warnings.warn(msg.format(portions[0]), ImportWarning)
/usr/lib64/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/lib/python3.6/site-packages/paste: missing __init__
  _warnings.warn(msg.format(portions[0]), ImportWarning)
/usr/lib/python3.6/site-packages/pytz/__init__.py:499: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/share/zoneinfo/zone.tab' mode='r' encoding='UTF-8'>
  for l in open(os.path.join(_tzinfo_dir, 'zone.tab'))
/usr/lib/python3.6/site-packages/eventlet/patcher.py:1: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
{'content-length': '342', 'accept-ranges': 'bytes', 'last-modified': 'Mon, 05 Aug 2019 21:05:52 GMT', 'etag': 'bf113fde97403b7faf0c563cb79111e2', 'x-timestamp': '1565039151.52974', 'content-type': 'application/octet-stream', 'x-trans-id': 'tx6d54556fd1d247e9914bd-005d48b924', 'x-openstack-request-id': 'tx6d54556fd1d247e9914bd-005d48b924', 'date': 'Mon, 05 Aug 2019 23:17:56 GMT'}
sys:1: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.1', 47004), raddr=('192.168.24.1', 8080)>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/674320
Committed: https://git.openstack.org/cgit/openstack/python-swiftclient/commit/?id=1f26c5736949e1c3b57c024a315e33fc419f126e
Submitter: Zuul
Branch: master

commit 1f26c5736949e1c3b57c024a315e33fc419f126e
Author: Alex Schultz <email address hidden>
Date: Fri Aug 2 08:20:56 2019 -0600

    Cleanup session on delete

    If an external http connection was not passed into the client, we
    create one with a requests.Session() on our own. Once this is used,
    it may still have an open socket when the connection is closed. We need
    to handle the closing of the requests.Session() ourselves if we created
    one. If you do not close it, a ResourceWarning may be reported about the
    socket that is left open.

    Change-Id: I200ad0cdc8b7999c3f5521b9a822122bd18714bf
    Closes-Bug: #1838775

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-swiftclient 3.9.0 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/736401

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/736401
Committed: https://git.openstack.org/cgit/openstack/python-swiftclient/commit/?id=0c70d164ba52d76a6dbbbe8765d15fb969fc07ff
Submitter: Zuul
Branch: master

commit 0c70d164ba52d76a6dbbbe8765d15fb969fc07ff
Author: Tim Burke <email address hidden>
Date: Wed Jun 17 15:44:22 2020 -0700

    (Mostly) revert "Cleanup session on delete"

    This reverts commit 1f26c5736949e1c3b57c024a315e33fc419f126e for py2.
    Apparently the existence of the __del__ method on Python 2 prevents us
    from cleaning up all file descriptors.

    Change-Id: Id6cff5dd7b9faf9c4240c0cb26b74d05ed37da5b
    Closes-Bug: #1873435
    Related-Bug: #1838775

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/721051
Committed: https://git.openstack.org/cgit/openstack/python-swiftclient/commit/?id=97aa3e65412ee241fce7721927b0b003daf51ed4
Submitter: Zuul
Branch: master

commit 97aa3e65412ee241fce7721927b0b003daf51ed4
Author: Tim Burke <email address hidden>
Date: Sat Apr 18 22:41:55 2020 -0700

    Close connections created when calling module-level functions

    Co-Authored-By: Clay Gerrard <email address hidden>
    Change-Id: Id62e63afc6f2ffa32eb8640787c78559481050f9
    Related-Change: I200ad0cdc8b7999c3f5521b9a822122bd18714bf
    Related-Bug: #1873435
    Closes-Bug: #1838775

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-swiftclient 3.11.0 release.