Established connection is never closed on Python 2.7

[Takashi Kajinami]

When we use python2.7 and python-swiftclient-3.9.0, we observe that an connection established during a request is not closed after request method like get_account, get_container, and etc. returns.

For example if you run 5 swiftclient.get_account
~~~
>>> import swiftclient
>>> url, token = swiftclient.get_auth(auth_url, ptoject + ':' + user, password, os_options={'user_domain_name': 'Default', 'project_name': ptojrvy}, auth_version=3)\
>>> swiftclient.get_account(url, token)
>>> swiftclient.get_account(url, token)
>>> swiftclient.get_account(url, token)
>>> swiftclient.get_account(url, token)
>>> swiftclient.get_account(url, token)
~~~

You see that the process still has 5 TCP connection in ESTABLISHED status.
~~~
vagrant@ubuntu1804:~$ lsof -p 16245
...
      Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 16245 vagrant cwd DIR 8,3 4096 400663 /home/vagrant/storlets
...
python 16245 vagrant 3u IPv4 2193105 0t0 TCP localhost:52412->localhost:http-alt (ESTABLISHED)
python 16245 vagrant 4u IPv4 2195534 0t0 TCP localhost:52800->localhost:http-alt (ESTABLISHED)
python 16245 vagrant 5u IPv4 2195673 0t0 TCP localhost:52974->localhost:http-alt (ESTABLISHED)
python 16245 vagrant 6u IPv4 2195695 0t0 TCP localhost:52978->localhost:http-alt (ESTABLISHED)
python 16245 vagrant 7u IPv4 2195717 0t0 TCP localhost:52982->localhost:http-alt (ESTABLISHED)
python 16245 vagrant 13r CHR 1,9 0t0 11 /dev/urandom
~~~

This issue was observed in storlets functional tests, which use swiftclient to test api requests, and this remaining connections cause 'too many open files' error because the test process keeps lots of opened connections.

We don't observe the same in our py3 tests, so it seems that this happens only on Python2.7.
Also, we don't reproduce the issue after we switch to swiftclient 3.8.1, so the issue caused by some changes during 3.8.1 and 3.9.0 .

[Tim Burke (1-tim-z)]

Thanks for the detailed bug report! Ironically, I think this was caused by https://review.opendev.org/#/c/674320/ which attempted to solve exactly the same problem :-( I think https://review.opendev.org/721051 should fix it reasonably well, though?

[John Leach (johnleach)]

We've not seen open file leaks, but we have seen a memory leak when making lots of requests under a long running process, on python 2.7, after upgrading to 3.9.0 from 3.8.1

I've tested this patch on 3.9.0 and it's fixed our memory leak!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/736401

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/736401
Committed: https://git.openstack.org/cgit/openstack/python-swiftclient/commit/?id=0c70d164ba52d76a6dbbbe8765d15fb969fc07ff
Submitter: Zuul
Branch: master

commit 0c70d164ba52d76a6dbbbe8765d15fb969fc07ff
Author: Tim Burke <email address hidden>
Date: Wed Jun 17 15:44:22 2020 -0700

    (Mostly) revert "Cleanup session on delete"

    This reverts commit 1f26c5736949e1c3b57c024a315e33fc419f126e for py2.
    Apparently the existence of the __del__ method on Python 2 prevents us
    from cleaning up all file descriptors.

    Change-Id: Id6cff5dd7b9faf9c4240c0cb26b74d05ed37da5b
    Closes-Bug: #1873435
    Related-Bug: #1838775

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/721051
Committed: https://git.openstack.org/cgit/openstack/python-swiftclient/commit/?id=97aa3e65412ee241fce7721927b0b003daf51ed4
Submitter: Zuul
Branch: master

commit 97aa3e65412ee241fce7721927b0b003daf51ed4
Author: Tim Burke <email address hidden>
Date: Sat Apr 18 22:41:55 2020 -0700

    Close connections created when calling module-level functions

    Co-Authored-By: Clay Gerrard <email address hidden>
    Change-Id: Id62e63afc6f2ffa32eb8640787c78559481050f9
    Related-Change: I200ad0cdc8b7999c3f5521b9a822122bd18714bf
    Related-Bug: #1873435
    Closes-Bug: #1838775

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-swiftclient 3.11.0 release.