Swift UI failures when deleting large numbers of objects

[Travis Tripp]

Failed to establish a new connection: [Errno 24] Too many open files',))

Basically I first create a bunch of containers objects and folders with this script:
 - https://github.com/openstack/searchlight/blob/master/test-scripts/generate-swift-data.py
 - ./generate-swift-data.py 1000 5 10
Then I add an empty nested sub-folder foo/bar/hello

Then I try to delete all.

I got an error 500

The browser screen freezes and I cannot do anything.

I manually refresh.

I then re-select all and click delete.

It works.

But I am able to make it do the above again.

Screen shot: http://imgur.com/a/4t4AV

ConnectionError: HTTPConnectionPool(host='192.168.200.200', port=8080): Max retries exceeded with url: /v1/AUTH_4bade81378e6428db0e896db77d68e02/scale_3/BL/object_669 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x108781990>: Failed to establish a new connection: [Errno 24] Too many open files',))
[20/May/2016 15:11:20] "DELETE /api/swift/containers/scale_3/object/BL/object_669 HTTP/1.1" 500 332
HTTP exception with no status/code
Traceback (most recent call last):
  File "/Users/ttripp/dev/openstack/horizon/openstack_dashboard/api/rest/utils.py", line 126, in _wrapped
    data = function(self, request, *args, **kw)
  File "/Users/ttripp/dev/openstack/horizon/openstack_dashboard/api/rest/swift.py", line 211, in delete
    api.swift.swift_delete_object(request, container, object_name)
  File "/Users/ttripp/dev/openstack/horizon/openstack_dashboard/api/swift.py", line 314, in swift_delete_object
    swift_api(request).delete_object(container_name, object_name)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 1721, in delete_object
    response_dict=response_dict)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 1565, in _retry
    service_token=self.service_token, **kwargs)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 1369, in delete_object
    conn.request('DELETE', path, '', headers)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 401, in request
    files=files, **self.requests_args)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/swiftclient/client.py", line 384, in _request
    return self.request_session.request(*arg, **kwarg)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/Users/ttripp/dev/openstack/horizon/.venv/lib/python2.7/site-packages/requests/adapters.py", line 467, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='192.168.200.200', port=8080): Max retries exceeded with url: /v1/AUTH_4bade81378e6428db0e896db77d68e02/scale_3/BL/object_667 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x1080f0cd0>: Failed to establish a new connection: [Errno 24] Too many open files',))
[20/May/2016 15:11:20] "DELETE /api/swift/containers/scale_3/object/BL/object_667 HTTP/1.1" 500 332

[Timur Sufiev (tsufiev-x)]

So, Richard, it seems that browser _could_ benefit from some throttling, couldn't it ;)?

[Rob Cresswell (robcresswell-deactivatedaccount)]

Timur sending Richard directly to the burn ward

[Richard Jones (r1chardj0n3s)]

Thanks for reporting this, Travis. I saw the same issue with Django running out of FDs but forgot to report the bug :-/

Someone isn't closing files. I can't see how slowing things down more than the browser does (according to this http://www.stevesouders.com/blog/2008/03/20/roundup-on-parallel-connections/ it's going to be about 6 parallel connections in Chrome when I'm testing) will change that.

[Timur Sufiev (tsufiev-x)]

Okay, it seems that I was trying to find a black cat in a dark room because the black cat was the only thing I could imagine being in a dark room. Richard, I'm taking my words back.

I think I might have seen something similar during mass objects upload, but wasn't sure. That looks a Swift bug to me, what do you think?

[Richard Jones (r1chardj0n3s)]

I found the culprit, patch incoming.

[Richard Jones (r1chardj0n3s)]

In other news though, I have a superior data-loading script for swift testing that parallelises the object creation like a browser does :-)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/321362

[Travis Tripp (travis-tripp)]

Richard, would you mind sharing your superior data loading script? We'd take it in searchlight as a replacement for the one i mention above if it truly is superior!

[Richard Jones (r1chardj0n3s)]

Here y'are https://gist.github.com/r1chardj0n3s/b59f3395de01854626bc58f5335623d0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/321362
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=2481d2b1ac2f0e5adaeb981476f882d8b390974b
Submitter: Jenkins
Branch: master

commit 2481d2b1ac2f0e5adaeb981476f882d8b390974b
Author: Richard Jones <email address hidden>
Date: Thu May 26 16:02:57 2016 +1000

    Remove memoize that holds connections open

    This memoize usage holds on to connection objects for a very long
    time, resulting in exhaustion of file descriptors.

    Change-Id: If7367819b050a65562b3e05175ab15bd93d0d398
    Fixes-Bug: 1584109