Rally hit the filedescriptor limit in multiple test runs.
This is what I got from the run output:
2017-01-19 16:00:04.786 27098 WARNING rally.plugins.openstack.cleanup.manager [-] Seems like rally.plugins.openstack.cleanup.resources.SwiftObject.list(self) method is broken. It shouldn't raise any exceptions.
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager [-] filedescriptor out of range in select()
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager Traceback (most recent call last):
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/cleanup/manager.py", line 140, in _publish
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager for raw_resource in rutils.retry(3, manager.list):
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/rally/common/utils.py", line 220, in retry
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager return func(*args, **kwargs)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/cleanup/resources.py", line 743, in list
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager containers = self._manager().get_account(full_listing=True)[1]
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/cleanup/resources.py", line 718, in _manager
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager return getattr(client, self._service)()
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/rally/osclients.py", line 208, in __call__
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager self.cache[key] = self.create_client(*args, **kwargs)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/rally/osclients.py", line 591, in create_client
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager auth_token = self.keystone.auth_ref.auth_token
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/rally/osclients.py", line 231, in auth_ref
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager sess, plugin = self.get_session()
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/rally/osclients.py", line 264, in get_session
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager password_args["auth_url"]).version_data()[0]["version"][0])
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager return wrapped(*args, **kwargs)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/discover.py", line 140, in __init__
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager authenticated=authenticated)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager return wrapped(*args, **kwargs)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/discover.py", line 40, in get_version_data
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager resp = session.get(url, headers=headers, authenticated=authenticated)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 758, in get
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager return self.request(url, 'GET', **kwargs)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager return wrapped(*args, **kwargs)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 616, in request
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager resp = send(**kwargs)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 674, in _send_request
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager resp = self.session.request(method, url, **kwargs)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager resp = self.send(prep, **send_kwargs)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager r = adapter.send(request, **kwargs)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 423, in send
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager timeout=timeout
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager chunked=chunked)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 350, in _make_request
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager self._validate_conn(conn)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 835, in _validate_conn
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager conn.connect()
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py", line 323, in connect
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager ssl_context=context)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py", line 324, in ssl_wrap_socket
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager return context.wrap_socket(sock, server_hostname=server_hostname)
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 419, in wrap_socket
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager rd, _, _ = select.select([sock], [], [], sock.gettimeout())
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager ValueError: filedescriptor out of range in select()
2017-01-19 16:00:04.786 27098 ERROR rally.plugins.openstack.cleanup.manager
I'm trying to do some kind of stress test using the following config:
---
SwiftObjects.create_container_and_object_then_delete_all:
{% for concurrent in range(2, 102, 2) %}
-
args:
objects_per_container: 1
object_size: 65536
runner:
type: constant
times: {{ concurrent * 10 }}
concurrency: {{ concurrent }}
{% endfor %}
Rally reached the limit at a concurrency of around 12 or 14.
I already increased the open file limit.
ulimit -n
150000
Rally Version: 0.7.1~dev286
It seems there is another limit. I found some similar issues in other projects like Ansible.
Unfortunately, there is a known issue(not related to rally) - https:/