Unable to establish connection: HTTPConnectionPool max retries exceeded

[Andrew Kalach]

Description
===========
During 9.0RC2 Certification Testing Sutie run, the following tasks:

  CinderVolumes.create_and_delete_volume
  CinderVolumes.create_and_list_volume

failed with error:

Unable to establish connection: HTTPConnectionPool(host='172.16.184.3', port=8776): Max retries exceeded with url: /v2/5f16bae23b244217816646f0276c564f/volumes (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fcb9c0e9210>: Failed to establish a new connection: [Errno 110] Connection timed out',))

Task error log (CinderVolumes.create_and_delete_volume):

2016-06-28 09:34:25.912 27013 DEBUG cinderclient.client [-] Connection error: HTTPConnectionPool(host='172.16.184.3', port=8776): Max retries exceeded with url: /v2/45d9d598880d43fd9efc3f3a4b2be905/volumes/ (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x6f35550>: Failed to establish a new connection: [Errno 110] Connection timed out',)) _cs_request /opt/stack/.venv/lib/python2.7/site-packages/cinderclient/client.py:325
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner [-] Failed to get the resource <Volume: 11fba6cf-a0e1-4ab8-a43b-ecaf7dde5070>: Unable to establish connection: HTTPConnectionPool(host='172.16.184.3', port=8776): Max retries exceeded with url: /v2/45d9d598880d43fd9efc3f3a4b2be905/volumes/11fba6cf-a0e1-4ab8-a43b-ecaf7dde5070 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x6f35550>: Failed to establish a new connection: [Errno 110] Connection timed out',))
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner Traceback (most recent call last):
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/runner.py", line 66, in _run_scenario_once
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner deprecated_output = getattr(scenario_inst, method_name)(**kwargs)
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/cinder/volumes.py", line 136, in create_and_delete_volume
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner self._delete_volume(volume)
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner f = func(self, *args, **kwargs)
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/cinder/utils.py", line 178, in _delete_volume
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner check_interval=CONF.benchmark.cinder_volume_delete_poll_interval
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 211, in wait_for_status
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner resource = update_resource(resource)
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 80, in _get_from_manager
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner raise exceptions.GetResourceFailure(resource=resource, err=e)
2016-06-28 09:34:25.917 27013 ERROR rally.task.runner GetResourceFailure: Failed to get the resource <Volume: 11fba6cf-a0e1-4ab8-a43b-ecaf7dde5070>: Unable to establish connection: HTTPConnectionPool(host='172.16.184.3', port=8776): Max retries exceeded with url: /v2/45d9d598880d43fd9efc3f3a4b2be905/volumes/11fba6cf-a0e1-4ab8-a43b-ecaf7dde5070 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x6f35550>: Failed to establish a new connection: [Errno 110] Connection timed out',))

Steps to reproduce
==================
Install Rally on masternode and run specified rally tasks (see attachment):

1. cd /opt/stack
2. source .venv/bin/activate
3. rally task start <task YAML file>

Expected result
===============
Tasks run without errors

Actual result
=============
Errors described above in logs

Environment
===========
Scale 200-node lab (ENV-10):
  3 controllers
  199 compute nodes
  20 ceph nodes
  DVR enabled
  XVLAN
Image RC2 build:
  fuel-9.0-mos-495-2016-06-16_18-18-00.iso

Diagnostic snapshot for more detailed information:
  http://mos-scale-share.mirantis.com/fuel-snapshot-2016-06-28_13-13-50.tar.gz

[Yuriy Nesenenko (ynesenenko)]

Looks like network is down. It's not enough to learn the cause of failing tests. Please add the complete cinder logs (necessary cinder-volume.log.*).

[Denis Meltsaykin (dmeltsaykin)]

Closing as Invalid after a month in Incomplete.