Connection to neutron failed: HTTPConnectionPool read timed out

[Andrew Kalach]

Description
===========
During 9.0RC2 Certification Testing Sutie run, the following tasks:

  NeutronNetworks.create_and_list_routers
  NeutronNetworks.create_and_delete_routers
  NeutronNetworks.create_and_update_networks
  NeutronSecGroupPlugin.create_and_delete_secgroups
  NeutronSecGroupPlugin.boot_and_delete_server_with_secgroups

failed with error:

Connection to neutron failed: HTTPConnectionPool(host='172.16.184.3', port=9696): Read timed out. (read timeout=180.0)

Task error log (NeutronNetworks.create_and_list_routers):

2016-06-27 04:58:07.004 21234 DEBUG neutronclient.client [-] throwing ConnectionFailed : HTTPConnectionPool(host='172.16.184.3', port=9696): Read timed out. (read timeout=180.0) _cs_request /opt/stack/.venv/lib/python2.7/site-packages/neutronclient/client.py:105
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner [-] Connection to neutron failed: HTTPConnectionPool(host='172.16.184.3', port=9696): Read timed out. (read timeout=180.0)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner Traceback (most recent call last):
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/runner.py", line 66, in _run_scenario_once
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner deprecated_output = getattr(scenario_inst, method_name)(**kwargs)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/neutron/network.py", line 239, in create_and_delete_routers
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner self._remove_interface_router(subnet["subnet"], router["router"])
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner f = func(self, *args, **kwargs)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/neutron/utils.py", line 373, in _remove_interface_router
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner router["id"], {"subnet_id": subnet["id"]})
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 752, in remove_interface_router
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner "/remove_router_interface", body=body)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 351, in put
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner headers=headers, params=params)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 319, in retry_request
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner headers=headers, params=params)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in do_request
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner resp, replybody = self.httpclient.do_request(action, method, body=body)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/client.py", line 170, in do_request
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner **kwargs)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/client.py", line 106, in _cs_request
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner raise exceptions.ConnectionFailed(reason=e)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='172.16.184.3', port=9696): Read timed out. (read timeout=180.0)

Steps to reproduce
==================
Install Rally on masternode and run specified rally tasks (see attachment):

1. cd /opt/stack
2. source .venv/bin/activate
3. rally task start <task YAML file>

Expected result
===============
Tasks run without errors

Actual result
=============
Errors described above in logs

Environment
===========
Scale 200-node lab (ENV-10):
  3 controllers
  199 compute nodes
  20 ceph nodes
  DVR enabled
  XVLAN
Image RC2 build:
  fuel-9.0-mos-495-2016-06-16_18-18-00.iso

Diagnostic snapshot for more detailed information:
  http://mos-scale-share.mirantis.com/fuel-snapshot-2016-06-28_13-13-50.tar.gz

[Oleg Bondarev (obondarev)]

Attachment from comment 1 has only some random log files from neutron dhcp and l3 agents. Would be good to have all neutron logs (especially neutron server logs) structured with respect to nodes (like in snapshot).

[Denis Meltsaykin (dmeltsaykin)]

Moving to Invalid after a month in Incomplete.