Mirantis OpenStack

neutron sometimes doesn't respond to nova's request within 30 secs

Bug #1429087 reported by Leontii Istomin on 2015-03-06

This bug report is a duplicate of: Bug #1498167: neutron response to nova with security-group-rules.json too long. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Mirantis OpenStack	Invalid	High	Dina Belova	Mirantis OpenStack 6.1

Bug Description

[root@fuel ~]# fuel --fuel-version
DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient.
api: '1.0'
astute_sha: 1be5b9b827f512d740fe907c7ff72486d4030938
auth_required: true
build_id: 2015-03-02_14-00-04
build_number: '154'
feature_groups:
- mirantis
fuellib_sha: b17e3810dbca407fca2a231c26f553a46e166343
fuelmain_sha: baf24424a4e056c6753913de5f8c94851903f718
nailgun_sha: f034fbb4b68be963e4dc5b5d680061b54efbf605
ostf_sha: 103d6cf6badd57b791cfaf4310ec8bd81c7a8a46
production: docker
python-fuelclient_sha: 3ebfa9c14a192d0298ff787526bf990055a23694
release: '6.1'
release_versions:
  2014.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1be5b9b827f512d740fe907c7ff72486d4030938
      build_id: 2015-03-02_14-00-04
      build_number: '154'
      feature_groups:
      - mirantis
      fuellib_sha: b17e3810dbca407fca2a231c26f553a46e166343
      fuelmain_sha: baf24424a4e056c6753913de5f8c94851903f718
      nailgun_sha: f034fbb4b68be963e4dc5b5d680061b54efbf605
      ostf_sha: 103d6cf6badd57b791cfaf4310ec8bd81c7a8a46
      production: docker
      python-fuelclient_sha: 3ebfa9c14a192d0298ff787526bf990055a23694
      release: '6.1'

Centos, HA, Neutron-vlan,Ceph-all, Debug, nova-quotas, 6.1_154
Controllers:3 Computes:47

Cluster was deployed successfully. But during boot_and_delete_server_with_secgroups rally test we got a lot of errors (80 errors from 470 iterations):
2015-03-06 05:50:37.173 17543 ERROR rally.benchmark.runners.base [-] The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-f1b92a31-fe94-4f5d-b29f-13617ffd4b28)
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base Traceback (most recent call last):
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 77, in _run_scenario_once
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base method_name)(**kwargs) or scenario_output
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/nova/security_group.py", line 125, in boot_and_delete_server_with_secgroups
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base self._delete_security_groups(security_groups)
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/nova/utils.py", line 627, in _delete_security_groups
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base self.clients("nova").security_groups.delete(sg.id)
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base File "/opt/stack/.venv/lib/python2.7/site-packages/novaclient/v2/security_groups.py", line 71, in delete
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base self._delete('/os-security-groups/%s' % base.getid(group))
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base File "/opt/stack/.venv/lib/python2.7/site-packages/novaclient/base.py", line 161, in _delete
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base _resp, _body = self.api.client.delete(url)
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base File "/opt/stack/.venv/lib/python2.7/site-packages/novaclient/client.py", line 449, in delete
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base return self._cs_request(url, 'DELETE', **kwargs)
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base File "/opt/stack/.venv/lib/python2.7/site-packages/novaclient/client.py", line 418, in _cs_request
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base resp, body = self._time_request(url, method, **kwargs)
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base File "/opt/stack/.venv/lib/python2.7/site-packages/novaclient/client.py", line 392, in _time_request
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base resp, body = self.request(url, method, **kwargs)
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base File "/opt/stack/.venv/lib/python2.7/site-packages/novaclient/client.py", line 386, in request
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base raise exceptions.from_response(resp, body, url, method)
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-f1b92a31-fe94-4f5d-b29f-13617ffd4b28)
2015-03-06 05:50:37.173 17543 TRACE rally.benchmark.runners.base
2015-03-06 05:50:37.175 17543 INFO rally.benchmark.runners.base [-] Task 9ac10584-2f7f-45e4-b4d2-719cb9212f79 | ITER: 40 END: Error ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-f1b92a31-fe94-4f5d-b29f-13617ffd4b28)

From nova-all.log file I can see many errors like:
<183>Mar 6 05:50:36 node-166 nova-api throwing ConnectionFailed : HTTPConnectionPool(host='192.168.0.2', port=9696): Read timed out. (read timeout=30)
<183>Mar 6 05:50:36 node-166 nova-api Calling method '<bound method Controller.show of <nova.api.openstack.compute.servers.Controller object at 0x39d2450>>'
<179>Mar 6 05:50:36 node-166 nova-api Neutron Error:
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver Traceback (most recent call last):
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver File "/usr/lib/python2.6/site-packages/nova/network/security_group/neutron_driver.py", line 204, in add_rules
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver body).get('security_group_rules')
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 98, in with_params
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver ret = self.function(instance, *args, **kwargs)
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 496, in create_security_group_rule
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver return self.post(self.security_group_rules_path, body=body)
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1325, in post
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver headers=headers, params=params)
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1240, in do_request
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver content_type=self.content_type())
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 180, in do_request
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver **kwargs)
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 133, in _cs_request
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver raise exceptions.ConnectionFailed(reason=e)
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='192.168.0.2', port=9696): Read timed out. (read timeout=30)
2015-03-06 05:50:36.320 24099 TRACE nova.network.security_group.neutron_driver
<183>Mar 6 05:50:36 node-166 nova-api
REQ: curl -i http://192.168.0.2:9696/v2.0/ports.json?device_id=3d14312d-2a87-49ed-8d9e-f98fb502fe94 -X GET -H "X-Auth-Token: 430e47cbf6d1460099571e2fba0f47ad" -H "User-Agent: python-neutronclient"
<183>Mar 6 05:50:36 node-166 nova-api RESP:200 CaseInsensitiveDict({'date': 'Fri, 06 Mar 2015 05:50:36 GMT', 'connection': 'close', 'content-type': 'application/json; charset=UTF-8', 'content-length': '13', 'x-openstack-request-id': 'req-2f27e2d8-9b93-4dab-bfbf-27d48e8886ff'}) {"ports": []}
<182>Mar 6 05:50:36 node-166 nova-api 192.168.0.2 "GET /v2/40059209ff8844a9b0c956602b2b1336/servers/3d14312d-2a87-49ed-8d9e-f98fb502fe94 HTTP/1.1" status: 200 len: 1562 time: 0.3478839
<179>Mar 6 05:50:37 node-166 nova-api Caught error: Connection to neutron failed: HTTPConnectionPool(host='192.168.0.2', port=9696): Read timed out. (read timeout=30)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack Traceback (most recent call last):
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 124, in __call__
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack return req.get_response(self.application)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack application, catch_exc_info=False)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack return resp(environ, start_response)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/keystonemiddleware/auth_token.py", line 748, in __call__
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack return self._call_app(env, start_response)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/keystonemiddleware/auth_token.py", line 684, in _call_app
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack return self._app(env, _fake_start_response)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack return resp(environ, start_response)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack return resp(environ, start_response)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/routes/middleware.py", line 131, in __call__
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack response = self.app(environ, start_response)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack return resp(environ, start_response)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 910, in __call__
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack content_type, body, accept)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 975, in _process_stack
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1059, in dispatch
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack return method(req=request, **action_args)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 415, in create
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack context, security_group, new_rule))
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/network/security_group/security_group_base.py", line 139, in create_security_group_rule
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack [new_rule])[0]
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/network/security_group/neutron_driver.py", line 204, in add_rules
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack body).get('security_group_rules')
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 98, in with_params
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack ret = self.function(instance, *args, **kwargs)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 496, in create_security_group_rule
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack return self.post(self.security_group_rules_path, body=body)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1325, in post
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack headers=headers, params=params)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1240, in do_request
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack content_type=self.content_type())
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 180, in do_request
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack **kwargs)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 133, in _cs_request
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack raise exceptions.ConnectionFailed(reason=e)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='192.168.0.2', port=9696): Read timed out. (read timeout=30)
2015-03-06 05:50:36.682 24099 TRACE nova.api.openstack
<182>Mar 6 05:50:37 node-166 nova-api http://172.16.45.101:8774/v2/40059209ff8844a9b0c956602b2b1336/os-security-group-rules returned with HTTP 500
<183>Mar 6 05:50:37 node-166 nova-api Returning 500 to user: The server has either erred or is incapable of performing the requested operation.
<183>Mar 6 05:50:37 node-166 nova-api Returning 500 to user: The server has either erred or is incapable of performing the requested operation.
<182>Mar 6 05:50:37 node-166 nova-api 192.168.0.2 "POST /v2/40059209ff8844a9b0c956602b2b1336/os-security-group-rules HTTP/1.1" status: 500 len: 354 time: 32.5121419
<183>Mar 6 05:50:37 node-166 nova-api throwing ConnectionFailed : HTTPConnectionPool(host='192.168.0.2', port=9696): Read timed out. (read timeout=30)
<179>Mar 6 05:50:37 node-166 nova-api Neutron Error: Connection to neutron failed: HTTPConnectionPool(host='192.168.0.2', port=9696): Read timed out. (read timeout=30)
<179>Mar 6 05:50:37 node-166 nova-api Caught error: Connection to neutron failed: HTTPConnectionPool(host='192.168.0.2', port=9696): Read timed out. (read timeout=30)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack Traceback (most recent call last):
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 124, in __call__
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack return req.get_response(self.application)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack application, catch_exc_info=False)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack return resp(environ, start_response)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/keystonemiddleware/auth_token.py", line 748, in __call__
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack return self._call_app(env, start_response)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/keystonemiddleware/auth_token.py", line 684, in _call_app
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack return self._app(env, _fake_start_response)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack return resp(environ, start_response)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack return resp(environ, start_response)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/routes/middleware.py", line 131, in __call__
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack response = self.app(environ, start_response)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack return resp(environ, start_response)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 910, in __call__
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack content_type, body, accept)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 975, in _process_stack
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1059, in dispatch
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack return method(req=request, **action_args)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 296, in delete
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack self.security_group_api.destroy(context, security_group)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/network/security_group/neutron_driver.py", line 180, in destroy
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack neutron.delete_security_group(security_group['id'])
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 98, in with_params
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack ret = self.function(instance, *args, **kwargs)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 491, in delete_security_group
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack return self.delete(self.security_group_path % (security_group))
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1316, in delete
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack headers=headers, params=params)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1297, in retry_request
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack headers=headers, params=params)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1240, in do_request
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack content_type=self.content_type())
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 180, in do_request
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack **kwargs)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 133, in _cs_request
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack raise exceptions.ConnectionFailed(reason=e)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='192.168.0.2', port=9696): Read timed out. (read timeout=30)
2015-03-06 05:50:37.149 24099 TRACE nova.api.openstack

It can be related to https://bugs.launchpad.net/mos/+bug/1429065

DG is coming

Tags:

Alexander Ignatov (aignatov) on 2015-03-06

Changed in mos:
assignee:	MOS Neutron (mos-neutron) → Ann Kamyshnikova (akamyshnikova)

Alexander Ignatov (aignatov) on 2015-03-06

Changed in mos:
assignee:	Ann Kamyshnikova (akamyshnikova) → MOS Neutron (mos-neutron)

Dmitry Mescheryakov (dmitrymex) on 2015-03-13

Changed in mos:
importance:	Undecided → High
milestone:	none → 6.1
status:	New → Confirmed

Alexander Ignatov (aignatov) on 2015-03-19

tags:

added: neutron

Revision history for this message

Alexander Ignatov (aignatov) wrote on 2015-03-24:

Waiting for diagnostinc snapshot still or repro with live environment

Changed in mos:
assignee:	MOS Neutron (mos-neutron) → Dina Belova (dbelova)
status:	Confirmed → Incomplete

Revision history for this message

Eugene Nikanorov (enikanorov) wrote on 2015-03-25:

That might not be a helpful comment, but this issue has rarely been seen in upstream gates and reported by different people.

The problem for sure exists in neutron, but repro or env with a repro not necessarily will help to analyze it. It's because 'lost' request probably doesn't leave a trace in neutron's logs.

So I'd propose the following ways to troubleshoot the issue (if another repro is made):
1) look at haproxy logs for the lost request, try to understand, which neutron server was serving it
2) if backend is identified - try to look at server logs for the request

I guess (2) will not give results.

Another guess is that the issue may have higher rate of occurence if load is increased, for that I'm proposing shutting down neutron-servers on 2 out of 3 controllers.

Sergey Kraynev (skraynev) on 2015-04-26

Changed in mos:
status:	Incomplete → Opinion
status:	Opinion → Confirmed
status:	Confirmed → Incomplete

Revision history for this message

Dina Belova (dbelova) wrote on 2015-05-05:

No repro yet. Will move to the confirmed if we'll face it.

Changed in mos:
status:	Incomplete → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

Duplicate of bug #1498167 Remove

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.