neutron

  1. Bug #1596075
  2. Comment #62

Comment 62 for bug 1596075

Revision history for this message
Don Bowman (donbowman) wrote :

And one more.
Here the l3-agent (n4) gets
RemoteError: Remote error: OperationalError (pymysql.err.OperationalError) (1045, u'failed to create new session') [SQL: u'SELECT 1'], and the api agents went non-responsive towards nginx

this seems to occur after a bunch of

2016-07-14 04:31:43.687 3995 DEBUG neutronclient.client [req-8678bb34-66ce-410a-b764-a30b5e2470f1 a4aab8472ebb41faa98f89e1dd34858a 469300d96f9f406cacda5f001d387e52 - - -] REQ: curl -i https://neutron.sandvine.rocks//v2.0/networks.json -X POST -H "User-Agent: python-neutronclient" -H "X-Auth-Token: {SHA1}e186ee7429bd4264f0a47d97e1f53f5ea52fc124" -d '{"network": {"shared": false, "name": "t28_n97", "admin_state_up": true}}' http_log_req /usr/local/lib/python2.7/dist-packages/neutronclient/common/utils.py:185
[332B blob data]
[44B blob data]
[23B blob data]
[42B blob data]
[43B blob data]
[8B blob data]
[8B blob data]
 http_log_resp /usr/local/lib/python2.7/dist-packages/neutronclient/common/utils.py:194
[198B blob data]
[44B blob data]
[23B blob data]
[42B blob data]
[43B blob data]
[8B blob data]
[8B blob data]
 _handle_fault_response /usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:247

has occurred in heat.

 _handle_fault_response /usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:247
2016-07-14 04:31:43.707 3995 INFO heat.engine.resource [req-8678bb34-66ce-410a-b764-a30b5e2470f1 a4aab8472ebb41faa98f89e1dd34858a 469300d96f9f406cacda5f001d387e52 - - -] CREATE: Net "n97" Stack "t28" [5f6fde63-931b-4bcc-bf10-66ebdba750e9]
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource Traceback (most recent call last):
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/heat/engine/resource.py", line 732, in _action_recorder
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource yield
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/heat/engine/resource.py", line 812, in _do_action
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 312, in wrapper
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource step = next(subtask)
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/heat/engine/resource.py", line 775, in action_handler_task
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource handler_data = handler(*args)
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/heat/engine/resources/openstack/neutron/net.py", line 155, in handle_create
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource net = self.client().create_network({'network': props})['network']
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 648, in create_network
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource return self.post(self.networks_path, body=body)
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 347, in post
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource headers=headers, params=params)
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 282, in do_request
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource self._handle_fault_response(status_code, replybody, resp)
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 257, in _handle_fault_response
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource exception_handler_v20(status_code, error_body)
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 84, in exception_handler_v20
2016-07-14 04:31:43.707 3995 ERROR heat.engine.resource request_ids=request_ids)

in turn it seems it was doing:

n1-a-nginx-neutron.log:172.17.0.8 - - [14/Jul/2016:04:31:43 +0000] "POST //v2.0/networks.json HTTP/1.1" 502 182 "-" "python-neutronclient"
n1-e-nginx-neutron.log:2016/07/14 04:31:43 [error] 31801#31801: *131651 no live upstreams while connecting to upstream, client: 172.17.0.8, server: neutron.sandvine.rocks, request: "POST //v2.0/networks.json HTTP/1.1", upstream: "http://neutron_proxy//v2.0/networks.json", host: "neutron.sandvine.rocks"