There was a lot of errors in neutron logs like this:
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 686, in _report_state 2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent True) 2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/rpc.py", line 87, in report_state 2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent return method(context, 'report_state', **kwargs) 2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call 2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent retry=self.retry) 2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 91, in _send 2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent timeout=timeout, retry=retry) 2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 512, in send 2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent retry=retry) 2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 503, in _send 2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent raise result 2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent RemoteError: Remote error: DBConnectionError (_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") [SQL: u'SELECT 1']
For some reason MySQL server was restarted when "openstack-network-routers" task had been executing. Seems that caused the issue.
After that bug haven't been reproduced.
There was a lot of errors in neutron logs like this:
2016-12-26 08:26:15.827 2080 ERROR neutron. agent.l3. agent Traceback (most recent call last): agent.l3. agent File "/usr/lib/ python2. 7/dist- packages/ neutron/ agent/l3/ agent.py" , line 686, in _report_state agent.l3. agent True) agent.l3. agent File "/usr/lib/ python2. 7/dist- packages/ neutron/ agent/rpc. py", line 87, in report_state agent.l3. agent return method(context, 'report_state', **kwargs) agent.l3. agent File "/usr/lib/ python2. 7/dist- packages/ oslo_messaging/ rpc/client. py", line 158, in call agent.l3. agent retry=self.retry) agent.l3. agent File "/usr/lib/ python2. 7/dist- packages/ oslo_messaging/ transport. py", line 91, in _send agent.l3. agent timeout=timeout, retry=retry) agent.l3. agent File "/usr/lib/ python2. 7/dist- packages/ oslo_messaging/ _drivers/ amqpdriver. py", line 512, in send agent.l3. agent retry=retry) agent.l3. agent File "/usr/lib/ python2. 7/dist- packages/ oslo_messaging/ _drivers/ amqpdriver. py", line 503, in _send agent.l3. agent raise result agent.l3. agent RemoteError: Remote error: DBConnectionError (_mysql_ exceptions. OperationalErro r) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") [SQL: u'SELECT 1']
2016-12-26 08:26:15.827 2080 ERROR neutron.
2016-12-26 08:26:15.827 2080 ERROR neutron.
2016-12-26 08:26:15.827 2080 ERROR neutron.
2016-12-26 08:26:15.827 2080 ERROR neutron.
2016-12-26 08:26:15.827 2080 ERROR neutron.
2016-12-26 08:26:15.827 2080 ERROR neutron.
2016-12-26 08:26:15.827 2080 ERROR neutron.
2016-12-26 08:26:15.827 2080 ERROR neutron.
2016-12-26 08:26:15.827 2080 ERROR neutron.
2016-12-26 08:26:15.827 2080 ERROR neutron.
2016-12-26 08:26:15.827 2080 ERROR neutron.
2016-12-26 08:26:15.827 2080 ERROR neutron.
2016-12-26 08:26:15.827 2080 ERROR neutron.
For some reason MySQL server was restarted when "openstack- network- routers" task had been executing. Seems that caused the issue.
After that bug haven't been reproduced.