[swarm 9.2] OSTF test Launch instance with file injection failed on Step 2

[Alexandra]

Swarm test is failed:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.install_plugin_after_cluster_create/102/testReport/(root)/five_ctrl_enable_installed_after_create_add/five_ctrl_enable_installed_after_create_add/

shotgun report - http://paste.openstack.org/show/585691/

extraction from ostf log with oslo_db.exception.DBConnectionError (http://paste.openstack.org/show/585688/):
2016-10-14 00:01:19 FAILURE Launch instance with file injection (fuel_health.tests.smoke.test_nova_create_instance_with_connectivity.TestNovaNetwork.test_009_create_server_with_file) Creating instance using the new security group has failed. Please refer to OpenStack logs for more details. File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
    yield
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 601, in run
    testMethod()
  File "/usr/lib/python2.7/site-packages/fuel_health/tests/smoke/test_nova_create_instance_with_connectivity.py", line 330, in test_009_create_server_with_file
    self.compute_client, name, security_groups, data_file=data_file
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 183, in verify
    " Please refer to OpenStack logs for more details.")
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
Step 2 failed: Creating instance using the new security group has failed. Please refer to OpenStack logs for more details.

neutron-server log with _mysql_exceptions.OperationalError (Lost connection to MySQL server during query):
http://paste.openstack.org/show/585689/

similar errors are in neutron-openvswitch-agent, neutron-l3-agent, neutron-dhcp-agent logs

[Dmitry Belyaninov (dbelyaninov)]

Reproduced on acceptance:
https://product-ci.infra.mirantis.net/view/9.x_acceptance/job/9.x.acceptance.ubuntu.ha_vlan_group_6/18/console

also in ostf.log:
NotFound: Subnet c8e4e9c4-47d3-4c54-86b2-b3adefa85ff0 could not be found.
Neutron server returns request_ids: ['req-048e322e-8fce-4ee3-93da-42a3165a84e9']

NetworkNotFoundClient: Network 23a7f216-600d-49d5-8fb5-da5df4c84618 could not be found.
Neutron server returns request_ids: ['req-b13dcff0-78f1-4fdd-83d3-e3292d57ee87']

[Sergii Golovatiuk (sgolovatiuk)]

MySQL connection has gone. Normally, neutron should have retried the transaction, but that never happened, thus it failed.

[Vladimir Khlyunev (vkhlyunev)]

Got reproduce in https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.bonding_ha/120/console
at cleanup after passed network connectivity check at removing security group.
neutron-server contains several tracebacks after returning 500 err code:

2016-11-11 05:05:55.388 19010 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 206, in __init__
2016-11-11 05:05:55.388 19010 ERROR neutron.api.v2.resource super(Connection, self).__init__(*args, **kwargs2)
2016-11-11 05:05:55.388 19010 ERROR neutron.api.v2.resource DBConnectionError: (_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") [SQL: u'SE
LECT 1']

[Alexander Ignatov (aignatov)]

In last failure Neutron failed to retrieve list of floatings ip from database because that was unavailable

2016-11-11 05:05:41.265 24203 ERROR neutron.api.v2.resource DBConnectionError: (_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") [SQL: u'SELECT 1']
2016-11-11 05:05:41.265 24203 ERROR neutron.api.v2.resource
2016-11-11 05:05:41.295 24203 INFO neutron.wsgi [req-7a9568ce-fc5c-4e9c-a4f3-ff5f1aee9c94 5da22b70e9574bfa91c7bf31364df7b6 7dad77e74cd2413ba01afeb6c55f87ab - - -] 192.168.0.7 - - [11/Nov/2016 05:05:41] "GET /v2.0/floatingips/e98a635d-1ef6-43ac-8775-be8f35faa190.json HTTP/1.1" 500 378 1.968047

In next 11 seconds mysql was down and Neutron could't predict user behavior and serve all incoming API requests and couldn't guarantee that all requests would have been properly handled since it's unknown how long mysql service was unavailable. There is no such logic exists in Neutron to handle all incoming API calls in case of broken underlay (rpc/db/etc.)

Next 11 seconds DB became up and Neutron successfully reconnected to it but original request got lost. I'm sure after reexecution of this test when db became up this case will be passed.

I consider this bug as Invalid in Neutron spaces since there is no way to fix this logic except investigation of what was the reason of mysql outage.