http://logs.openstack.org/04/637304/7/check/neutron-tempest-plugin-scenario-linuxbridge/e562437/testr_results.html.gz
http://logs.openstack.org/57/651357/1/check/neutron-tempest-dvr-ha-multinode-full/3a0f4ee/testr_results.html.gz
http://logs.openstack.org/82/643882/4/check/tempest-slow-py3/3432a15/testr_results.html.gz

You may noticed, these are 3 different check, but they have some similar fail LOGs. Here are what I found:
(1) all of them have the following floating IP DOWN state log:

Response - Headers: {'content-type': 'application/json', 'content-length': '875', 'x-openstack-request-id': 'req-51260e4b-e58a-4010-89f9-8b721cfbf274', 'date': 'Mon, 25 Mar 2019 15:14:41 GMT', 'connection': 'close', 'status': '201', 'content-location': 'http://158.69.67.5:9696/v2.0/floatingips'}
        Body: b'{"floatingip": {"id": "2819b03c-efc1-4b9c-b3f1-709be52d3226", "tenant_id": "3a06df8878b94df5888472572bad7661", "floating_ip_address": "172.24.5.232", "floating_network_id": "beea0fd8-378a-4c91-a820-dde4f9759c8d", "router_id": "60850e98-af58-4c4c-8f02-e613e74d070a", "port_id": "dd095439-8cd1-4273-a9d4-638171bacb61", "fixed_ip_address": "10.1.0.3", "status": "DOWN", "project_id": "3a06df8878b94df5888472572bad7661", "description": "", "qos_policy_id": null, "port_details": {"name": "tempest-port-1415223031", "network_id": "97a59130-9f64-469b-acc2-e8e5c7f101ec", "mac_address": "fa:16:3e:77:8f:f5", "admin_state_up": true, "status": "ACTIVE", "device_id": "703c47eb-fb28-432e-bc02-c6b7308bbf22", "device_owner": "compute:nova"}, "dns_domain": "", "dns_name": "", "tags": [], "created_at": "2019-03-25T15:14:36Z", "updated_at": "2019-03-25T15:14:36Z", "revision_number": 0}}'


Response - Headers: {u'content-type': 'application/json', u'server': 'Apache/2.4.29 (Ubuntu)', u'date': 'Tue, 09 Apr 2019 19:59:03 GMT', u'content-length': '794', 'status': '200', u'x-openstack-request-id': 'req-dd2f3b1f-cf38-48d9-89fc-fcf01f74191d', 'content-location': 'https://149.202.189.177:9696/v2.0/floatingips/58373a20-471b-4c07-943f-003b5406d1e3', u'connection': 'close'}
        Body: {"floatingip": {"router_id": "830816db-b9ed-4683-8e13-24f5e3d07cad", "status": "DOWN", "description": "", "tags": [], "tenant_id": "7f9af55aaed34654b39faeaa75df5159", "created_at": "2019-04-09T19:58:38Z", "updated_at": "2019-04-09T19:59:02Z", "floating_network_id": "5dc501ef-3da6-4d88-8b72-189d5b1f75e4", "port_details": {"status": "ACTIVE", "name": "", "admin_state_up": true, "network_id": "012b87c3-3dff-4929-83c6-df007b6b4ffe", "device_owner": "compute:nova", "mac_address": "fa:16:3e:1e:cd:83", "device_id": "fe17980f-c80a-40ac-b300-27e09a470c91"}, "fixed_ip_address": "10.1.0.8", "floating_ip_address": "172.24.5.88", "revision_number": 1, "project_id": "7f9af55aaed34654b39faeaa75df5159", "port_id": "24431c68-dbe1-4546-88ff-40457f3992f3", "id": "58373a20-471b-4c07-943f-003b5406d1e3"}}

2019-04-04 08:30:08,077 28397 INFO     [tempest.scenario.manager] FloatingIP: {'id': 'aab6996a-a477-4b7f-a8f6-dda02091d18b', 'tenant_id': 'e590fad8c3984c84adfe3bad5f3830ac', 'floating_ip_address': '172.24.5.121', 'floating_network_id': '27f82c48-3f0d-4bc6-8839-dc5498191f63', 'router_id': '9709370a-babb-4891-aa9f-2ff51a84b789', 'port_id': '4176c492-e6c6-4956-9a34-17feafd64a40', 'fixed_ip_address': '10.1.0.9', 'status': 'DOWN', 'project_id': 'e590fad8c3984c84adfe3bad5f3830ac', 'description': '', 'port_details': {'name': '', 'network_id': '000e4dda-2136-47e6-a384-571c450277d6', 'mac_address': 'fa:16:3e:99:3f:b1', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'e5ce05a3-526b-4d00-b3da-e773da0cf091', 'device_owner': 'compute:nova'}, 'tags': [], 'created_at': '2019-04-04T08:30:04Z', 'updated_at': '2019-04-04T08:30:04Z', 'revision_number': 0} is at status: ACTIVE


(2) some have "ssh Authentication failed"

(3) "([Errno None] Unable to connect to port 22 on XXX.XXX.XX.XXX). Number attempts: 13. Retry after 14 seconds."
LOG raised during the Cirros booting. And most of them then had many times retry. 


So, some causes of the test failure could be:
(1) floating IP DOWN state
(2) ssh auth key did not set to the VM, but ssh connection has been started since the VM is ACTIVE
(3) VM operating system was not actually UP, but nova marked it is ACTIVE, then ssh it anyway.