[ostf] 'Check network connectivity from instance via floating IP' double delete instance

[Sergey Murashov]

ISO: http://mc0n4-msk.msk.mirantis.net/fuelweb-iso/fuel-gerrit-5.0.1-467-2014-07-23_20-46-56.iso
Steps to reproduce:
1. Install OS with HA mode(3 controllers)CentOS
2. Destroy one of the controllers
3. Run OSTF tests.

Actual result:
Test 'Check network connectivity from instance via floating IP' fails

In OSTF logs we can see:
2014-07-23 06:20:11 DEBUG (client) New session created for: (http://10.108.6.2:5000)
2014-07-23 06:20:11 INFO (connectionpool) Starting new HTTP connection (1): 10.108.5.3
2014-07-23 06:20:11 DEBUG (connectionpool) "POST http://10.108.6.2:5000/v2.0/tokens HTTP/1.1" 200 3325
2014-07-23 06:20:11 DEBUG (client) New session created for: (http://10.108.6.2:8774)
2014-07-23 06:20:11 INFO (connectionpool) Starting new HTTP connection (1): 10.108.5.3
2014-07-23 06:20:11 DEBUG (connectionpool) "GET http://10.108.6.2:8774/v2/5a847d190cae4ea8998e0a345f78be49/os-networks HTTP/1.1" 200 1142
2014-07-23 06:20:17 INFO (connectionpool) Starting new HTTP connection (1): 10.108.5.3
2014-07-23 06:20:17 DEBUG (connectionpool) "GET http://10.108.6.2:8774/v2/5a847d190cae4ea8998e0a345f78be49/servers/6959578e-1649-44eb-8bc0-eb21463daf87 HTTP/1.1" 200 1747
2014-07-23 06:20:17 DEBUG (test) Waiting for <Server: ost1_test-server-smoke-1955734893> to get to ACTIVE status. Currently in build status
2014-07-23 06:20:17 DEBUG (test) Sleeping for 10 seconds
2014-07-23 06:20:27 INFO (connectionpool) Starting new HTTP connection (1): 10.108.5.3
2014-07-23 06:20:27 DEBUG (connectionpool) "GET http://10.108.6.2:8774/v2/5a847d190cae4ea8998e0a345f78be49/servers/6959578e-1649-44eb-8bc0-eb21463daf87 HTTP/1.1" 200 1764
2014-07-23 06:20:27 INFO (connectionpool) Starting new HTTP connection (1): 10.108.5.3
2014-07-23 06:20:27 DEBUG (connectionpool) "GET http://10.108.6.2:8774/v2/5a847d190cae4ea8998e0a345f78be49/servers/6959578e-1649-44eb-8bc0-eb21463daf87 HTTP/1.1" 200 1764
2014-07-23 06:20:27 DEBUG (test) Adding <Server: ost1_test-server-smoke-1955734893> to shared resources of TestNovaNetwork
2014-07-23 06:20:27 DEBUG (nmanager) Deleting server.
2014-07-23 06:20:27 INFO (connectionpool) Starting new HTTP connection (1): 10.108.5.3
2014-07-23 06:20:28 DEBUG (connectionpool) "DELETE http://10.108.6.2:8774/v2/5a847d190cae4ea8998e0a345f78be49/servers/6959578e-1649-44eb-8bc0-eb21463daf87 HTTP/1.1" 204 0
2014-07-23 06:20:28 INFO (connectionpool) Starting new HTTP connection (1): 10.108.5.3
2014-07-23 06:20:28 DEBUG (connectionpool) "GET http://10.108.6.2:8774/v2/5a847d190cae4ea8998e0a345f78be49/servers/6959578e-1649-44eb-8bc0-eb21463daf87 HTTP/1.1" 200 1770
2014-07-23 06:20:28 DEBUG (test) Sleeping for 10 seconds
2014-07-23 06:20:38 INFO (connectionpool) Starting new HTTP connection (1): 10.108.5.3
2014-07-23 06:20:38 DEBUG (connectionpool) "GET http://10.108.6.2:8774/v2/5a847d190cae4ea8998e0a345f78be49/servers/6959578e-1649-44eb-8bc0-eb21463daf87 HTTP/1.1" 404 73
2014-07-23 06:20:38 DEBUG (nmanager) Deleting server.
2014-07-23 06:20:38 INFO (connectionpool) Starting new HTTP connection (1): 10.108.5.3
2014-07-23 06:20:38 DEBUG (connectionpool) "DELETE http://10.108.6.2:8774/v2/5a847d190cae4ea8998e0a345f78be49/servers/6959578e-1649-44eb-8bc0-eb21463daf87 HTTP/1.1" 404 73
2014-07-23 06:20:38 DEBUG (test_nova_create_instance_with_connectivity) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/tests/smoke/test_nova_create_instance_with_connectivity.py", line 74, in tearDown
    self._delete_server(server)
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 283, in _delete_server
    self.compute_client.servers.delete(server)
  File "/usr/lib/python2.6/site-packages/novaclient/v1_1/servers.py", line 901, in delete
    self._delete("/servers/%s" % base.getid(server))
  File "/usr/lib/python2.6/site-packages/novaclient/base.py", line 161, in _delete
    _resp, _body = self.api.client.delete(url)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 292, in delete
    return self._cs_request(url, 'DELETE', **kwargs)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 260, in _cs_request
    **kwargs)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 242, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 236, in request
    raise exceptions.from_response(resp, body, url, method)
NotFound: Instance could not be found (HTTP 404) (Request-ID: req-969e3bf1-5413-4c8b-b4ff-5ab99a5c6eb5)

OSTF try to delete instance repeatedly

2014-07-23 06:23:54 INFO (connectionpool) Starting new HTTP connection (1): 10.108.5.3
2014-07-23 06:23:54 DEBUG (connectionpool) "DELETE http://10.108.7.2:35357/v2.0/tenants/9c14a50926964fce91cb4f3658e792ac HTTP/1.1" 204 0
2014-07-23 06:23:54 DEBUG (session) RESP: [204] CaseInsensitiveDict({'content-length': '0', 'via': '1.1 node-1.test.domain.local:35357', 'vary': 'X-Auth-Token', 'connection': 'close', 'date': 'Wed, 23 Jul
2014 06:23:54 GMT', 'content-type': 'text/plain; charset=UTF-8'})
RESP BODY:

2014-07-23 06:23:54 DEBUG (httpclient) No body was returned.
2014-07-23 06:23:54 DEBUG (nmanager) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 358, in is_deletion_complete
    thing.get()
  File "/usr/lib/python2.6/site-packages/keystoneclient/openstack/common/apiclient/base.py", line 464, in __getattr__
    raise AttributeError(k)
AttributeError: get

2014-07-23 06:23:54 DEBUG (test) Sleeping for 10 seconds
2014-07-23 06:24:04 DEBUG (nmanager) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 358, in is_deletion_complete
    thing.get()
  File "/usr/lib/python2.6/site-packages/keystoneclient/openstack/common/apiclient/base.py", line 464, in __getattr__
    raise AttributeError(k)
AttributeError: get

[Tatyanka (tatyana-leontovich)]

Actually it is not the problem - Some times instance hang to delete - so we need to delete it several times in case if we receive 404 code - we know that instace deleted succesfully