[OSTF] Sahara OSTF test failed with NotFound: Instance could not be found (HTTP 404)

[Andrey Sledzinskiy]

http://jenkins-product.srt.mirantis.net:8080/view/0_0_swarm/job/master_fuelmain.system_test.centos.services_simple/23/testReport/junit/%28root%29/deploy_savanna_simple/deploy_savanna_simple/

Test failed trying to delete server but it was already deleted:

2014-06-15 23:17:16 DEBUG (test_nova_create_instance_with_connectivity) Server was already deleted.
2014-06-15 23:17:16 DEBUG (nmanager) Deleting server.
2014-06-15 23:17:16 INFO (connectionpool) Starting new HTTP connection (1): 10.108.5.3
2014-06-15 23:17:16 DEBUG (connectionpool) "DELETE http://10.108.9.2:8774/v2/1f834d47763f4404a73c85ff0f4690a6/servers/cef3c534-5579-4ed1-98aa-928d67a76c7d HTTP/1.1" 404 73
2014-06-15 23:17:16 DEBUG (test_nova_create_instance_with_connectivity) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/tests/smoke/test_nova_create_instance_with_connectivity.py", line 74, in tearDown
    self._delete_server(server)
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 280, in _delete_server
    self.compute_client.servers.delete(server)
  File "/usr/lib/python2.6/site-packages/novaclient/v1_1/servers.py", line 901, in delete
    self._delete("/servers/%s" % base.getid(server))
  File "/usr/lib/python2.6/site-packages/novaclient/base.py", line 161, in _delete
    _resp, _body = self.api.client.delete(url)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 292, in delete
    return self._cs_request(url, 'DELETE', **kwargs)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 260, in _cs_request
    **kwargs)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 242, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 236, in request
    raise exceptions.from_response(resp, body, url, method)
NotFound: Instance could not be found (HTTP 404) (Request-ID: req-169fb8c5-6be2-4099-b008-abea2322a215)

Snapshot is attached

[Tatyanka (tatyana-leontovich)]

Actually not. This trace that you post do not fail the tests. It ia additional verification that cluster is really delete.
And actually you can see in logs that test failed with next error
fuel_health.savanna: DEBUG: CLUSTER STATUS:3550 sec:Starting
urllib3.connectionpool: INFO: Starting new HTTP connection (1): 10.108.5.3
urllib3.connectionpool: DEBUG: "GET http://10.108.5.3:8386/v1.0/1f834d47763f4404a73c85ff0f4690a6/clusters/16a105de-465e-49f0-a6da-b080e9ce18c1 HTTP/1.1" 401 23
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 186, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuel_health/savanna.py", line 339, in create_sahara_cluster
    neutron_management_network=neutron_m_network_id)
  File "/usr/lib/python2.6/site-packages/fuel_health/savanna.py", line 215, in _create_cluster_and_get_info
    self._check_cluster_state(body.id)
  File "/usr/lib/python2.6/site-packages/fuel_health/savanna.py", line 185, in _check_cluster_state
    data = self.savanna_client.clusters.get(cluster_id)
  File "/usr/lib/python2.6/site-packages/saharaclient/api/clusters.py", line 71, in get
    return self._get('/clusters/%s' % cluster_id, 'cluster')
  File "/usr/lib/python2.6/site-packages/saharaclient/api/base.py", line 120, in _get
    self._raise_api_exception(resp)
  File "/usr/lib/python2.6/site-packages/saharaclient/api/base.py", line 132, in _raise_api_exception
    error_data = get_json(resp)
  File "/usr/lib/python2.6/site-packages/saharaclient/api/base.py", line 145, in get_json
    return response.json()
  File "/usr/lib/python2.6/site-packages/requests/models.py", line 651, in json
    return json.loads(self.text or self.content, **kwargs)
  File "/usr/lib64/python2.6/site-packages/simplejson/__init__.py", line 488, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.6/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python2.6/site-packages/simplejson/decoder.py", line 389, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

--------------------- >> end captured logging << ---------------------
2014-06-16 00:19:49 INFO (connectionpool) Starting new HTTP connection (1): 10.108.5.3
2014-06-16 00:19:49 ERROR (nose_storage_plugin) <nose.suite.ContextSuite context=PlatformSavannaTests>:teardown
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nose/suite.py", line 227, in run
    self.tearDown()
  File "/usr/lib/python2.6/site-packages/nose/suite.py", line 350, in tearDown
    self.teardownContext(ancestor)
  File "/usr/lib/python2.6/site-packages/nose/suite.py", line 366, in teardownContext
    try_run(context, names)
  File "/usr/lib/python2.6/site-packages/nose/util.py", line 469, in try_run
    return func()
  File "/usr/lib/python2.6/site-packages/fuel_health/savanna.py", line 587, in tearDownClass
    cls._clean_clusters()
  File "/usr/lib/python2.6/site-packages/fuel_health/savanna.py", line 567, in _clean_clusters
    cls...

[Tatyanka (tatyana-leontovich)]

actually latest run says that all good with deletion(Btw we still need to update expected fails value- but as I remeber it is a different issue)
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services_simple/fuelweb_test/helpers/decorators.py", line 51, in wrapper
    return func(*args, **kwagrs)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services_simple/fuelweb_test/tests/test_services.py", line 126, in deploy_savanna_simple
    timeout=60 * 200, failed_test_name=failed_test_name)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services_simple/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services_simple/fuelweb_test/models/fuel_web_client.py", line 533, in run_single_ostf_test
    failed_test_name=failed_test_name)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services_simple/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services_simple/fuelweb_test/models/fuel_web_client.py", line 177, in assert_ostf_run
    failed_tests_res))
  File "/usr/lib/python2.7/dist-packages/proboscis/asserts.py", line 163, in assert_true
    raise ASSERTION_ERROR(message)
AssertionError: Failed tests, fails: 3 should fail: 1 failed tests name: [{u'Sahara tests to create, list, start, checks and delete cluster': u''}, {u'Create volume and boot instance from it': u'There are no cinder nodes or ceph storage for volume'}, {u'Create volume and attach it to instance': u'There are no cinder nodes or ceph storage for volume'}]

[Dmitry Mescheryakov (dmitrymex)]

The bug is incomplete, so no reason to have someone assigned to it

[Tatyanka (tatyana-leontovich)]

Guys, lets move it in invalid based on next criteria:
Nobody can not confirmed it during 2 month
CI history here http://jenkins-product.srt.mirantis.net:8080/view/0_master_swarm/job/master_fuelmain.system_test.centos.services_simple/ shows that sahara test(include ostf) passed