[System test] False negative test result when we can't restore successful snapshot

[Dmitry Tyzhnenko]

False negative result on test - 'Deploy cluster in HA mode, Neutron with GRE segmentation'

http://jenkins-product.srt.mirantis.net:8080/job/6.1.system_test.ubuntu.thread_5/24/

{
    u'build_id': u'2015-02-03_22-55-01',
    u'ostf_sha': u'c9100263140008abfcc2704732e98fbdfd644068',
    u'build_number': u'96',
    u'auth_required': True,
    u'nailgun_sha': u'62dd62897850795fa35d2f359cf4f310d33f65c5',
    u'production': u'docker',
    u'api': u'1.0',
    u'python-fuelclient_sha': u'2ea7b3e91c1d2ff85110bf5abb161a6f4e537358',
    u'astute_sha': u'ed5270bf9c6c1234797e00bd7d4dd3213253a413',
    u'fuelmain_sha': u'',
    u'feature_groups': [u'mirantis'],
    u'release': u'6.1',
    u'release_versions': {u'2014.2-6.1': {u'VERSION': {
        u'build_id': u'2015-02-03_22-55-01',
        u'ostf_sha': u'c9100263140008abfcc2704732e98fbdfd644068',
        u'build_number': u'96',
        u'api': u'1.0',
        u'nailgun_sha': u'62dd62897850795fa35d2f359cf4f310d33f65c5',
        u'production': u'docker',
        u'python-fuelclient_sha': u'2ea7b3e91c1d2ff85110bf5abb161a6f4e537358',
        u'astute_sha': u'ed5270bf9c6c1234797e00bd7d4dd3213253a413',
        u'feature_groups': [u'mirantis'],
        u'release': u'6.1',
        u'fuelmain_sha': u'',
        u'fuellib_sha': u'2147da0c583a7944f440ceb51236e7cb2e6610c9',
        }}},
    u'fuellib_sha': u'2147da0c583a7944f440ceb51236e7cb2e6610c9',
    }

When we can't connect to master node after revert successful snapshot we can raise false negative result in test

2015-02-04 05:20:37,548 - INFO fuel_web_client.py:842 -- Update network settings of cluster 1
2015-02-04 05:20:42,295 - INFO fuel_web_client.py:436 -- Deploy cluster 1
2015-02-04 05:20:42,295 - INFO fuel_web_client.py:454 -- Launch deployment of a cluster #1
2015-02-04 05:20:47,869 - INFO fuel_web_client.py:232 -- Assert task {u'status': u'running', u'name': u'deploy', u'cluster': 1, u'result': {}, u'progress': 0, u'message': None, u'id': 2, u'uuid': u'98f1a371-5181-49ae-8aa1-c2b170ab1190'} is success
2015-02-04 05:20:47,869 - INFO fuel_web_client.py:623 -- Wait for task {u'status': u'running', u'name': u'deploy', u'cluster': 1, u'result': {}, u'progress': 0, u'message': None, u'id': 2, u'uuid': u'98f1a371-5181-49ae-8aa1-c2b170ab1190'} 7800 seconds
2015-02-04 06:21:03,694 - INFO decorators.py:143 -- <<<<<****************************************************************************************************>>>>>
2015-02-04 06:21:03,694 - INFO decorators.py:144 -- Make snapshot: deploy_ha_neutron
2015-02-04 06:21:03,695 - INFO decorators.py:153 -- You could revert this snapshot using [dos.py revert 6.1.system_test.ubuntu.thread_5.24.2015-02-04_02-01-12 --snapshot-name deploy_ha_neutron && dos.py resume 6.1.system_test.ubuntu.thread_5.24.2015-02-04_02-01-12 && virsh net-dumpxml 6.1.system_test.ubuntu.thread_5.24.2015-02-04_02-01-12_admin | grep -P "(\d+\.){3}" -o | awk '{print "Admin node IP: "$0"2"}']
2015-02-04 06:21:03,695 - INFO decorators.py:158 -- <<<<<****************************************************************************************************>>>>>
2015-02-04 06:22:11,248 - ERROR environment.py:387 -- Admin node is unavailable via SSH after environment resume
2015-02-04 06:22:14,250 - ERROR decorators.py:80 -- Fetching of diagnostic snapshot failed: Traceback (most recent call last):
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_5/fuelweb_test/helpers/decorators.py", line 77, in wrapper
    "fail", name)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_5/fuelweb_test/helpers/decorators.py", line 193, in create_diagnostic_snapshot
    task = env.fuel_web.task_wait(env.fuel_web.client.generate_logs(), 60 * 5)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_5/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_5/fuelweb_test/helpers/decorators.py", line 103, in wrapped
    response = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_5/fuelweb_test/models/nailgun_client.py", line 294, in generate_logs
    return self.client.put("/api/logs/package")
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_5/fuelweb_test/helpers/http.py", line 83, in put
    return self._open(req)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_5/fuelweb_test/helpers/http.py", line 92, in _open
    return self._get_response(req)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_5/fuelweb_test/helpers/http.py", line 109, in _get_response
    return self.opener.open(req)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 113] No route to host>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/152975

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/152975
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=60d274f2d1006a4b10ec647c3318e4961b9a2cba
Submitter: Jenkins
Branch: master

commit 60d274f2d1006a4b10ec647c3318e4961b9a2cba
Author: Dennis Dmitriev <email address hidden>
Date: Wed Feb 4 18:37:13 2015 +0200

    Hotfix for revert admin node after suspending

    Remove resuming the environment after all snapshots due to
    a lot of failures.

    Move the method 'store_astute_yaml' to the separate decorator
    to be used with methods that perform deploy a cluster.

    Change-Id: Icbc81170697d7a5aa85c16c50a42563fe7fbcd60
    Closes-Bug: #1418111

[Dennis Dmitriev (ddmitriev)]

Fix released.

ISO version: {u'build_id': u'2015-02-07_22-55-01', u'ostf_sha': u'6c046b69d29021524906109f18092363505ee222', u'build_number': u'105', u'auth_required': True, u'nailgun_sha': u'6d1769b21819f8fb4195f1bd9c44c038721ae3d4', u'production': u'docker', u'api': u'1.0', u'python-fuelclient_sha': u'521c2491f7f04f31d8c85db68499cd193d4904e3', u'astute_sha': u'7e6e6f9188bd69c603853b10d4a55149363323cc', u'fuelmain_sha': u'', u'feature_groups': [u'mirantis'], u'release': u'6.1', u'release_versions': {u'2014.2-6.1': {u'VERSION': {u'build_id': u'2015-02-07_22-55-01', u'ostf_sha': u'6c046b69d29021524906109f18092363505ee222', u'build_number': u'105', u'api': u'1.0', u'nailgun_sha': u'6d1769b21819f8fb4195f1bd9c44c038721ae3d4', u'production': u'docker', u'python-fuelclient_sha': u'521c2491f7f04f31d8c85db68499cd193d4904e3', u'astute_sha': u'7e6e6f9188bd69c603853b10d4a55149363323cc', u'feature_groups': [u'mirantis'], u'release': u'6.1', u'fuelmain_sha': u'', u'fuellib_sha': u'769af7fe30225cd15638ea2e6dffaa286bc06da1'}}}, u'fuellib_sha': u'769af7fe30225cd15638ea2e6dffaa286bc06da1'}