[system-tests] Need to wait that nodes are ready if revert snapshot with slaves

Bug #1355660 reported by Andrey Sledzinskiy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Tatyanka

Bug Description

http://jenkins-product.srt.mirantis.net:8080/view/0_master_swarm/job/master_fuelmain.system_test.ubuntu.thread_3/130/testReport/junit/%28root%29/ceph_multinode_restart/ceph_multinode_restart/

After revert nodes aren't ready and we try to restart it so get 'No route to host'
Need to add additional check if we revert snapshot with slaves that these slaves are ready too

Error Message

[Errno 113] No route to host

Stacktrace

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/master_fuelmain.system_test.ubuntu.thread_3/fuelweb_test/helpers/decorators.py", line 52, in wrapper
    return func(*args, **kwagrs)
  File "/home/jenkins/workspace/master_fuelmain.system_test.ubuntu.thread_3/fuelweb_test/tests/tests_strength/test_restart.py", line 61, in ceph_multinode_restart
    self.fuel_web.warm_restart_nodes(self.env.nodes().slaves[:4])
  File "/home/jenkins/workspace/master_fuelmain.system_test.ubuntu.thread_3/fuelweb_test/models/fuel_web_client.py", line 857, in warm_restart_nodes
    remote = self.get_ssh_for_node(node.name)
  File "/home/jenkins/workspace/master_fuelmain.system_test.ubuntu.thread_3/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master_fuelmain.system_test.ubuntu.thread_3/fuelweb_test/models/fuel_web_client.py", line 501, in get_ssh_for_node
    return self.environment.get_ssh_to_remote(ip)
  File "/home/jenkins/workspace/master_fuelmain.system_test.ubuntu.thread_3/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master_fuelmain.system_test.ubuntu.thread_3/fuelweb_test/models/environment.py", line 287, in get_ssh_to_remote
    private_keys=self.get_private_keys())
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 160, in __init__
    self.reconnect()
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 191, in reconnect
    self.connect()
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 28, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 181, in connect
    password=self.password, pkey=private_key)
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/paramiko/client.py", line 236, in connect
    retry_on_signal(lambda: sock.connect(addr))
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/paramiko/util.py", line 278, in retry_on_signal
    return function()
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/paramiko/client.py", line 236, in <lambda>
    retry_on_signal(lambda: sock.connect(addr))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 113] No route to host

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Changed in fuel:
status: New → Confirmed
Changed in fuel:
importance: High → Medium
Changed in fuel:
status: Confirmed → Triaged
milestone: 5.1 → 6.0
Changed in fuel:
milestone: 6.0 → 6.1
summary: - [System tests] Need to wait that nodes are ready if revert snapshot with
+ [system-tests] Need to wait that nodes are ready if revert snapshot with
slaves
Changed in fuel:
importance: Medium → High
Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Tatyanka (tatyana-leontovich)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/178168

tags: added: non-release
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/178168
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=d506b57798d0bf3cb90f4aa51592dd193448bd33
Submitter: Jenkins
Branch: master

commit d506b57798d0bf3cb90f4aa51592dd193448bd33
Author: Tatyana Leontovich <email address hidden>
Date: Mon Apr 27 15:26:02 2015 +0300

    Check slaves are online in nailgun after revert

    Closes-Bug: #1355660

    Change-Id: I409db14844ea57cf8dd4552b21df931c0658f9ca

Changed in fuel:
status: In Progress → Fix Committed
Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.