Deploy failed with AssertionError: Task 'verify_networks' has incorrect status. error != ready

[Alexander Ignatov]

Systest job failed with error during the deployment stage:

======================================================================
FAIL: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 83, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/tests/test_ceph.py", line 493, in ceph_rados_gw
    self.fuel_web.verify_network(cluster_id)
  File "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 341, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/__init__.py", line 57, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 1324, in verify_network
    self.assert_task_success(task, timeout, interval=10)
  File "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/__init__.py", line 57, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 320, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
AssertionError: Task 'verify_networks' has incorrect status. error != ready, 'Method verify_networks. Network verification not avaliable because nodes ["1", "2", "3", "4", "5", "6"] not avaliable via mcollective.
Inspect Astute logs for the details'

Job link:
https://packaging-ci.infra.mirantis.net/job/8.0-pkg-systest-ubuntu/2028/console

[Maksim Malchuk (mmalchuk)]

duplicate https://bugs.launchpad.net/fuel/+bug/1537678

[Alexander Ignatov (aignatov)]

Critical because this job is just always RED https://packaging-ci.infra.mirantis.net/job/8.0-pkg-systest-ubuntu/

[Maksim Malchuk (mmalchuk)]

correct link: https://bugs.launchpad.net/fuel/+bug/1537386

[Albert Syriy (asyriy)]

Could you please provide(show) the interface names from the failed bootstraped nodes?
I have a hunch, that issue may be connected to Rados_gw, but want to be sure

[Maksim Malchuk (mmalchuk)]

@asyriy, network assignment is passed and it it looks line mcollective issue:

2016-01-25 14:12:54,229 - INFO fuel_web_client.py:1361 -- Assigned networks are: {'enp0s6': ['private'], 'enp0s7': ['storage'], 'enp0s4': ['public'], 'enp0s5': ['management'], 'enp0s3': ['fuelweb_admin']}
2016-01-25 14:12:54,770 - INFO fuel_web_client.py:1014 -- Run network verification on the cluster 1
2016-01-25 14:13:15,401 - ERROR fuel_web_client.py:1312 -- Network verification failed: