Provisioning timeout, "Nodes not booted:" (controller)

[Mykhaylo Slobodyan]

6.1 system_test - check_nodes_disks
Scenario:
            1. Create cluster in Ha mode with 1 controller
            2. Add 1 node with controller role
            3. Add 1 node with compute role
            4. Add 1 node with cinder role
            5. Deploy the cluster
            6. Verify hard drive sizes for deployed nodes

failed with:
Task 'deploy' has incorrect status. error != ready

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_1/fuelweb_test/helpers/decorators.py", line 64, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_1/fuelweb_test/tests/test_ha_one_controller.py", line 720, in check_nodes_disks
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_1/fuelweb_test/helpers/decorators.py", line 294, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_1/fuelweb_test/helpers/decorators.py", line 234, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_1/fuelweb_test/models/fuel_web_client.py", line 440, in deploy_cluster_wait
    self.assert_task_success(task, interval=interval)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_1/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.thread_1/fuelweb_test/models/fuel_web_client.py", line 239, in assert_task_success
    task['status'], 'ready', name=task["name"]
  File "/usr/lib/python2.7/dist-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: Task 'deploy' has incorrect status. error != ready

Astute log:
2015-02-11 09:14:54 ERR [397] Timeout of provisioning is exceeded. Nodes not booted: ["3"]

As a side note, on node-3 I see a bunch of:
ns_IPaddr2(vip__management)[(\d+)]: ERROR: Device "eth2-hapr" does not exist.
ns_IPaddr2(vip__public)[(\d+)]: ERROR: Device "eth1-hapr" does not exist.
Also, slave-03_cinder shows as provisioned, but I see it just booted in bootstrap.

Jenkins link: http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.ubuntu.thread_1/29/

[Vladimir Kuklin (vkuklin)]

This bug seems to be one month old and has no reproducers since then