master|mitaka.fuel-ostf.pkgs.ubuntu.gate_ostf_update failed on deploying ceph task

[Artem Hrechanychenko]

Detailed bug description:
  File "/home/jenkins/workspace/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/fuel-qa/fuelweb_test/helpers/decorators.py", line 121, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/fuel-qa/gates_tests/tests/test_review_in_ostf.py", line 83, in gate_ostf_ceph_ha
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/fuel-qa/fuelweb_test/helpers/decorators.py", line 458, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/fuel-qa/fuelweb_test/helpers/decorators.py", line 443, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/fuel-qa/fuelweb_test/helpers/decorators.py", line 494, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/fuel-qa/fuelweb_test/helpers/decorators.py", line 501, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/fuel-qa/fuelweb_test/helpers/decorators.py", line 385, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 908, in deploy_cluster_wait
    self.assert_task_success(task, interval=interval, timeout=timeout)
  File "/home/jenkins/workspace/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 330, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[ceph_ready_check/1] Stopping the deployment process!'

Steps to reproduce:
    Scenario:
            1. Create cluster
            2. Add 3 nodes with controller and ceph OSD roles
            3. Add 2 nodes with compute
            4. Deploy the cluster <<<<<<failed here
            5. Run OSTF

Expected results:
 deployment successfully

Actual result:
 Failed tasks: Task[ceph_ready_check/1]

Reproducibility:
 https://ci.fuel-infra.org/view/deployment%20tests/job/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/505/console
Workaround:
 -
Impact:
 deployment tests for fuel-ostf repository

[Artem Hrechanychenko (agrechanichenko)]

Last 4 runs of this job don't have such problems
https://ci.fuel-infra.org/job/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/510/console
https://ci.fuel-infra.org/job/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/511/console
https://ci.fuel-infra.org/job/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/512/console
https://ci.fuel-infra.org/job/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/513/console
https://ci.fuel-infra.org/job/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/514/console

so move to invalid

[Artem Hrechanychenko (agrechanichenko)]

P.S.
Typo error, not invalid - Incomplete

[Artem Hrechanychenko (agrechanichenko)]

Verified on 261 iso with latest fuel-qa code
https://custom-ci.infra.mirantis.net/job/10.0.custom.packages_test.ubuntu/58/console
Status - Pass

but on https://ci.fuel-infra.org/job/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/517/console we see Failed again.
Failed task - AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[ceph_ready_check/4] Stopping the deployment process!'

In this deployment test we only update fuel-ostf, so ostf cannot impact on deployment proccess.

[Oleksiy Molchanov (omolchanov)]

Diagnostic snapshot is missing here and on CI server. Marking as Incomplete.

[Artem Hrechanychenko (agrechanichenko)]

New failure on mitaka job
https://custom-ci.infra.mirantis.net/job/10.0.custom.packages_test.ubuntu/68/