Failed task ceph_ready_check during ironic_deploy_ceph test run

[Kyrylo Romanenko]

Test 9.0.system_test.ubuntu.ironic_deploy_ceph failed with error of ceph_ready_check task.

Actual result: test failed with Task[ceph_ready_check/1]
Traceback:
http://paste.openstack.org/show/495026/

Full job log: https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ironic_deploy_ceph/78/console

Diagnostic snapshot attached.

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Serge Kovaleff (serge-kovaleff)]

It looks for me as a Ceph glitch.

node-1.test.domain.local 2016-04-20T22:24:47.947755 err: (/Stage[main]/Ceph::Osds/Ceph::Osds::Osd[/dev/vdc3]/Exec[ceph-deploy osd prepare node-1:/dev/vdc3]/returns) change from notrun to 0 failed: ceph-deploy osd prepare node-1:/dev/vdc3 returned 1 instead of one of [0]

@ https://product-ci.infra.mirantis.net/view/9.0_swarm/job/9.0.system_test.ubuntu.ironic_deploy_ceph/78/artifact/logs/fail_error_ironic_deploy_ceph-fuel-snapshot-2016-04-20_23-09-55.tar.xz.filtered.log

[Dina Belova (dbelova)]

Moving to Confirmed state for now. Let's investigate the issue.

[Anastasia Kuznetsova (akuznetsova)]

Bug was reproduced again on ISO 242

[Timur Nurlygayanov (tnurlygayanov)]

The last job is green, so, it looks like the issue reproduced not in 100 % of cases:
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ironic_deploy_ceph/

[Oleksiy Molchanov (omolchanov)]

For 10.0 it is not valid, we have moved to puppet-ceph. For 9.0 I need an environment, because I cannot reproduce this and there is no info in logs I can successfully analyze.

[Kyrylo Romanenko (kromanenko)]

ironic_deploy_ceph job is fixed back after iso 246.

[Timur Nurlygayanov (tnurlygayanov)]

Does it mean that the issue was completely fixed?

[Anastasia Kuznetsova (akuznetsova)]

Bug was reproduced again on ISO 9.0-mos 416
System test "ironic_deploy_cehp" failed with following error:

"Error Message
Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[ceph_ready_check/1] Stopping the deployment process!'"

[Oleksiy Molchanov (omolchanov)]

https://review.openstack.org/#/c/318012/

[Oleksiy Molchanov (omolchanov)]

ETA - 2/06

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/318012
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=08e0381fd0bb9472fd2390b593d37f8747bb25fe
Submitter: Jenkins
Branch: stable/mitaka

commit 08e0381fd0bb9472fd2390b593d37f8747bb25fe
Author: Oleksiy Molchanov <email address hidden>
Date: Wed May 18 15:12:58 2016 +0300

    Fix for OSD activation

    We have a rare situation when after 'osd prepare' command marked osd
    as active but actually it is not up. This patch activate OSD after each
    time it is being prepared.

    Change-Id: Ic0e60ed196f9b8f876fe002f171d27878bac55a0
    Closes-Bug: 1573133

[Dmitriy Kruglov (dkruglov)]

After the fix was introduced the system test is successfully executed for a week (8 times in a row).
Moving issue to fixed state.