Ironic related OSTF tests failed

[Sergey Novikov]

Detailed bug description:
 The issue was found by
https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.ironic_deploy_swift/149/testReport/(root)/ironic_deploy_swift/ironic_deploy_swift/

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.ironic_deploy_ceph/149/testReport/(root)/ironic_deploy_ceph/ironic_deploy_ceph/

Steps to reproduce:
            1. Create cluster
            2. Add 1 node with controller role
            3. Add 2 nodes with ironic+ceph-osd role
            4. Add 1 node with compute+ceph-osd role
            5. Add 1 nodes with ironic role
            6. Run OSTF sanity check
Expected results: tests passed
Actual result: the following tests failed
 Check that Ironic can operate nodes
 Check that Ironic services are running
 List Ironic entities

There is a general error "Failed to discover service /usr/bin/ironic-conductor within specified timeout" http://paste.openstack.org/show/586211/

ironic-conductor.log gives interesting details http://paste.openstack.org/show/586215/.
For nore deatils please take a look at attached logs

[Oleksiy Molchanov (omolchanov)]

https://review.openstack.org/#/c/390514/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/390514
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=25743cba14fabcf9d1dde85e52b2be1c5813012f
Submitter: Jenkins
Branch: master

commit 25743cba14fabcf9d1dde85e52b2be1c5813012f
Author: Oleksiy Molchanov <email address hidden>
Date: Tue Oct 25 13:42:29 2016 +0300

    Fix dependency for ironic-conductor

    Change 'ironic' to 'ironic-api', as we do not have such task.

    Change-Id: Ie6b322f778e9f6b84bfc0e473c33613eca31f565
    Closes-Bug: 1634539

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/390516
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=5f37c3b5ec6e497c524eb2215ddde65e3f1c51a0
Submitter: Jenkins
Branch: stable/mitaka

commit 5f37c3b5ec6e497c524eb2215ddde65e3f1c51a0
Author: Oleksiy Molchanov <email address hidden>
Date: Tue Oct 25 13:42:29 2016 +0300

    Fix dependency for ironic-conductor

    Change 'ironic' to 'ironic-api', as we do not have such task.

    Change-Id: Ie6b322f778e9f6b84bfc0e473c33613eca31f565
    Closes-Bug: 1634539

[Ekaterina Shutova (eshutova)]

All latest runs are passed, problem seems fixed:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ironic_deploy_ceph/122/
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ironic_deploy_swift/122/

[TatyanaGladysheva (tgladysheva)]

Issue is reproduced on 10.0-1135 for the following tests:

1) https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.ironic_deploy_ceph/159/testReport/(root)/ironic_deploy_ceph/ironic_deploy_ceph/

Actual result:
the following OSTF tests failed:
 Check that Ironic can operate nodes
 Check that Ironic services are running
 List Ironic entities

2) https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.ironic_deploy_swift/158/testReport/(root)/ironic_deploy_swift/ironic_deploy_swift/

Actual result:
the following OSTF test failed:
 Check that Ironic services are running

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/421364

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421364
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=5786220bddad61600c4761480fc5b78bfb0e456a
Submitter: Jenkins
Branch: stable/newton

commit 5786220bddad61600c4761480fc5b78bfb0e456a
Author: Oleksiy Molchanov <email address hidden>
Date: Tue Oct 25 13:42:29 2016 +0300

    [sync/mitaka] Fix dependency for ironic-conductor

    Change 'ironic' to 'ironic-api', as we do not have such task.

    Change-Id: Ie6b322f778e9f6b84bfc0e473c33613eca31f565
    Closes-Bug: 1634539

[Yury Tregubov (ytregubov)]

The problem seems to be fixed for swift on mos 10.0 iso #1292.
However for ceph it is reproduced on that baseline:
https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.ironic_deploy_ceph/166/testReport/(root)/ironic_deploy_ceph/ironic_deploy_ceph/

Diagnistic snapshot is: https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.ironic_deploy_ceph/166/artifact/logs/fail_error_ironic_deploy_ceph-fuel-snapshot-2017-01-29_23-38-22.tar

[Oleksiy Molchanov (omolchanov)]

Yury, this is a brand new bug - https://bugs.launchpad.net/fuel/+bug/1665390

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

[Vladimir Khlyunev (vkhlyunev)]

https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.plugins.fuel_plugin_example/
Ironic is fine, verified