[10.0] Task ceph_create_pools failed

[Dmitry Belyaninov]

There is faled acceptance test:

https://product-ci.infra.mirantis.net/job/10.0.acceptance.ubuntu.sahara_os_component/3/testReport/(root)/sahara_ceph_ephemeral/sahara_ceph_ephemeral/

Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[ceph_create_pools/3] Stopping the deployment process!'

snapshot https://drive.google.com/open?id=0B1CktchMwAXHcEU5WkwxX3FjQUU

[Oleksiy Molchanov (omolchanov)]

We need env to debug this issue, there is no meaningful info in logs.

[Dmitry Belyaninov (dbelyaninov)]

https://product-ci.infra.mirantis.net/job/10.0.acceptance.ubuntu.ha_scale_group_4/6/testReport/(root)/add_delete_cinder_ceph/add_delete_cinder_ceph/

[Dmitry Belyaninov (dbelyaninov)]

https://product-ci.infra.mirantis.net/job/10.0.acceptance.ubuntu.ha_scale_group_5/6/testReport/(root)/add_delete_compute_cinder_ceph/add_delete_compute_cinder_ceph/

[Kyrylo Romanenko (kromanenko)]

This bug was reproduced today in following job: https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.bvt_2/1418/

[Kostiantyn Danylov (kdanylov)]

Working on it

[Kostiantyn Danylov (kdanylov)]

PG calculation is correct (well it's a gives a bit bigger numbers, than expected, but this is not a problem).
But puppet manifests, which install ceph, probably, instead of creating pool with correct PG creates it
with default PG and then increase PG count to required value. During this procedure error appears,
as PG count can't be increased so fast. I need help with puppet manifests to investigate and fix this.

[Kostiantyn Danylov (kdanylov)]

We have found that error happened, when FUEL updates PG during adding new OSD node to cluster. This should not happened, changing PG number is long process, which hardly affect performance,
and also can't be done in timely matter in some cases (e.g. in case of this test).
PG should be updated only manually by user, and only after new node is successfully joined the cluster and rebalance is finished.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/450969

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/451801

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/450969
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=b46dd24aac993e41507da70163055caeb574502f
Submitter: Jenkins
Branch: stable/newton

commit b46dd24aac993e41507da70163055caeb574502f
Author: Alexander Kislitsky <email address hidden>
Date: Tue Mar 28 23:26:44 2017 +0300

    Ceph pool creation should be called only once

    PG number should'n be updated on adding ceph-osd nodes.
    It should be done manually by user only after new node is
    sucessfully joined and rebalance is finished.

    Change-Id: I4ec7ab8b713fd488dd63e83491d1b60966420d00
    Closes-Bug: #1668553

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/452814

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/452815

[Ilya Bumarskov (ibumarskov)]

Verified on Fuel 10 build #1575
https://product-ci.infra.mirantis.net/job/10.0.acceptance.ubuntu.sahara_os_component/15/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/452814
Reason: This repo is retired now, no further work will get merged.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/newton
Review: https://review.opendev.org/452815
Reason: This repo is retired now, no further work will get merged.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/451801
Reason: This repo is retired now, no further work will get merged.