deadlock can occur in between clustercheck vs ceph-mon setup in pacemaker scenario

[Giulio Fidente]

During step2, in pacemaker scenario, all nodes will try to perform the ceph-mon setup which blocks puppet until all initial members are available when forming the ceph cluster.

On the bootstrap node this can happen *before* galera setup is initialized but if in the same deployment, on non-bootstrap nodes clustercheck is launched first, the deployment will stop in a deadlock where:

1 ceph-mon on bootstrap node is waiting for the non-bootstrap nodes
2 clustercheck on non-bootstrap nodes is waiting galera to come up on the bootstrap nodes

The clustercheck resource should really only be used on bootstrap nodes.

[Giulio Fidente (gfidente)]

I have a WIP patch at https://review.openstack.org/#/c/337302/1

More infos on the BZ can also be found in https://bugzilla.redhat.com/show_bug.cgi?id=1349456

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/337302
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3b9544b265ff239225ef7a23986182019a14d34b
Submitter: Jenkins
Branch: master

commit 3b9544b265ff239225ef7a23986182019a14d34b
Author: Giulio Fidente <email address hidden>
Date: Mon Jul 4 18:22:19 2016 +0200

    Merge pacemaker_master/sync_db conditionals

    By condensing the pacemaker_master and sync_db conditions we ensure
    there won't be unrelevant (clustercheck) execs deployed on
    non-bootstrap nodes.

    Closes-Bug: 1598907

    Change-Id: Iae6aa13682d63096265f4751b2f71019a49f6fa6

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/339444

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/339444
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=64d70eb2f61de3ab1e47f54c932b4421da4a816a
Submitter: Jenkins
Branch: stable/mitaka

commit 64d70eb2f61de3ab1e47f54c932b4421da4a816a
Author: Giulio Fidente <email address hidden>
Date: Fri Jul 8 12:04:16 2016 +0200

    Merge pacemaker_master/sync_db conditionals

    By condensing the pacemaker_master and sync_db conditions we ensure
    there won't be unrelevant (clustercheck) execs deployed on
    non-bootstrap nodes.

    Closes-Bug: 1598907
    Change-Id: Iae6aa13682d63096265f4751b2f71019a49f6fa6
    (cherry picked from commit 3b9544b265ff239225ef7a23986182019a14d34b)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/tripleo-heat-templates 2.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 5.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 2.1.0 release.