issue replacing the pacemaker cib during an update

[Marios Andreou]

there is more context @ https://bugzilla.redhat.com/show_bug.cgi?id=1285485 but during an overcloud node update and when trying to push the new cib like at https://github.com/openstack/tripleo-heat-templates/blob/bce5f65f2e036f6ef5232fdfc8025d7f864faa56/extraconfig/tasks/yum_update.sh#L131 we sometimes see an error like:

Dec 16 04:53:40 overcloud-controller-2 os-collect-config: Error: unable to push cib
Dec 16 04:53:40 overcloud-controller-2 os-collect-config: Call cib_replace failed (-205): Update was older than existing configuration

even though it is reported that

Dec 16 04:53:40 overcloud-controller-2 os-collect-config: Finished yum_update.sh on server 79060f60-432d-4022-b1ae-90974e17adf9 at Wed Dec 16 04:53:40 EST 2015

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/259009
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ac7467bacddb98fbb9fc9716ba56cad0d6f282de
Submitter: Jenkins
Branch: master

commit ac7467bacddb98fbb9fc9716ba56cad0d6f282de
Author: Jiri Stransky <email address hidden>
Date: Thu Dec 17 14:40:15 2015 +0100

    Wait for cluster to settle in yum_update.sh

    Occasionally we hit "Error: unable to push cib" during update. This is
    probably due to the fact that when we try to replace cib in
    yum_update.sh, services on the previous updated controller are still
    coming up and changing cib, and racing/conflicting with the cib push
    from yum_update.sh.

    This commit adds waiting for the cluster to settle before exiting from
    yum_update.sh, to avoid this kind of conflict.

    Also a check for cib-push success is added, to make the update fail
    properly instead of hanging indefinitely as we've observed with this
    issue.

    Change-Id: I953087e0e565474ac553fd57bea2459d2e3a6081
    Closes-Bug: #1527644

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/263673

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/263673
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3ec6ef46e87456a820bd21d6ba475accc19dc502
Submitter: Jenkins
Branch: stable/liberty

commit 3ec6ef46e87456a820bd21d6ba475accc19dc502
Author: Jiri Stransky <email address hidden>
Date: Thu Dec 17 14:40:15 2015 +0100

    Wait for cluster to settle in yum_update.sh

    Occasionally we hit "Error: unable to push cib" during update. This is
    probably due to the fact that when we try to replace cib in
    yum_update.sh, services on the previous updated controller are still
    coming up and changing cib, and racing/conflicting with the cib push
    from yum_update.sh.

    This commit adds waiting for the cluster to settle before exiting from
    yum_update.sh, to avoid this kind of conflict.

    Also a check for cib-push success is added, to make the update fail
    properly instead of hanging indefinitely as we've observed with this
    issue.

    Change-Id: I953087e0e565474ac553fd57bea2459d2e3a6081
    Closes-Bug: #1527644
    (cherry picked from commit ac7467bacddb98fbb9fc9716ba56cad0d6f282de)