[crm] Deployment of controller node fails: "Error: Unable to update cib; Call cib_replace failed (-206): Application of an update diff failed"

[Artem Panchenko]

api: '1.0'
astute_sha: 0085021fe327f6f910901b3ca55051b1df33a96e
auth_required: true
build_id: 2014-11-13_22-57-40
build_number: '106'
feature_groups:
- mirantis
fuellib_sha: 045f2f77f35b79b0321c92882f39203f315faabb
fuelmain_sha: 4614db0701a8add2f60b8fffdc72f106621cbce2
nailgun_sha: ba5604db9424319e002dd9b5deafe28d31036840
ostf_sha: 82465a94eed4eff1fc8d8e1f2fb7e9993c22f068
production: docker
release: '6.0'

Environment deployment failed due to errors in puppet on 2nd controller:

http://jenkins-product.srt.mirantis.net:8080/job/6.0.centos.bvt_1/85/testReport/junit/(root)/deploy_neutron_vlan_ha/deploy_neutron_vlan_ha/

http://paste.openstack.org/show/133105/

I tried to run the same 'pcs' commands manually and they failed few times:

http://paste.openstack.org/show/133108/

Also, re-deployment of controller fixed the issue with deployment (error from pacemaker wasn't reproduced again), so I beleive that this bug is floating.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/134534

[Dennis Dmitriev (ddmitriev)]

The same issue reproduced on another CI job, details in the https://bugs.launchpad.net/fuel/+bug/1283062

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/134534
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=ee7f5f743a96351085a0c8049cdb526f24a0eeee
Submitter: Jenkins
Branch: master

commit ee7f5f743a96351085a0c8049cdb526f24a0eeee
Author: Dmitry Ilyin <email address hidden>
Date: Fri Nov 14 16:51:32 2014 +0300

    Add retries to pcs resource cleanup

    'pcs resource cleanup' command used in provider
    invokes 'crm_resource' which could fail some times
    with 'No such device' error.
    In order to fix this, the common approach is used
    as well: retry failed operation for given timeout

    Change-Id: I8a07613ef89b67a019d7e38a0432b4b1ca773c9b
    Closes-Bug: 1392722

[Nastya Urlapova (aurlapova)]

{
build_id: "2014-11-20_22-01-01",
ostf_sha: "8be702d5cafcb78ac1d93e5fc27d9272cce4773e",
build_number: "122",
auth_required: true,
api: "1.0",
nailgun_sha: "e4c59607c852f1b3761ef06c9c348994c98a7269",
production: "docker",
fuelmain_sha: "3a6833b3e5223c8eacadb1a0a22600813419e9a9",
astute_sha: "46d23177f9c0e53fb7a4729f394d9841b1bae338",
feature_groups: [
"mirantis"
],
release: "6.0",
release_versions: {
2014.2-6.0: {
VERSION: {
build_id: "2014-11-20_22-01-01",
ostf_sha: "8be702d5cafcb78ac1d93e5fc27d9272cce4773e",
build_number: "122",
api: "1.0",
nailgun_sha: "e4c59607c852f1b3761ef06c9c348994c98a7269",
production: "docker",
fuelmain_sha: "3a6833b3e5223c8eacadb1a0a22600813419e9a9",
astute_sha: "46d23177f9c0e53fb7a4729f394d9841b1bae338",
feature_groups: [
"mirantis"
],
release: "6.0",
fuellib_sha: "666c40d0f5c0bb3dbeaa02b7ba62d038453bae0f"
}
}
},
fuellib_sha: "666c40d0f5c0bb3dbeaa02b7ba62d038453bae0f"
}

Scenario:
            1. Create cluster
            2. Add 3 nodes with controller role
            3. Add 2 nodes with compute role
            4. Enable assign public networks to all nodes option
            5. Deploy the cluster
            6. Check that public network was assigned to all nodes
            7. Run network verification
            8. Run OSTF

Deployment failed(node-5):
 Execution of '/usr/sbin/pcs resource meta p_neutron-l3-agent target-role=Started' returned 1: Error: Unable to update cib

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/136385

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/136385
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=0dbe3724daa7edd5b51f7a1693873ed5d6e0d078
Submitter: Jenkins
Branch: master

commit 0dbe3724daa7edd5b51f7a1693873ed5d6e0d078
Author: Dmitry Ilyin <email address hidden>
Date: Fri Nov 21 18:24:35 2014 +0300

    Wrap all pcs commands into retry blocks

    Sometimes Pacemaker commands fail for unknown
    reason but retry helps because they work next time
    they are ran.

    Some rspec fixes

    Change-Id: Ieca6935df9de641b1c2f2c88dce3ce765b679812
    Closes-Bug: 1392722

[Tatyana Dubyk (tdubyk)]

Verified on 6.0-56 iso