"Cobbler error" appears if two clusters are started to deploy in the same moment

[Dennis Dmitriev]

Steps to reproduce:
    1. Create two clusters with 1 controller and 1 compute in each,
    2. Open UI for both clusters in different tabs on your browser,
    3. Press "Deploy changes" in both tabs, but do not confirm yet.
    4. As quick as possible press "Deploy" button in both tabs to start deploy the clusters.

As the result: the first cluster where "Deploy" button was pressed, failed with the notification in UI "Cobbler error".
Provisioning of nodes hasn't even started.

{"build_id": "2014-11-17_13-47-05", "ostf_sha": "82465a94eed4eff1fc8d8e1f2fb7e9993c22f068", "build_number": "502", "auth_required": true, "api": "1.0", "nailgun_sha": "87100dca641d76fd560eac7f7894d9413d8c186d", "production": "docker", "fuelmain_sha": "b585c7082511936ca3ac27e7ed12d1e2386feb90", "astute_sha": "d4fd7048befd22feecf5ba40f17981df5b608621", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-17_13-47-05", "ostf_sha": "82465a94eed4eff1fc8d8e1f2fb7e9993c22f068", "build_number": "502", "api": "1.0", "nailgun_sha": "87100dca641d76fd560eac7f7894d9413d8c186d", "production": "docker", "fuelmain_sha": "b585c7082511936ca3ac27e7ed12d1e2386feb90", "astute_sha": "d4fd7048befd22feecf5ba40f17981df5b608621", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "0f51ec2c95ea031ef1190b86d336bee5779b7ed7"}}}, "fuellib_sha": "0f51ec2c95ea031ef1190b86d336bee5779b7ed7"}

[Dennis Dmitriev (ddmitriev)]

Workaround that was proposed in the review https://review.openstack.org/#/c/134929/ didn't help in this case.

[Dima Shulyak (dshulyak)]

@Dennis, it looks like that this bug is duplicate of https://bugs.launchpad.net/fuel/+bug/1387699

[Vladimir Sharshov (vsharshov)]

Looks like reproducing this bug: https://bugs.launchpad.net/fuel/+bug/1318716

Yes, nature of this bug also looks like cobbler overloading (maybe race condition in case of parallel operation).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/134929
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=0bcc91185984a1f9e4bd68d9f247266c35349dae
Submitter: Jenkins
Branch: master

commit 0bcc91185984a1f9e4bd68d9f247266c35349dae
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Mon Nov 17 16:01:14 2014 +0300

    Add retries in case of Cobbler overloading

    Offen if Cobbler run two big task in parallel
    we got error based on Cobbler overloading.
    If we wait and try request again, it will
    process without problem. This change
    implement such behavior.

    Change-Id: I1e9774b9398dfb502159d7ba21e7e5b55b75b3a1
    Closes-Bug: #1387699
    Closes-Bug: #1396181

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/137553

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/137553
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=ef8aa0fd0e3ce20709612906f1f0551b5682a6ce
Submitter: Jenkins
Branch: stable/5.1

commit ef8aa0fd0e3ce20709612906f1f0551b5682a6ce
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Mon Nov 17 16:01:14 2014 +0300

    Add retries in case of Cobbler overloading

    Offen if Cobbler run two big task in parallel
    we got error based on Cobbler overloading.
    If we wait and try request again, it will
    process without problem. This change
    implement such behavior.

    Change-Id: I1e9774b9398dfb502159d7ba21e7e5b55b75b3a1
    Closes-Bug: #1387699
    Closes-Bug: #1396181
    (cherry picked from commit 0bcc91185984a1f9e4bd68d9f247266c35349dae)

[Anastasia Palkina (apalkina)]

Cannot reproduce on latest ISOs for 6.0 #49 and #56

"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}}, "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"