Environment deployment failed with astute error

[Vadim Rovachev]

{"build_id": "2014-09-13_14-34-19", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "6", "auth_required": true, "api": "1.0", "nailgun_sha": "b8d8189cc37d6d1b26f4479be6be7313beefb1c8", "production": "docker", "fuelmain_sha": "d7ed7973034bde73d3f42c000984423b59b2312b", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["experimental"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-13_14-34-19", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "6", "api": "1.0", "nailgun_sha": "b8d8189cc37d6d1b26f4479be6be7313beefb1c8", "production": "docker", "fuelmain_sha": "d7ed7973034bde73d3f42c000984423b59b2312b", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["experimental"], "release": "5.1", "fuellib_sha": "395fd9d20a003603cc9ad26e16cb13c1c45e24e6"}}}, "fuellib_sha": "395fd9d20a003603cc9ad26e16cb13c1c45e24e6"}

Precondition steps.
Fuel master installed.

Steps to reproduce.
1. Create env with params:
   3 controller + monto nodes
   1 compute node
   KVM hypervizor
   neutron GRE network with VLAN tagging
2. Deploy env.
Expected result:
   Env has successfully installed
Actual result:
   Env Deployment has failed with message:
Deployment has failed. Error occurred while running method 'deploy'. Inspect Astute logs for the details

[Ihor Kalnytskyi (ikalnytskyi)]

Moving to Fuel-Library because something wrong with Astute calls.

Log Snippet: http://xsnippet.org/360177/raw/

[Bogdan Dobrelya (bogdando)]

Some more messages in astute logs: http://pastebin.com/ng7FdUpj
Looks like primary-controller role wasn't found in post deployment hook?

[Vladimir Sharshov (vsharshov)]

Bogdan is right, no node with primary-controller role found.

2014-09-16T13:10:17 debug: [424] Data received by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"1", "status"=>"ready", "role"=>"controller"}]}
2014-09-16T13:26:02 debug: [424] Data received by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"2", "status"=>"ready", "role"=>"controller"}]}
2014-09-16T13:46:57 debug: [424] Data received by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"3", "status"=>"ready", "role"=>"controller"}]}
2014-09-16T13:51:58 debug: [424] Data received by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"4", "status"=>"ready", "role"=>"compute"}]}

This happened, because cluster has not HA, but multinode mode. ("deployment_mode\": \"multinode\")

Solution: add deployment_mode for UpdateNoQuorumPolicy callback.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/121928

[Bogdan Dobrelya (bogdando)]

Please backport it to 5.1 as well

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/122103

[Bogdan Dobrelya (bogdando)]

According to logs, the rc of failed deployment was not the astute post depoy hook issue reported:

2014-09-16T13:51:59.425958 node-1 ./10.20.1.2/var/log/docker-logs/remote/node-1.domain.tld/nova-conductor.log:2014-09-16T13:51:59.425958+01:00 err: 2014-09-16 12:51:59.412 23743 ERROR oslo.messaging.rpc.dispatcher [req-be21f940-2f42-49ce-a3a4-56001616a132 ] Exception during message handling: (IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova`.`compute_nodes`,

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/121928
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=913b58d2b8726f9bf4ab2897f0d964cd69c71093
Submitter: Jenkins
Branch: master

commit 913b58d2b8726f9bf4ab2897f0d964cd69c71093
Author: Vladimir Sharshov <email address hidden>
Date: Tue Sep 16 21:09:23 2014 +0400

    Run update no quorum policy only for HA

    Change-Id: Ie5655a0782403d27056189bd5cdac7ba8c0f9dbb
    Closes-Bug: #1370056

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/122103
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=702af3db6f5bca92525bc8322d7d5d7675ec857e
Submitter: Jenkins
Branch: stable/5.1

commit 702af3db6f5bca92525bc8322d7d5d7675ec857e
Author: Vladimir Sharshov <email address hidden>
Date: Tue Sep 16 21:09:23 2014 +0400

    Run update no quorum policy only for HA

    Change-Id: Ie5655a0782403d27056189bd5cdac7ba8c0f9dbb
    Closes-Bug: #1370056

[Kirill Omelchenko (komelchenko)]

Verified.
{
   "build_id":"2014-11-17_21-00-23",
   "ostf_sha":"64cb59c681658a7a55cc2c09d079072a41beb346",
   "build_number":"19",
   "auth_required":true,
   "api":"1.0",
   "nailgun_sha":"2fcab95dc43a248ba867065e96ab764ee73882d1",
   "production":"docker",
   "fuelmain_sha":"ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9",
   "astute_sha":"fce051a6d013b1c30aa07320d225f9af734545de",
   "feature_groups":[
      "mirantis"
   ],
   "release":"5.1.1",
   "release_versions":{
      "2014.1.3-5.1.1":{
         "VERSION":{
            "build_id":"2014-11-17_21-00-23",
            "ostf_sha":"64cb59c681658a7a55cc2c09d079072a41beb346",
            "build_number":"19",
            "api":"1.0",
            "nailgun_sha":"2fcab95dc43a248ba867065e96ab764ee73882d1",
            "production":"docker",
            "fuelmain_sha":"ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9",
            "astute_sha":"fce051a6d013b1c30aa07320d225f9af734545de",
            "feature_groups":[
               "mirantis"
            ],
            "release":"5.1.1",
            "fuellib_sha":"add3fdd3e2af57b20dbb73a6bc53a9ccc4701c9a"
         }
      }
   },
   "fuellib_sha":"add3fdd3e2af57b20dbb73a6bc53a9ccc4701c9a"
}

[Andrey Sledzinskiy (asledzinskiy)]

verified on {

    "build_id": "2014-12-26_14-25-46",
    "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4",
    "build_number": "58",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90",
    "production": "docker",
    "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8",
    "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91",
    "feature_groups": [
        "mirantis"
    ],
    "release": "6.0",
    "release_versions": {
        "2014.2-6.0": {
            "VERSION": {
                "build_id": "2014-12-26_14-25-46",
                "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4",
                "build_number": "58",
                "api": "1.0",
                "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90",
                "production": "docker",
                "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8",
                "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "6.0",
                "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"
            }
        }
    },
    "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"

}