Fail of deployment of a compute sets all hosts to error when using plugins

[Marcin Iwinski]

During deployment of 5 node cluster (3 controllers and 2 computes) with contrail plugin (https://github.com/stackforge/fuel-plugin-contrail) failure of one of the compute nodes switches all nodes into Error state forcing redeployment of entire environment instead of just failed node.

VERSION:
  feature_groups:
    - mirantis
    - experimental
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "521"
  build_id: "2015-06-08_06-13-27"
  nailgun_sha: "4340d55c19029394cd5610b0e0f56d6cb8cb661b"
  python-fuelclient_sha: "4fc55db0265bbf39c369df398b9dc7d6469ba13b"
  astute_sha: "7766818f079881e2dbeedb34e1f67e517ed7d479"
  fuel-library_sha: "f43c2ae1af3b493ee0e7810eab7bb7b50c986c7d"
  fuel-ostf_sha: "7c938648a246e0311d05e2372ff43ef1eb2e2761"
  fuelmain_sha: "bcc909ffc5dd5156ba54cae348b6a07c1b607b24"

Steps to reproduce:
1) install the plugin
2) create new environment
3) add 3 controllers and two compute nodes
4) configure interfaces for all hosts intentionally misconfiguring one of the compute nodes
5) deploy changes

Current behavior:
Although controllers got deployed successfully prior to deployment of misconfigured compute node, failure of this node is switching all nodes in the cluster (also those marked as "Ready") into error state forcing user to redeploy everything from scratch.

Expected behavior:
misconfigured compute node is switched into error state, controllers are still marked as "Ready" and don't require redeployment

[Oleksiy Molchanov (omolchanov)]

Contrail team, please take a look and pass it to python team if needed.

[Oleksiy Molchanov (omolchanov)]

Please attach diagnostic snapshot.

[Marcin Iwinski (iwi)]

Diagnostic snapshot contains a sensitive information, we can share it offline.
I don't believe this bug is caused by Contrail Plugin - we should test it without any plugin as well as with any other plugin different from contrail.

[Andrew Woodward (xarses)]

Any task that runs on a wild card of nodes in post_deployment cases this

[Ihor Kalnytskyi (ikalnytskyi)]

Nailgun's receiver is responsible for setting proper node/cluster statuses. Looks like we should dive in into this code..

P.S: See deployment_resp in receiver.py