Deployment fails with fuel_agent KeyError: 'mco_identity'

[Alex Schultz]

Attempted to deploy 1 controler, 1 compute. Deployment fails with the following error:

Error
Provision has failed. Too many nodes failed to provision

astute.log reports
2015-12-18 17:19:40 ERROR [863] Provision of some nodes failed. Failed nodes: ["2", "1"]
2015-12-18 17:19:40 INFO [863] Nodes failed to reboot: ["2", "1"]
2015-12-18 17:19:36 ERROR [863] 901f05f9-3677-4f34-b6a9-07131246abe9: Provision command returned non zero exit code on node: 1
2015-12-18 17:19:36 ERROR [863] 901f05f9-3677-4f34-b6a9-07131246abe9: Provision command returned non zero exit code on node: 2

fuel-agent logs on the node has the following stack trace:
2015-12-18 17:19:36 INFO fuel_agent.cmd.agent
2015-12-18 17:19:36 INFO fuel_agent.cmd.agent KeyError: 'mco_identity'
2015-12-18 17:19:36 INFO fuel_agent.cmd.agent identity=data['ks_meta']['mco_identity']
2015-12-18 17:19:36 INFO fuel_agent.cmd.agent File "/usr/lib/python2.7/dist-packages/fuel_agent/drivers/nailgun.py", line 529, in parse_configdrive_scheme
2015-12-18 17:19:36 INFO fuel_agent.cmd.agent self._configdrive_scheme = self.parse_configdrive_scheme()
2015-12-18 17:19:36 INFO fuel_agent.cmd.agent File "/usr/lib/python2.7/dist-packages/fuel_agent/drivers/nailgun.py", line 95, in __init__
2015-12-18 17:19:36 INFO fuel_agent.cmd.agent self.driver = utils.get_driver(CONF.data_driver)(data)
2015-12-18 17:19:36 INFO fuel_agent.cmd.agent File "/usr/lib/python2.7/dist-packages/fuel_agent/manager.py", line 167, in __init__
2015-12-18 17:19:36 INFO fuel_agent.cmd.agent mgr = manager.Manager(data)
2015-12-18 17:19:36 INFO fuel_agent.cmd.agent File "/usr/lib/python2.7/dist-packages/fuel_agent/cmd/agent.py", line 98, in main
2015-12-18 17:19:36 INFO fuel_agent.cmd.agent Traceback (most recent call last):
2015-12-18 17:19:36 INFO fuel_agent.cmd.agent [-] 'mco_identity'
2015-12-18 17:19:36 INFO fuel_agent.drivers.nailgun [-] Adding mcollective parameters

Expected results:
Deployment successful

Actual results:
Deployment fails as part of provisioning.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "313"
  build_id: "313"
  fuel-nailgun_sha: "405404d28fdd09a3fb6d803ab154aee591aedb3a"
  python-fuelclient_sha: "6810231ec6d333fc10f7020f9bdf5122e05e998a"
  fuel-agent_sha: "d56cf9f79f340e2268692e1e24ab3302d2c6a47f"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "c56dfde2da034151a7e707b381c4cf9d213b4ba2"
  fuel-library_sha: "d883f349813d7b4637f9b79cbf5be567023cecc0"
  fuel-ostf_sha: "9910a4726cbd038c257582b429527e40c4c3cb20"
  fuel-mirror_sha: "e9b9e24ce26aa8f13582797f5cec3651866d28c3"
  fuelmenu_sha: "d7b36d3fa439166d0afec965eb7403c4df578806"
  shotgun_sha: "cacb93cbc28910ff0dc38f30a855efa9af50d8ce"
  network-checker_sha: "d443ef47abeda58d319bc8d33d5005dd09440a02"
  fuel-upgrade_sha: "718aa3d7021fee2970f0fa6791cf5188578cc516"
  fuelmain_sha: "024d63b6ae368ba7bf38a1e5620114cc213e205b"

[Alexander Gordeev (a-gordeev)]

Marking as Invalid.

I'm pretty sure that it was just intermittent failure on our product CI and ISO with packages became out of sync. I don't know what stands behind that failure.

Let me prove that.

On friday, we merged 2 patches which should fix https://bugs.launchpad.net/fuel/+bug/1518306

First one to fuel-web, which adds 'mco_identity' to node attibutes. https://review.openstack.org/#/c/257332/ had been merged at 16:41
The last one to fuel-agent, which adds the rest of 'mco_identity' handling. https://review.openstack.org/#/c/257340/ had been mered at 16:46

Sequence of merges was totally proper. Moreover, these patches were tested against custom_8.0 iso on BVT and gave *green* results. So, the code is absolutely operable and doesn't contain visual flaws.

Then, CI magic happened. First build after that, which in turn was #313, passed BVT, but did include those changes.
Then, after a couple of broken builds, #318 passed BVT too. Commited changes were in place according to the logs from diagnostic snapshot. Every newer build (#318 and the above) showed the same *green* BVT results.

So, i think it's just how our CI works. Feel free to poke build-team/ci-team in order to figure out what was wrong with CI.