[fuel] Deploy failed on base-os node

[Alexander Kurenyshev]

Steps to reproduce:
1) Create new environment
2) Add nodes with 1 controller, 1 compute, 1 cinder, 1 base-os with Neutron Gre network.

Result:

post-deploy hook hosts failed on 'base os' nodes.

Puppet log error:

Mon Feb 16 12:22:58 +0000 2015 Puppet (debug): hiera(): Hiera YAML backend starting
Mon Feb 16 12:22:58 +0000 2015 Puppet (debug): hiera(): Looking up nodes in YAML backend
Mon Feb 16 12:22:58 +0000 2015 Puppet (debug): hiera(): Looking for data source common
Mon Feb 16 12:22:58 +0000 2015 Puppet (err): Could not find data item nodes in any Hiera data file and no default supplied at /etc/puppet/modules/osnailyfacter/modular/hosts.pp:4 on node node-4.test.domain.local

In the logs:
[400] Error running RPC method granular_deploy: Failed to deploy plugin , trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:50:in `block in process'",

2015-02-16 12:23:12 WARNING

[400] Puppet run failed. Check puppet logs for details

2015-02-16 12:23:12 INFO

[400] 38ab7beb-47fb-4bab-98c8-d9fc5bc2f504: Spent 17.888454722 seconds on puppet run for following nodes(uids): 1,2,3,4

2015-02-16 12:23:12 DEBUG

[400] {"nodes"=>[{"status"=>"error", "error_type"=>"deploy", "uid"=>"4", "role"=>"hook"}]}

2015-02-16 12:23:12 DEBUG

[400] Node 4 has failed to deploy. There is no more retries for puppet run.

2015-02-16 12:23:12 DEBUG

[400] Node 4(hook) status: error

2015-02-16 12:23:12 DEBUG

[400] 38ab7beb-47fb-4bab-98c8-d9fc5bc2f504: MC agent 'puppetd', method 'last_run_summary', results: {:sender=>"4", :statuscode=>0, :statusmsg=>"OK", :data=>{:output=>"Currently stopped; last completed run 1424089392 seconds ago", :enabled=>1, :status=>"stopped", :runtime=>1424089392, :idling=>0, :stopped=>1, :version=>nil, :resources=>{"changed"=>0, "out_of_sync"=>0, "restarted"=>0, "total"=>0, "failed"=>1}, :lastrun=>0, :time=>{"last_run"=>0}, :running=>0, :changes=>nil, :err_msg=>"Potencial error. Looks like expecting puppet and actual are different. Expecting pid(lockfile): 5679, actual(ps): ", :events=>nil}}

See all logs at snapshot attached.

Fuel used:
"build_id": "2015-02-15_22-54-44",
"ostf_sha": "f9c37d0876141e1550eb4e703a8e500cd463282f",
"build_number": "126",
"release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-02-15_22-54-44",
"ostf_sha": "f9c37d0876141e1550eb4e703a8e500cd463282f",
"build_number": "126",
"api": "1.0",
"nailgun_sha": "1e3a40dd8a17abe1d38f42da1e0dc1a6d4572666",
"production": "docker",
"python-fuelclient_sha": "61431ed16fc00039a269424bdbaa410277eff609",
"astute_sha": "1f87a9b9a47de7498b4061d15a8c7fb9435709d5",
"feature_groups": ["mirantis"], "release": "6.1",
"fuelmain_sha": "2054229e275d08898b5d079a6625ffcc79ae23b8",
"fuellib_sha": "7f8d4382abfcd4338964182ebfea1d539f963e66"}}}, "auth_required": true, "api": "1.0",
"nailgun_sha": "1e3a40dd8a17abe1d38f42da1e0dc1a6d4572666",
"production": "docker",
"python-fuelclient_sha": "61431ed16fc00039a269424bdbaa410277eff609",
"astute_sha": "1f87a9b9a47de7498b4061d15a8c7fb9435709d5",
"feature_groups": ["mirantis"], "release": "6.1",
"fuelmain_sha": "2054229e275d08898b5d079a6625ffcc79ae23b8",
"fuellib_sha": "7f8d4382abfcd4338964182ebfea1d539f963e66"

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/156555

[Vladimir Sharshov (vsharshov)]

Update bug information. The core of problem: missing hiera and global tasks for base-os system.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/156620

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/156555
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=e048bee5c4727c10ec4a0a7d3e2f2a9b66d036a8
Submitter: Jenkins
Branch: master

commit e048bee5c4727c10ec4a0a7d3e2f2a9b66d036a8
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Tue Feb 17 14:41:46 2015 +0300

    Remove misleading message about pids

    If puppet failed and pid did not remove,
    but actual process already finished
    we show message:

        "Looks like expecting puppet and actual are different"

    Now we will show it only if we have different between
    running actual and pid from file.

    Change-Id: I74b3d280329f98fd064a78a91024001a948ef6c1
    Related-Bug: #1422364

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/156620
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=652a99b5d3c96df57186e27f5d5acb8cd8dfcd43
Submitter: Jenkins
Branch: master

commit 652a99b5d3c96df57186e27f5d5acb8cd8dfcd43
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Tue Feb 17 16:36:52 2015 +0300

    Steps 'hiera' and 'globals' are requiring for all roles

    Without it base-os nodes could not run post-deployment hooks,
    for example: hosts.

    Co-Authored-By: Dmitry Ilyin <email address hidden>

    Change-Id: I6ed5023bb8dca64ecf456a081a57b7b8d283e330
    Closes-Bug: #1422364

[Anastasia Palkina (apalkina)]

Verified on ISO #143

"build_id": "2015-02-24_22-54-44", "ostf_sha": "1a0b2c6618fac098473c2ed5a9af11d3a886a3bb", "build_number": "143", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-02-24_22-54-44", "ostf_sha": "1a0b2c6618fac098473c2ed5a9af11d3a886a3bb", "build_number": "143", "api": "1.0", "nailgun_sha": "f4487769f5526bd50ecd42af296f5587b7fc231f", "production": "docker", "python-fuelclient_sha": "5657dbf06fddb74adb61e9668eb579a1c57d8af8", "astute_sha": "6d6ad68e0cde286d74ac7d52e21da4fc8dcbe9ab", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "b975019fabdb429c1869047df18dd792d2163ecc", "fuellib_sha": "8384b8ca4db84794fb21e287202f05e31f78841c"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "f4487769f5526bd50ecd42af296f5587b7fc231f", "production": "docker", "python-fuelclient_sha": "5657dbf06fddb74adb61e9668eb579a1c57d8af8", "astute_sha": "6d6ad68e0cde286d74ac7d52e21da4fc8dcbe9ab", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "b975019fabdb429c1869047df18dd792d2163ecc", "fuellib_sha": "8384b8ca4db84794fb21e287202f05e31f78841c"