Deployment fails with "Error running RPC method granular_deploy: Failed to deploy plugin "

[Anastasia Kuznetsova]

{"build_id": "2015-02-16_22-54-44", "ostf_sha": "f9c37d0876141e1550eb4e703a8e500cd463282f", "build_number": "128", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-02-16_22-54-44", "ostf_sha": "f9c37d0876141e1550eb4e703a8e500cd463282f", "build_number": "128", "api": "1.0", "nailgun_sha": "3b5a6fdb7254d25a3b7000405f70e242bca200eb", "production": "docker", "python-fuelclient_sha": "33ce5fb5e483b91b1448ba57aa705df4d052f066", "astute_sha": "8af8e88c3cb17b66368e7a038f1899e5c7c13e98", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "2054229e275d08898b5d079a6625ffcc79ae23b8", "fuellib_sha": "c74044e8397d6b3019b4794e2c3bb30754b8bc71"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "3b5a6fdb7254d25a3b7000405f70e242bca200eb", "production": "docker", "python-fuelclient_sha": "33ce5fb5e483b91b1448ba57aa705df4d052f066", "astute_sha": "8af8e88c3cb17b66368e7a038f1899e5c7c13e98", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "2054229e275d08898b5d079a6625ffcc79ae23b8", "fuellib_sha": "c74044e8397d6b3019b4794e2c3bb30754b8bc71"}

Environments deployment in system tests fails with error. (6.1, ISO 128, CentOS, One Controller)

Error in astute logs:
[399] Error running RPC method granular_deploy: Failed to deploy plugin , trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:51:in `block in process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:26:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:26:in `process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine/granular_deployment.rb:219:in `post_deployment_actions'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine.rb:74:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:312:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:63:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/dispatcher.rb:99:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

[Ryan Moe (rmoe)]

Invalid for 6.0.1 as this is a bug with granular deployment.

[Kyrylo Romanenko (kromanenko)]

Also reproduced it on build
{"build_id": "2015-03-11_21-47-59", "ostf_sha": "ecb8e294b0acbdc5b0300d5e39028fb26ecc9088", "build_number": "187", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-11_21-47-59", "ostf_sha": "ecb8e294b0acbdc5b0300d5e39028fb26ecc9088", "build_number": "187", "api": "1.0", "nailgun_sha": "a720a2da99690eb2d2c19ddc5d739384312a8ac2", "production": "docker", "python-fuelclient_sha": "0f4ca9c2798da34797dd082130d22cac04c998a9", "astute_sha": "5cdd4ae4037aa29f4c876d441af15cad82f5a6cb", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0791400dd8224647ff9a5cb8051ce82b2c8863b1", "fuellib_sha": "cfdfcbdb0197f606b4c93e6dd4011525df9a3ff8"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "a720a2da99690eb2d2c19ddc5d739384312a8ac2", "production": "docker", "python-fuelclient_sha": "0f4ca9c2798da34797dd082130d22cac04c998a9", "astute_sha": "5cdd4ae4037aa29f4c876d441af15cad82f5a6cb", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0791400dd8224647ff9a5cb8051ce82b2c8863b1", "fuellib_sha": "cfdfcbdb0197f606b4c93e6dd4011525df9a3ff8"}

[Dmitry Ilyin (idv1985)]

Logs show that it's actually the controller task have failed

2015-02-17T07:01:39 err: [399] Task '{"priority"=>1100, "type"=>"puppet", "uids"=>["3"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/controller.pp", "timeout"=>3600, "cwd"=>"/"}}' on node 3 valid, but failed
2015-02-17T07:01:39 info: [399] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"964835d0-f828-4686-8ad2-22dc9c7d7350", "nodes"=>[{"uid"=>"3", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-controller", "task"=>{"priority"=>1100, "type"=>"puppet", "uids"=>["3"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/controller.pp", "

because:
Could not find resource 'Package[pacemaker]' for relationship on 'File[ceilometer-agent-central-ocf]' on node node-3

It should not happen or our current code base.

[Dmitry Ilyin (idv1985)]

hosts.pp hook is being run every deployment and we don't see it failing.
Unfortunately i don't have logs from kromanenko's deployment with fuel lib cfdfcbdb0197f606b4c93e6dd4011525df9a3ff8

Perhaps it's just Astute saying that plugin failed when it's actually a task failed for whatever reason.
I guess I can mark this bug invalid until I have a reproduction on the current code base.