Saraha deployment fails: Error running RPC method granular_deploy

[Vladimir]

Sahara deployment fails during environment deploy for bug https://bugs.launchpad.net/fuel/+bug/1487397
Build #239
Steps to reproduce:
1. Create new environment
2. Choose Neutron, VLAN
3. Choose Ceph for images
4. Choose Sahara
5. Choose Ceilometer
6. Add 1 controller+ceph
7. Add 1 compute+ceph
8. Add 1 cinder+ceph
9. Add 2 mongo
10. Change disk configuration for both Mongo nodes. Change 'MongoDB' volume for vdc
11. Deploy the environment

Expected result:environment deployed.
Actual result: deployment fails with the following error:

ERROR [740] Error running RPC method granular_deploy: Deployment failed on nodes 1, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:68:in `block (3 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:51:in `each_slice'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:51:in `block (2 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:49:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:49:in `block in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:45:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:45:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/orchestrator.rb:179:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/orchestrator.rb:45:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:92:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:189:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:146:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:123:in `block in perform_main_job'"]
ERROR [740] No more tasks will be executed on the node 1
ERROR [740] Task '{"priority"=>7000, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/sahara/sahara.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 1

Environment snapshot could be found in attachment.

Gdrive link:
https://drive.google.com/a/mirantis.com/file/d/0BzGc8pMVuherYXBIc3BWeEl2clk/view?usp=sharing

[magicboiz (magicboiz)]

Hi

I'm facing the same problem with this config: MOS61 (fully updated) + QEMU + GRE + CONTRAIL PLUGIN.

2015-12-02 17:05:44 ERR
[475] Error running RPC method granular_deploy: Deployment failed on nodes 53, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:68:in `block (3 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:51:in `each_slice'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:51:in `block (2 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:49:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:49:in `block in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:45:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:45:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:179:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:56:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:111:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]
2015-12-02 17:05:44 ERR
[475] No more tasks will be executed on the node 53
2015-12-02 17:05:44 ERR
[475] Task '{"priority"=>3500, "type"=>"puppet", "uids"=>["53"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/sahara/sahara.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 53

[magicboiz (magicboiz)]

checking the puppet.log in the failed node, it looks like there is a problem creating templates:

2015-12-02 21:37:21 +0000 /Stage[main]/Sahara::Templates::Create_templates/Sahara_node_group_template[cdh-5-manager] (info): Starting to evaluate the resource
2015-12-02 21:37:21 +0000 Sahara_node_group_template[cdh-5-manager](provider=ruby) (debug): Call: exists?
2015-12-02 21:37:21 +0000 Sahara_node_group_template[cdh-5-manager](provider=ruby) (debug): Call: network_connection
2015-12-02 21:37:21 +0000 Sahara_node_group_template[cdh-5-manager](provider=ruby) (debug): Call: compute_connection
2015-12-02 21:37:23 +0000 Sahara_node_group_template[cdh-5-manager](provider=ruby) (debug): Set flavor_id to: 4
2015-12-02 21:37:23 +0000 Sahara_node_group_template[cdh-5-manager](provider=ruby) (debug): Call: extract
2015-12-02 21:37:23 +0000 Sahara_node_group_template[cdh-5-manager](provider=ruby) (debug): Call: connection
2015-12-02 21:37:23 +0000 Sahara_node_group_template[cdh-5-manager](provider=ruby) (debug): Existing state: {:ensure=>:absent}
2015-12-02 21:37:23 +0000 Sahara_node_group_template[cdh-5-manager](provider=ruby) (debug): Result: false
2015-12-02 21:37:23 +0000 Sahara_node_group_template[cdh-5-manager](provider=ruby) (debug): Call: create
2015-12-02 21:37:23 +0000 /Stage[main]/Sahara::Templates::Create_templates/Sahara_node_group_template[cdh-5-manager]/ensure (notice): created
2015-12-02 21:37:23 +0000 Sahara_node_group_template[cdh-5-manager](provider=ruby) (debug): Call: flush
2015-12-02 21:37:23 +0000 /Stage[main]/Sahara::Templates::Create_templates/Sahara_node_group_template[cdh-5-manager] (err): Could not evaluate: The server returned status 400
/usr/lib/ruby/vendor_ruby/openstack/connection.rb:501:in `rescue in raise_exception'
/usr/lib/ruby/vendor_ruby/openstack/connection.rb:484:in `raise_exception'
/etc/puppet/modules/sahara/lib/puppet/provider/sahara_openstack.rb:116:in `create_node_group_template'
/etc/puppet/modules/sahara/lib/puppet/provider/sahara_node_group_template/ruby.rb:212:in `flush'
/usr/lib/ruby/vendor_ruby/puppet/type.rb:977:in `flush'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:24:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:174:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:187:in `eval_resource'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `call'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:327:in `block in thinmark'
/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:326:in `thinmark'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/con...

[Stanislaw Bogatkin (sbogatkin)]

Problem is that network scheme changed and we use net04 by default for sahara templates. But this network now named admin_internal_net by default, so templates cannot be created.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/253567

[OpenStack Infra (hudson-openstack)]

Change abandoned by Stanislaw Bogatkin (<email address hidden>) on branch: master
Review: https://review.openstack.org/253567
Reason: In favor of 252882

[magicboiz (magicboiz)]

Hi

can change #252882 (patch set 4) be applied to MOS6.1?

thx

[magicboiz (magicboiz)]

Hi Denis,

you can find an snapshot here: https://dl.dropboxusercontent.com/u/1425379/fuel-snapshot-2015-12-09_16-12-39.tar.xz