[vbox] - MOS Build 474 - Fails to install Ubuntu 14.0.4 with Ceph OSD with Astute Error

[Bruce Basil Mathews]

Install Fuel 474 ISO
run launch.sh
Access Fuel Web UI
Create cloud with Ubuntu 14.0.4, 3 Controllers, 3 Compute - Storage nodes with Ceph OSD Storage on each.
Deploy fails for Compute-Storage nodes

Fuel Snapshot: https://drive.google.com/a/mirantis.com/file/d/0B9pI3TCDy1b8b3l3OHJsT3BNYlk/view?usp=sharing

Astute Error Log excerpts:

[569] Error running RPC method granular_deploy: Deployment failed on nodes 5, 6, 4, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:68:in `block (3 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:51:in `each_slice'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:51:in `block (2 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:49:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:49:in `block in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:45:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:45:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:171:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:56:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:111:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

2015-06-01 03:26:09 ERR

[569] No more tasks will be executed on the node 6

2015-06-01 03:26:09 ERR

[569] Task '{"priority"=>1100, "type"=>"puppet", "uids"=>["6"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/roles/ceph-osd.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 6

2015-06-01 03:26:09 ERR

[569] No more tasks will be executed on the node 5

2015-06-01 03:26:09 ERR

[569] Task '{"priority"=>1100, "type"=>"puppet", "uids"=>["5"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/roles/ceph-osd.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 5

2015-06-01 03:25:56 ERR

[569] No more tasks will be executed on the node 4

2015-06-01 03:25:56 ERR

[569] Task '{"priority"=>1100, "type"=>"puppet", "uids"=>["4"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/roles/ceph-osd.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 4

2015-06-01 03:21:45 ERR

[569] No more tasks will be executed on the node 4

2015-06-01 03:21:45 ERR

[569] Task '{"priority"=>1200, "type"=>"puppet", "uids"=>["4"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/ceph/compute.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 4

2015-06-01 03:21:17 ERR

[569] No more tasks will be executed on the node 6

2015-06-01 03:21:17 ERR

[569] Task '{"priority"=>1200, "type"=>"puppet", "uids"=>["6"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/ceph/compute.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 6

2015-06-01 03:21:14 ERR

[569] No more tasks will be executed on the node 5

2015-06-01 03:21:14 ERR

[569] Task '{"priority"=>1200, "type"=>"puppet", "uids"=>["5"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/ceph/compute.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 5

[Vladimir Sharshov (vsharshov)]

Original problem on nodes: 4,5,6,:

    ssh: connect to host node-1 port 22: No route to host

Puppet apply source trace:

2015-06-01T03:26:09.126457+00:00 notice: (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) ssh: connect to host node-1 port 22: No route to host
2015-06-01T03:26:09.126457+00:00 notice: (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) [ceph_deploy.config][ERROR ] Unable to pull /etc/ceph/ceph.conf from node-1
2015-06-01T03:26:09.126457+00:00 notice: (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) [ceph_deploy][ERROR ] GenericError: Failed to fetch config from 1 hosts
2015-06-01T03:26:09.126457+00:00 err: ceph-deploy --overwrite-conf config pull node-1 returned 1 instead of one of [0]
2015-06-01T03:26:09.126457+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:97:in `fail'
2015-06-01T03:26:09.126705+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:142:in `sync'
2015-06-01T03:26:09.126705+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:191:in `sync'
2015-06-01T03:26:09.126705+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
2015-06-01T03:26:09.128200+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `block in perform_changes'
2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `each'
2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `perform_changes'
2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:174:in `apply'
2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:187:in `eval_resource'
2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `call'
2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block (2 levels) in evaluate'
2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:327:in `block in thinmark'
2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:326:in `thinmark'
2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block in evaluate'
2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `block in apply'
2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/p...

[Bogdan Dobrelya (bogdando)]

There were a lots of granular related and tasks based changes for the Ceph deployment in the 6.1, though this bug is likely invalid for other milestones

[Stanislaw Bogatkin (sbogatkin)]

I cannot reproduce this on last ISO. Bruce, can you tell, how you create slave nodes for this deployment?

[Bruce Basil Mathews (bmathews-l)]

I selected the Compute and Storage roles (Ceph-OSD) for the final three nodes after having selected the first three nodes as Controller and Telemetry nodes of the cluster using the Fuel GUI. I did not change any other defaults on any of the additional tabs and I hit the Deploy Changes button. That was pretty much what was attempted with the result described above.

[Stanislaw Bogatkin (sbogatkin)]

Bruce, could you please clarify next moments:
1. How much total RAM did you have for this deployment? Ceilometer needs much memory.
2. How did you created final three nodes? Which network topology they had?

[Alexander Kurenyshev (akurenyshev)]

The mongoDB node should have at least 5 GB of the memory. Here we can see at the screenshot, nodes with roles controller+mongo have only 2 GB of the memory. Because of that deploy failed.