Original problem on nodes: 4,5,6,:
ssh: connect to host node-1 port 22: No route to host
Puppet apply source trace:
2015-06-01T03:26:09.126457+00:00 notice: (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) ssh: connect to host node-1 port 22: No route to host 2015-06-01T03:26:09.126457+00:00 notice: (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) [ceph_deploy.config][ERROR ] Unable to pull /etc/ceph/ceph.conf from node-1 2015-06-01T03:26:09.126457+00:00 notice: (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) [ceph_deploy][ERROR ] GenericError: Failed to fetch config from 1 hosts 2015-06-01T03:26:09.126457+00:00 err: ceph-deploy --overwrite-conf config pull node-1 returned 1 instead of one of [0] 2015-06-01T03:26:09.126457+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:97:in `fail' 2015-06-01T03:26:09.126705+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:142:in `sync' 2015-06-01T03:26:09.126705+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:191:in `sync' 2015-06-01T03:26:09.126705+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed' 2015-06-01T03:26:09.128200+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `block in perform_changes' 2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `each' 2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `perform_changes' 2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate' 2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:174:in `apply' 2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:187:in `eval_resource' 2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `call' 2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block (2 levels) in evaluate' 2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:327:in `block in thinmark' 2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime' 2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:326:in `thinmark' 2015-06-01T03:26:09.128636+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block in evaluate' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `block in apply' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `block in apply_catalog' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block (2 levels) in run' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block in run' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run' 2015-06-01T03:26:09.129944+00:00 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute' 2015-06-01T03:26:09.129944+00:00 err: /usr/bin/puppet:4:in `<main>' 2015-06-01T03:26:09.129944+00:00 err: (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) change from notrun to 0 failed: ceph-deploy --overwrite-conf config pull node-1 returned 1 instead of one of [0]
Original problem on nodes: 4,5,6,:
ssh: connect to host node-1 port 22: No route to host
Puppet apply source trace:
2015-06- 01T03:26: 09.126457+ 00:00 notice: (/Stage[ main]/Ceph: :Conf/Exec[ ceph-deploy config pull]/returns) ssh: connect to host node-1 port 22: No route to host 01T03:26: 09.126457+ 00:00 notice: (/Stage[ main]/Ceph: :Conf/Exec[ ceph-deploy config pull]/returns) [ceph_deploy. config] [ERROR ] Unable to pull /etc/ceph/ceph.conf from node-1 01T03:26: 09.126457+ 00:00 notice: (/Stage[ main]/Ceph: :Conf/Exec[ ceph-deploy config pull]/returns) [ceph_deploy][ERROR ] GenericError: Failed to fetch config from 1 hosts 01T03:26: 09.126457+ 00:00 err: ceph-deploy --overwrite-conf config pull node-1 returned 1 instead of one of [0] 01T03:26: 09.126457+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util/errors. rb:97:in `fail' 01T03:26: 09.126705+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ type/exec. rb:142: in `sync' 01T03:26: 09.126705+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction/ resource_ harness. rb:191: in `sync' 01T03:26: 09.126705+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction/ resource_ harness. rb:128: in `sync_if_needed' 01T03:26: 09.128200+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction/ resource_ harness. rb:89:in `block in perform_changes' 01T03:26: 09.128636+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction/ resource_ harness. rb:88:in `each' 01T03:26: 09.128636+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction/ resource_ harness. rb:88:in `perform_changes' 01T03:26: 09.128636+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction/ resource_ harness. rb:20:in `evaluate' 01T03:26: 09.128636+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction. rb:174: in `apply' 01T03:26: 09.128636+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction. rb:187: in `eval_resource' 01T03:26: 09.128636+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction. rb:117: in `call' 01T03:26: 09.128636+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction. rb:117: in `block (2 levels) in evaluate' 01T03:26: 09.128636+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util.rb: 327:in `block in thinmark' 01T03:26: 09.128636+ 00:00 err: /usr/lib/ ruby/1. 9.1/benchmark. rb:295: in `realtime' 01T03:26: 09.128636+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util.rb: 326:in `thinmark' 01T03:26: 09.128636+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction. rb:117: in `block in evaluate' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ graph/relations hip_graph. rb:118: in `traverse' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction. rb:108: in `evaluate' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ resource/ catalog. rb:164: in `block in apply' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util/log. rb:149: in `with_destination' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction/ report. rb:108: in `as_logging_ destination' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ resource/ catalog. rb:163: in `apply' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ configurer. rb:125: in `block in apply_catalog' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util.rb: 161:in `block in benchmark' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/1. 9.1/benchmark. rb:295: in `realtime' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util.rb: 160:in `benchmark' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ configurer. rb:124: in `apply_catalog' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ configurer. rb:192: in `run' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application/ apply.rb: 268:in `apply_catalog' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application/ apply.rb: 218:in `main' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application/ apply.rb: 146:in `run_command' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application. rb:364: in `block (2 levels) in run' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application. rb:470: in `plugin_hook' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application. rb:364: in `block in run' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util.rb: 478:in `exit_on_fail' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application. rb:364: in `run' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util/command_ line.rb: 137:in `run' 01T03:26: 09.129944+ 00:00 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util/command_ line.rb: 91:in `execute' 01T03:26: 09.129944+ 00:00 err: /usr/bin/ puppet: 4:in `<main>' 01T03:26: 09.129944+ 00:00 err: (/Stage[ main]/Ceph: :Conf/Exec[ ceph-deploy config pull]/returns) change from notrun to 0 failed: ceph-deploy --overwrite-conf config pull node-1 returned 1 instead of one of [0]
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-
2015-06-