Deployment failed on node-6:
2015-08-31T15:40:38 err: [585] Task '{"priority"=>1100, "type"=>"puppet", "uids"=>["6"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/pupp et/modules/osnailyfacter/modular/cluster/cluster.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 6 2015-08-31T15:40:38 info: [585] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"242311c4-06df-41cd-803c-3baf8f56e73d", "nodes"=>[{"uid"=>"6", "st atus"=>"error", "error_type"=>"deploy", "role"=>"primary-controller", "task"=>{"priority"=>1100, "type"=>"puppet", "uids"=>["6"], "parameters"=>{"puppet_modules"=>"/etc/puppe t/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/cluster/cluster.pp", "timeout"=>3600, "cwd"=>"/"}}}]}} 2015-08-31T15:40:38 info: [585] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"242311c4-06df-41cd-803c-3baf8f56e73d", "status"=>"error", "error"
Here are errors from puppet logs:
node-6 2015-08-31T14:50:31.437391 notice: (Scope(Class[main])) MODULAR: netconfig.pp node-6 2015-08-31T14:50:33.002479 notice: Compiled catalog for node-6.domain.tld in environment production in 1.59 seconds node-6 2015-08-31T14:50:36.420798 err: Could not apply complete catalog: Found 1 dependency cycle: node-6 2015-08-31T14:50:36.420798 err: (L23_stored_config[br-fw-admin] => L2_bridge[br-fw-admin] => L2_port[eth2] => L23network::L2::Port[eth2] => L23network::L2::Port[eth2.1 41] => L23_stored_config[eth2.141] => L23network::L2::Port[eth2.141] => L23network::L2::Bridge[br-fw-admin] => L23_stored_config[br-fw-admin]) node-6 2015-08-31T14:50:36.420798 err: Try the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz node-6 2015-08-31T14:50:36.420798 err: /usr/lib/ruby/vendor_ruby/puppet/graph/simple_graph.rb:242:in `report_cycles_in_graph' node-6 2015-08-31T14:50:36.420798 err: /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:106:in `traverse' node-6 2015-08-31T14:50:36.420798 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate' node-6 2015-08-31T14:50:36.420798 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `block in apply' node-6 2015-08-31T14:50:36.420798 err: /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination' node-6 2015-08-31T14:50:36.420798 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination' node-6 2015-08-31T14:50:36.420948 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `block in apply_catalog' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block (2 levels) in run' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block in run' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run' node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute' node-6 2015-08-31T14:50:36.421951 err: /usr/bin/puppet:4:in `<main>' ... node-6 2015-08-31T15:01:49.322761 err: (/Stage[main]/Main/Pcmk_nodes[pacemaker]) Could not evaluate: Execution timeout after 600 seconds! ... node-6 2015-08-31T15:05:49.364171 err: (/Stage[main]/Openstack::Corosync/Cs_property[no-quorum-policy]) Could not evaluate: execution expired
According to `ip a` command output (see attachment) on node-6 network wasn't configured properly by netconfig.pp task, so pacemaker/corosync deployment failed.
Deployment failed on node-6:
2015-08-31T15:40:38 err: [585] Task '{"priority"=>1100, "type"=>"puppet", "uids"=>["6"], "parameters" =>{"puppet_ modules" =>"/etc/ puppet/ modules" , "puppet_ manifest" =>"/etc/ pupp osnailyfacter/ modular/ cluster/ cluster. pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 6 =>"deploy_ resp", "args"= >{"task_ uuid"=> "242311c4- 06df-41cd- 803c-3baf8f56e7 3d", "nodes" =>[{"uid" =>"6", "st type"=> "deploy" , "role"= >"primary- controller" , "task"= >{"priority" =>1100, "type"=>"puppet", "uids"=>["6"], "parameters" =>{"puppet_ modules" =>"/etc/ puppe manifest" =>"/etc/ puppet/ modules/ osnailyfacter/ modular/ cluster/ cluster. pp", "timeout"=>3600, "cwd"=>"/"}}}]}} =>"deploy_ resp", "args"= >{"task_ uuid"=> "242311c4- 06df-41cd- 803c-3baf8f56e7 3d", "status"=>"error", "error"
et/modules/
2015-08-31T15:40:38 info: [585] Casting message to Nailgun: {"method"
atus"=>"error", "error_
t/modules", "puppet_
2015-08-31T15:40:38 info: [585] Casting message to Nailgun: {"method"
Here are errors from puppet logs:
node-6 2015-08- 31T14:50: 31.437391 notice: (Scope( Class[main] )) MODULAR: netconfig.pp 31T14:50: 33.002479 notice: Compiled catalog for node-6.domain.tld in environment production in 1.59 seconds 31T14:50: 36.420798 err: Could not apply complete catalog: Found 1 dependency cycle: 31T14:50: 36.420798 err: (L23_stored_ config[ br-fw-admin] => L2_bridge[ br-fw-admin] => L2_port[eth2] => L23network: :L2::Port[ eth2] => L23network: :L2::Port[ eth2.1 config[ eth2.141] => L23network: :L2::Port[ eth2.141] => L23network: :L2::Bridge[ br-fw-admin] => L23_stored_ config[ br-fw-admin] ) 31T14:50: 36.420798 err: Try the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz 31T14:50: 36.420798 err: /usr/lib/ ruby/vendor_ ruby/puppet/ graph/simple_ graph.rb: 242:in `report_ cycles_ in_graph' 31T14:50: 36.420798 err: /usr/lib/ ruby/vendor_ ruby/puppet/ graph/relations hip_graph. rb:106: in `traverse' 31T14:50: 36.420798 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction. rb:108: in `evaluate' 31T14:50: 36.420798 err: /usr/lib/ ruby/vendor_ ruby/puppet/ resource/ catalog. rb:164: in `block in apply' 31T14:50: 36.420798 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util/log. rb:149: in `with_destination' 31T14:50: 36.420798 err: /usr/lib/ ruby/vendor_ ruby/puppet/ transaction/ report. rb:108: in `as_logging_ destination' 31T14:50: 36.420948 err: /usr/lib/ ruby/vendor_ ruby/puppet/ resource/ catalog. rb:163: in `apply' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ configurer. rb:125: in `block in apply_catalog' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util.rb: 161:in `block in benchmark' 31T14:50: 36.421951 err: /usr/lib/ ruby/1. 9.1/benchmark. rb:295: in `realtime' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util.rb: 160:in `benchmark' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ configurer. rb:124: in `apply_catalog' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ configurer. rb:192: in `run' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application/ apply.rb: 268:in `apply_catalog' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application/ apply.rb: 218:in `main' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application/ apply.rb: 146:in `run_command' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application. rb:364: in `block (2 levels) in run' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application. rb:470: in `plugin_hook' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application. rb:364: in `block in run' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util.rb: 478:in `exit_on_fail' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ application. rb:364: in `run' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util/command_ line.rb: 137:in `run' 31T14:50: 36.421951 err: /usr/lib/ ruby/vendor_ ruby/puppet/ util/command_ line.rb: 91:in `execute' 31T14:50: 36.421951 err: /usr/bin/ puppet: 4:in `<main>' 31T15:01: 49.322761 err: (/Stage[ main]/Main/ Pcmk_nodes[ pacemaker] ) Could not evaluate: Execution timeout after 600 seconds! 31T15:05: 49.364171 err: (/Stage[ main]/Openstack ::Corosync/ Cs_property[ no-quorum- policy] ) Could not evaluate: execution expired
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
41] => L23_stored_
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
node-6 2015-08-
...
node-6 2015-08-
...
node-6 2015-08-
According to `ip a` command output (see attachment) on node-6 network wasn't configured properly by netconfig.pp task, so pacemaker/corosync deployment failed.