Fuel 7.0 GA fails to deploy a multi-node (3) deployment with KVM and Neutron w/ tunneling segmentation. I have repeated this in several environments (VMs) and have yet another one this time with some nice stable physical hardware so it's time to report it. I have found that this issue can be avoided if one deploys a single controller node first, then in subsequent deployments adds a controller until you have the 3 that you want, but that is not the way it is supposed to to work.
I guess I'll start with the error message in the UI:
Error
Deployment has failed. Method granular_deploy. Deployment failed on nodes 5.
Inspect Astute logs for the details
The astute log is attached to this bug report, but it always comes down to a trace similar to below (I say similar, because sometimes it's a different service's puppet deployments, keystone, network controller, glance etc.. but it's always the similar) which shows a timeout of the puppet deployment. I have spent hours trying to figure out why the deployment has failed and the closest I can come is that haproxy is having some issues and preventing Keystone from responding to requests for endpoint information, but maybe I don't have that correct either.
2015-11-13T01:22:23 debug: [485] Node 5 has failed to deploy. There is no more retries for puppet run.
2015-11-13T01:22:23 debug: [485] {"nodes"=>[{"status"=>"error", "error_type"=>"deploy", "uid"=>"5", "role"=>"primary-controller"}]}
2015-11-13T01:22:23 err: [485] Task '{"priority"=>5700, "type"=>"puppet", "uids"=>["5"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/openstack-network/openstack-network-controller.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 5
2015-11-13T01:22:23 err: [485] No more tasks will be executed on the node 5
2015-11-13T01:22:23 debug: [485] Data received by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"5", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-controller", "task"=>{"priority"=>5700, "type"=>"puppet", "uids"=>["5"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/openstack-network/openstack-network-controller.pp", "timeout"=>3600, "cwd"=>"/"}}}]}
2015-11-13T01:22:23 debug: [485] Data send by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"5", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-controller", "task"=>{"priority"=>5700, "type"=>"puppet", "uids"=>["5"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/openstack-network/openstack-network-controller.pp", "timeout"=>3600, "cwd"=>"/"}}}]}
2015-11-13T01:22:23 info: [485] f7a6b2ae-5fca-4916-a6a4-a04271dd885e: Finished deployment of nodes => roles: {"5"=>"primary-controller"}
2015-11-13T01:22:23 err: [485] Error running RPC method granular_deploy: Deployment failed on nodes 5, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:68:in `block (3 levels) in deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:51:in `each_slice'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:51:in `block (2 levels) in deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:49:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:49:in `block in deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:45:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:45:in `deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/orchestrator.rb:179:in `deploy_cluster'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/orchestrator.rb:45:in `granular_deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/dispatcher.rb:91:in `granular_deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:146:in `dispatch_message'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:107:in `block in dispatch'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:64:in `call'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:56:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:56:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:105:in `each_with_index'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:105:in `dispatch'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:89:in `block in perform_main_job'"]
2015-11-13T01:22:23 info: [485] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"f7a6b2ae-5fca-4916-a6a4-a04271dd885e", "nodes"=>[{"uid"=>"5", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-controller", "task"=>{"priority"=>5700, "type"=>"puppet", "uids"=>["5"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/openstack-network/openstack-network-controller.pp", "timeout"=>3600, "cwd"=>"/"}}}]}}
2015-11-13T01:22:23 info: [485] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"f7a6b2ae-5fca-4916-a6a4-a04271dd885e", "status"=>"error", "error"=>"Method granular_deploy. Deployment failed on nodes 5.\nInspect Astute logs for the details"}}
Ed,
Please provide a fuel snapshot. Also, you may want to look in the puppet.log for that node for additional information as to why it failed.