Steps to reproduce:
1. Create cluster
2. Add 3 nodes with controller role
3. Add 2 node with compute role
4. Setup bonding for all interfaces except admin/pxe
5. Run network verification
6. Deploy the cluster
Expected result:
Deploy passed.
Actual result:
Deploy failed.
At the Astute log:
2015-05-27T04:29:21 err: [653] Error running RPC method granular_deploy: Deployment failed on nodes 3, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:68:in `block (3 levels) in deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:51:in `each_slice'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:51:in `block (2 levels) in deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:49:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:49:in `block in deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:45:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:45:in `deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:171:in `deploy_cluster'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:56:in `granular_deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:111:in `granular_deploy'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:142:in `dispatch_message'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:103:in `block in dispatch'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `call'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `block in each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `each_with_index'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `dispatch'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]
At the puppet log from node-3:
Wed May 27 04:19:26 +0000 2015 Puppet (err): Could not run: execution expired
/usr/lib/ruby/1.8/timeout.rb:64:in `open'
/usr/lib/ruby/1.8/net/http.rb:560:in `connect'
/usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
/usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
/usr/lib/ruby/1.8/net/http.rb:560:in `connect'
/usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
/usr/lib/ruby/1.8/net/http.rb:542:in `start'
/usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
/usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
/usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
/usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
/usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
/usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
/usr/lib/ruby/1.8/open-uri.rb:518:in `open'
/usr/lib/ruby/1.8/open-uri.rb:30:in `open'
/etc/puppet/modules/haproxy/lib/puppet/provider/haproxy_backend_status/haproxy.rb:135:in `get_csv_url'
/etc/puppet/modules/haproxy/lib/puppet/provider/haproxy_backend_status/haproxy.rb:20:in `csv'
/etc/puppet/modules/haproxy/lib/puppet/provider/haproxy_backend_status/haproxy.rb:14:in `csv'
/etc/puppet/modules/haproxy/lib/puppet/provider/haproxy_backend_status/haproxy.rb:12:in `times'
/etc/puppet/modules/haproxy/lib/puppet/provider/haproxy_backend_status/haproxy.rb:12:in `csv'
/etc/puppet/modules/haproxy/lib/puppet/provider/haproxy_backend_status/haproxy.rb:41:in `stats'
/etc/puppet/modules/haproxy/lib/puppet/provider/haproxy_backend_status/haproxy.rb:101:in `exists?'
/etc/puppet/modules/haproxy/lib/puppet/provider/haproxy_backend_status/haproxy.rb:71:in `ensure'
/usr/lib/ruby/site_ruby/1.8/puppet/property.rb:471:in `send'
/usr/lib/ruby/site_ruby/1.8/puppet/property.rb:471:in `retrieve'
/usr/lib/ruby/site_ruby/1.8/puppet/type.rb:1035:in `retrieve'
/usr/lib/ruby/site_ruby/1.8/puppet/type.rb:1063:in `retrieve_resource'
/usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:223:in `from_resource'
/usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:17:in `evaluate'
/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:174:in `apply'
/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:187:in `eval_resource'
/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `call'
/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
/usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
/usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:108:in `evaluate'
/usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:164:in `apply'
/usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:108:in `as_logging_destination'
/usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:163:in `apply'
/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:125:in `apply_catalog'
/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
/usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:124:in `apply_catalog'
/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:192:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:268:in `apply_catalog'
/usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:218:in `main'
/usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:146:in `run_command'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:470:in `plugin_hook'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:478:in `exit_on_fail'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'
/usr/bin/puppet:4
Fuel used:
feature_groups:
- mirantis
production: "docker"
release: "6.1"
openstack_version: "2014.2.2-6.1"
api: "1.0"
build_number: "469"
build_id: "2015-05-26_16-19-56"
nailgun_sha: "f737675091bd1903aace0e36812e855ce47dfec7"
python-fuelclient_sha: "e19f1b65792f84c4a18b5a9473f85ef3ba172fce"
astute_sha: "0bd72c72369e743376864e8e8dabfe873d40450a"
fuel-library_sha: "5cde6f16688486a37f3b280501e98a70883e0d06"
fuel-ostf_sha: "87819878bc0ca572900e1f6933d9b99e666d6f62"
fuelmain_sha: "13b3e9cf074ba1cf1ae06509c55fbab613c73f4e"
I'm not sure exactly what's going on here. Only node-3 can't connect to the haproxy stats port. During the time node-3 fails to connect, node-2(the other non-primary controller) is able to connect to that port with no issue.
Node-3: backend_ status[ mysql]( provider= haproxy) (debug): Get CSV from url 'http:// 10.109. 11.34:10000/ ;csv'
Wed May 27 04:16:40 +0000 2015 Haproxy_
Wed May 27 04:16:58 +0000 2015 Puppet (err): Could not run: execution expired
Node-2: backend_ status[ mysql]( provider= haproxy) (debug): Get CSV from url 'http:// 10.109. 11.34:10000/ ;csv' backend_ status[ mysql]( provider= haproxy) (debug): Return: up
Wed May 27 04:17:03 +0000 2015 Haproxy_
Wed May 27 04:17:03 +0000 2015 Haproxy_
There doesn't seem to be a problem with the management network on node-3 as it's able to join the corosync and galera clusters.