SWARM gates with system tests for Ceilo and Heat (it install Ceilometer too) have failed
http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.services_ha.ceilometer/50/
http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.services_ha/51/
ENV: ISO 139, HA (3 controllers), Mongo on all controllers
Following error in astute.log:
2015-08-06T05:57:30 debug: [669] Node 1(primary-mongo) status: error
2015-08-06T05:57:30 debug: [669] Node 1 has failed to deploy. There is no more retries for puppet run.
2015-08-06T05:57:30 debug: [669] {"nodes"=>[{"status"=>"error", "error_type"=>"deploy", "uid"=>"1", "role"=>"primary-mongo"}]}
2015-08-06T05:57:30 err: [669] Task '{"priority"=>1000, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/roles/mongo.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 1
2015-08-06T05:57:30 err: [669] No more tasks will be executed on the node 1
2015-08-06T05:57:30 debug: [669] Data received by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"1", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-mongo", "task"=>{"priority"=>1000, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/roles/mongo.pp", "timeout"=>3600, "cwd"=>"/"}}}]}
2015-08-06T05:57:30 debug: [669] Data send by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"1", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-mongo", "task"=>{"priority"=>1000, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/roles/mongo.pp", "timeout"=>3600, "cwd"=>"/"}}, "progress"=>100}]}
2015-08-06T05:57:30 info: [669] 0bd8ab3a-a97b-4fd1-8583-d42f0fb0f694: Finished deployment of nodes => roles: {"1"=>"primary-mongo"}
2015-08-06T05:57:30 info: [669] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"0bd8ab3a-a97b-4fd1-8583-d42f0fb0f694", "nodes"=>[{"uid"=>"1", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-mongo", "task"=>{"priority"=>1000, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/roles/mongo.pp", "timeout"=>3600, "cwd"=>"/"}}, "progress"=>100}]}}
2015-08-06T05:57:30 err: [669] Error running RPC method granular_deploy: Deployment failed on nodes 1, trace:
2015-08-06T05:57:30 info: [669] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"0bd8ab3a-a97b-4fd1-8583-d42f0fb0f694", "status"=>"error", "error"=>"Method granular_deploy. Deployment failed on nodes 1.\nInspect Astute logs for the details"}}
Error in node-1 puppet logs:
2015-08-06 05:57:26 +0000 Puppet (debug): Executing '/usr/bin/mongo admin --quiet --port 27017 --eval load('/root/.mongorc.js'); rs.slaveOk(); printjson(db.system.users.find().toArray())'
2015-08-06 05:57:26 +0000 Puppet::Type::Mongodb_user::ProviderMongodb (debug): Request failed: 'Execution of '/usr/bin/mongo admin --quiet --port 27017 --eval load('/root/.mongorc.js'); rs.slaveOk(); printjson(db.system.users.find().toArray())' returned 252: 2015-08-06T05:57:26.853+0000 error: {
"$err" : "not master or secondary; cannot currently read from this replSet member",
"code" : 13436
} at src/mongo/shell/query.js:131
' Retry: '9'
2015-08-06 05:57:29 +0000 Puppet (err): Could not prefetch mongodb_user provider 'mongodb': Could not evalute MongoDB shell command: load('/root/.mongorc.js'); rs.slaveOk(); printjson(db.system.users.find().toArray())
/etc/puppet/modules/mongodb/lib/puppet/provider/mongodb.rb:70:in `mongo_eval'
/etc/puppet/modules/mongodb/lib/puppet/provider/mongodb_user/mongodb.rb:30:in `instances'
/etc/puppet/modules/mongodb/lib/puppet/provider/mongodb_user/mongodb.rb:45:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:277:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:167:in `prefetch_if_necessary'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:67:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `call'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
/usr/bin/puppet:4:in `<main>'
2015-08-06 05:57:29 +0000 /Stage[main]/Mongodb::Server/Mongodb_user[admin] (info): Starting to evaluate the resource
2015-08-06 05:57:29 +0000 /Stage[main]/Mongodb::Server/Mongodb_user[admin] (notice): Dependency Mongodb_replset[ceilometer] has failures: true
2015-08-06 05:57:29 +0000 /Stage[main]/Mongodb::Server/Mongodb_user[admin] (warning): Skipping because of failed dependencies
2015-08-06 05:57:29 +0000 /Stage[main]/Mongodb::Server/Mongodb_user[admin] (info): Evaluated in 0.00 seconds
raised to critical according a lot of duplicated here