Env deployments with Ceilo failed with Could not prefetch mongodb_user provider 'mongodb'

Bug #1482154 reported by Anastasia Kuznetsova
38
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Ivan Berezovskiy

Bug Description

SWARM gates with system tests for Ceilo and Heat (it install Ceilometer too) have failed
http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.services_ha.ceilometer/50/
http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.services_ha/51/

ENV: ISO 139, HA (3 controllers), Mongo on all controllers

Following error in astute.log:
2015-08-06T05:57:30 debug: [669] Node 1(primary-mongo) status: error
2015-08-06T05:57:30 debug: [669] Node 1 has failed to deploy. There is no more retries for puppet run.
2015-08-06T05:57:30 debug: [669] {"nodes"=>[{"status"=>"error", "error_type"=>"deploy", "uid"=>"1", "role"=>"primary-mongo"}]}
2015-08-06T05:57:30 err: [669] Task '{"priority"=>1000, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/roles/mongo.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 1
2015-08-06T05:57:30 err: [669] No more tasks will be executed on the node 1
2015-08-06T05:57:30 debug: [669] Data received by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"1", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-mongo", "task"=>{"priority"=>1000, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/roles/mongo.pp", "timeout"=>3600, "cwd"=>"/"}}}]}
2015-08-06T05:57:30 debug: [669] Data send by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"1", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-mongo", "task"=>{"priority"=>1000, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/roles/mongo.pp", "timeout"=>3600, "cwd"=>"/"}}, "progress"=>100}]}
2015-08-06T05:57:30 info: [669] 0bd8ab3a-a97b-4fd1-8583-d42f0fb0f694: Finished deployment of nodes => roles: {"1"=>"primary-mongo"}
2015-08-06T05:57:30 info: [669] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"0bd8ab3a-a97b-4fd1-8583-d42f0fb0f694", "nodes"=>[{"uid"=>"1", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-mongo", "task"=>{"priority"=>1000, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/roles/mongo.pp", "timeout"=>3600, "cwd"=>"/"}}, "progress"=>100}]}}
2015-08-06T05:57:30 err: [669] Error running RPC method granular_deploy: Deployment failed on nodes 1, trace:

2015-08-06T05:57:30 info: [669] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"0bd8ab3a-a97b-4fd1-8583-d42f0fb0f694", "status"=>"error", "error"=>"Method granular_deploy. Deployment failed on nodes 1.\nInspect Astute logs for the details"}}

Error in node-1 puppet logs:
2015-08-06 05:57:26 +0000 Puppet (debug): Executing '/usr/bin/mongo admin --quiet --port 27017 --eval load('/root/.mongorc.js'); rs.slaveOk(); printjson(db.system.users.find().toArray())'
2015-08-06 05:57:26 +0000 Puppet::Type::Mongodb_user::ProviderMongodb (debug): Request failed: 'Execution of '/usr/bin/mongo admin --quiet --port 27017 --eval load('/root/.mongorc.js'); rs.slaveOk(); printjson(db.system.users.find().toArray())' returned 252: 2015-08-06T05:57:26.853+0000 error: {
 "$err" : "not master or secondary; cannot currently read from this replSet member",
 "code" : 13436
} at src/mongo/shell/query.js:131
' Retry: '9'
2015-08-06 05:57:29 +0000 Puppet (err): Could not prefetch mongodb_user provider 'mongodb': Could not evalute MongoDB shell command: load('/root/.mongorc.js'); rs.slaveOk(); printjson(db.system.users.find().toArray())
/etc/puppet/modules/mongodb/lib/puppet/provider/mongodb.rb:70:in `mongo_eval'
/etc/puppet/modules/mongodb/lib/puppet/provider/mongodb_user/mongodb.rb:30:in `instances'
/etc/puppet/modules/mongodb/lib/puppet/provider/mongodb_user/mongodb.rb:45:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:277:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:167:in `prefetch_if_necessary'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:67:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `call'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
/usr/bin/puppet:4:in `<main>'
2015-08-06 05:57:29 +0000 /Stage[main]/Mongodb::Server/Mongodb_user[admin] (info): Starting to evaluate the resource
2015-08-06 05:57:29 +0000 /Stage[main]/Mongodb::Server/Mongodb_user[admin] (notice): Dependency Mongodb_replset[ceilometer] has failures: true
2015-08-06 05:57:29 +0000 /Stage[main]/Mongodb::Server/Mongodb_user[admin] (warning): Skipping because of failed dependencies
2015-08-06 05:57:29 +0000 /Stage[main]/Mongodb::Server/Mongodb_user[admin] (info): Evaluated in 0.00 seconds

Revision history for this message
Anastasia Kuznetsova (akuznetsova) wrote :
Changed in fuel:
assignee: nobody → Ivan Berezovskiy (iberezovskiy)
milestone: none → 7.0
Changed in fuel:
status: New → Confirmed
Changed in fuel:
status: Confirmed → In Progress
tags: added: tricky
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

raised to critical according a lot of duplicated here

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/211606

Revision history for this message
Ivan Berezovskiy (iberezovskiy) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/207512
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d375befc22a3f173a5dd47b5abc5a19a7ef556ff
Submitter: Jenkins
Branch: master

commit d375befc22a3f173a5dd47b5abc5a19a7ef556ff
Author: iberezovskiy <email address hidden>
Date: Thu Jul 30 18:46:55 2015 +0300

    Improvements for MongoDB replica set configuration

    * Remove upstream pull request which state is uncertain
      https://github.com/puppetlabs/puppetlabs-mongodb/pull/194
    * Move mongorc file creation in openstack class
    * Configure admin user in openstack class
    * Forbid to configure databases and users from slave nodes
      ('if' condition was added defore executing such commands)
    * Remove outdated noop tests
    * Remove useless connection validator because it runs
      during replica creation
      Partial-bug: #1475948

    Fix method which check existing master mongo:
      Closes-bug: #1482154
    It contains a few changes in mongodb module,
    but they will be proposed to upstream according to
    https://bugs.launchpad.net/fuel/+bug/1475948

    Change-Id: I108d2c4a70d5511c5d6601cfdf08191dabb31c1b

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Vitaly Gusev (vgusev) wrote :
Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to puppet-modules/puppetlabs-mongodb (0.10.0-mos)

Fix proposed to branch: 0.10.0-mos
Change author: Denis Egorenko <email address hidden>
Review: https://review.fuel-infra.org/11065

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to puppet-modules/puppetlabs-mongodb (0.10.0-mos)

Reviewed: https://review.fuel-infra.org/11065
Submitter: Ivan Berezovskiy <email address hidden>
Branch: 0.10.0-mos

Commit: 0c1b59ae152f3d4ad1a31dff2ba2457a909c7bbe
Author: Denis Egorenko <email address hidden>
Date: Mon Sep 21 10:44:08 2015

Improvements for MongoDB replica set configuration

* Remove upstream pull request which state is uncertain
  puppetlabs/puppetlabs-mongodb#194
* Move mongorc file creation in openstack class
* Configure admin user in openstack class
* Forbid to configure databases and users from slave nodes
  ('if' condition was added defore executing such commands)
* Remove outdated noop tests
* Remove useless connection validator because it runs
  during replica creation
  Partial-bug: #1475948

Fix method which check existing master mongo:
  Closes-bug: #1482154
It contains a few changes in mongodb module,
but they will be proposed to upstream according to
https://bugs.launchpad.net/fuel/+bug/1475948

Change-Id: I108d2c4a70d5511c5d6601cfdf08191dabb31c1b

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.