ERR: (/Stage[main]/Mongodb::Replset/Mongodb_replset[ceilometer]) Could not evaluate: rs.initiate() failed for replicaset ceilometer: not authorized on admin to execute command

[Anastasia Palkina]

1. Create new environment
2. Choose Neutron, tunnelling segmentation
3. Choose Ceilometer
4. Add 3 controller+mongo, 1 compute, 1 cinder, 3 mongo
5. Change disk configuration for all Mongo nodes. Change 'MongoDB' volume for vdc
6. Start deployment. Controller and mongo nodes in 'Error' state. Deployment has failed.

Error in puppet.log:

2016-01-27 10:32:37 ERR (/Stage[main]/Mongodb::Replset/Mongodb_replset[ceilometer]) Could not evaluate: rs.initiate() failed for replicaset ceilometer: not authorized on admin to execute command { replSetInitiate: { _id: "ceilometer", members: [ { _id: 0.0, host: "192.168.0.7:27017" }, { _id: 1.0, host: "192.168.0.6:27017" }, { _id: 2.0, host: "192.168.0.4:27017" }, { _id: 3.0, host: "192.168.0.9:27017" }, { _id: 4.0, host: "192.168.0.10:27017" } ] } }

This error appeared on primary controller (node-8), mongo nodes (node-4,5)

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "478"
  build_id: "478"
  fuel-nailgun_sha: "ae949905142507f2cb446071783731468f34a572"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "481ed135de2cb5060cac3795428625befdd1d814"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "420c6fa5f8cb51f3322d95113f783967bde9836e"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "6c6b088a3d52dd0eaf43d59f3a3a149c93a07e7e"

[Ivan Berezovskiy (iberezovskiy)]

I've researched environment and found root cause of this problem. Admin user wasn't created, because replica initialization took a lot of time. 10 retires wasn't enough to bring replica set up. After mongodb logs analysis I can say that the problem in very slow environment (weak vm for mongodb node). This retries should be improved later: timeout should not be just increased, we should be able to configure it somehow and pass into mongodb_replset provider.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/281198

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/281198
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f607d2b9a0d0e42386a84afc3ae0b2e6889edeee
Submitter: Jenkins
Branch: master

commit f607d2b9a0d0e42386a84afc3ae0b2e6889edeee
Author: iberezovskiy <email address hidden>
Date: Wed Feb 17 14:13:12 2016 +0300

    Pull upstream version of mongodb module

    0.12.0 is finally released. Our customization is included
    in it as it was implemented in following pull request:
    https://github.com/puppetlabs/puppetlabs-mongodb/pull/194
    This patch also contains improvements for timeouts during
    replica initialization and user/db creation.

    Additional changes:
      * get rid of useless openstack/mongo class
      * add an ability to configure some mongo parameters
        through hiera
      * move firewall rule to appropriate place

    Closes-bug: #1538521
    Closes-bug: #1475948

    Change-Id: I757b77fcf836f37a8136fa0987cca13ff646712a

[magicboiz (magicboiz)]

Hi

is there any patch/workaround available to deploy into MOS 8.0 (just released)? I'm facing this problem in our lab environment, we're getting messages like:

Notice: /Stage[main]/Mongodb::Replset/Mongodb_replset[ceilometer]/ensure: created
Debug: Checking replicaset member 10.100.2.4:27017 ...
Debug: Executing '/usr/bin/mongo admin --quiet --port 27017 --eval load('/root/.mongorc.js'); db.version()'
Debug: Executing '/usr/bin/mongo --quiet --host 10.100.2.4:27017 --eval load('/root/.mongorc.js'); printjson(rs.status())'
Warning: Host 10.100.2.4:27017 is available, but you are unauthorized because of authentication is enabled: true
Debug: Checking replicaset member 10.100.2.5:27017 ...
Debug: Executing '/usr/bin/mongo --quiet --host 10.100.2.5:27017 --eval load('/root/.mongorc.js'); printjson(rs.status())'
Warning: Host 10.100.2.5:27017 is available, but you are unauthorized because of authentication is enabled: true
Debug: Checking replicaset member 10.100.2.6:27017 ...
Debug: Executing '/usr/bin/mongo --quiet --host 127.0.0.1 --eval load('/root/.mongorc.js'); printjson(rs.status())'
Debug: Host ceilometer is alive but unconfigured: run rs.initiate(...) if not yet done for the set
Debug: Alive members: ["10.100.2.4:27017", "10.100.2.5:27017", "10.100.2.6:27017"]
Debug: Executing '/usr/bin/mongo --quiet --host 10.100.2.4:27017 --eval load('/root/.mongorc.js'); printjson(rs.status())'
Debug: Executing '/usr/bin/mongo --quiet --host 10.100.2.5:27017 --eval load('/root/.mongorc.js'); printjson(rs.status())'
Debug: Executing '/usr/bin/mongo --quiet --host 127.0.0.1 --eval load('/root/.mongorc.js'); printjson(rs.status())'
Debug: Initializing the replset ceilometer
Debug: Checking replicaset member 10.100.2.4:27017 ...
Debug: Executing '/usr/bin/mongo --quiet --host 10.100.2.4:27017 --eval load('/root/.mongorc.js'); printjson(rs.status())'
Warning: Host 10.100.2.4:27017 is available, but you are unauthorized because of authentication is enabled: true
Debug: Checking replicaset member 10.100.2.5:27017 ...
Debug: Executing '/usr/bin/mongo --quiet --host 10.100.2.5:27017 --eval load('/root/.mongorc.js'); printjson(rs.status())'
Warning: Host 10.100.2.5:27017 is available, but you are unauthorized because of authentication is enabled: true
Debug: Checking replicaset member 10.100.2.6:27017 ...
Debug: Executing '/usr/bin/mongo --quiet --host 127.0.0.1 --eval load('/root/.mongorc.js'); printjson(rs.status())'
Debug: Host ceilometer is alive but unconfigured: run rs.initiate(...) if not yet done for the set
Debug: Alive members: ["10.100.2.4:27017", "10.100.2.5:27017", "10.100.2.6:27017"]
Debug: Executing '/usr/bin/mongo --quiet --host 10.100.2.4:27017 --eval load('/root/.mongorc.js'); printjson(rs.initiate({ _id: 'ceilometer', members: [ { _id: 0, host: '10.100.2.4:27017'},{ _id: 1, host: '10.100.2.5:27017'},{ _id: 2, host: '10.100.2.6:27017'} ] }))'
Error: /Stage[main]/Mongodb::Replset/Mongodb_replset[ceilometer]: Could not evaluate: rs.initiate() failed for replicaset ceilometer: not authorized on admin to execute command { replSetInitiate: { _id: "ceilometer", members: [ { _id: 0.0, host: "10.100.2.4:27017" }, { _id: 1.0, host: "10.100.2.5:27017" ...

[Dmitry Bilunov (dbilunov)]

magicboiz, please try this one.

[Vladimir (vushakov)]

Verified on:
    Fuel 9.0 build #91 liberty-9.0