[fuel-library] rs.initiate() failed on secondary node during task-based deployment

Bug #1549090 reported by Andrey Bubyr on 2016-02-24
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Critical
Dmitry Bilunov

Bug Description

Fuel 9.0 (task-based deployment is enabled by default there)
Environment:
- 3 controllers + mongo + cinder
- 1 compute node
- Ceilometer enabled

On in fact secondary Mongo node (node-2, IP 10.109.11.7) Puppet claims about this error:

/Stage[main]/Mongodb::Replset/Mongodb_replset[ceilometer] (err): Could not evaluate: rs.initiate() failed for replicaset ceilometer: not authorized on admin to execute command { replSetInitiate: { _id: "ceilometer", members: [ { _id: 0.0, host: "10.109.11.6:27017" }, { _id: 1.0, host: "10.109.11.7:27017" } ] } }
/etc/puppet/modules/mongodb/lib/puppet/provider/mongodb_replset/mongodb.rb:265:in `set_members'
/etc/puppet/modules/mongodb/lib/puppet/provider/mongodb_replset/mongodb.rb:62:in `flush'
...
2016-02-24 03:07:27 +0000 Puppet (warning): Impossible to get users info from slave
2016-02-24 03:07:27 +0000 /Stage[main]/Openstack::Mongo/Mongodb_user[admin] (notice): Dependency Mongodb_replset[ceilometer] has failures: true
2016-02-24 03:07:27 +0000 /Stage[main]/Openstack::Mongo/Mongodb_user[admin] (warning): Skipping because of failed dependencies
2016-02-24 03:07:27 +0000 /Stage[main]/Openstack::Mongo/Notify[mongodb configuring ceilometer database] (notice): Dependency Mongodb_replset[ceilometer] has failures: true
2016-02-24 03:07:27 +0000 /Stage[main]/Openstack::Mongo/Notify[mongodb configuring ceilometer database] (warning): Skipping because of failed dependencies
2016-02-24 03:07:27 +0000 Puppet (warning): Impossible to get databases from slave
2016-02-24 03:07:27 +0000 /Stage[main]/Openstack::Mongo/Mongodb::Db[ceilometer]/Mongodb_database[ceilometer] (notice): Dependency Mongodb_replset[ceilometer] has failures: true
2016-02-24 03:07:27 +0000 /Stage[main]/Openstack::Mongo/Mongodb::Db[ceilometer]/Mongodb_database[ceilometer] (warning): Skipping because of failed dependencies
2016-02-24 03:07:27 +0000 /Stage[main]/Openstack::Mongo/Mongodb::Db[ceilometer]/Mongodb_user[ceilometer] (notice): Dependency Mongodb_replset[ceilometer] has failures: true
2016-02-24 03:07:27 +0000 /Stage[main]/Openstack::Mongo/Mongodb::Db[ceilometer]/Mongodb_user[ceilometer] (warning): Skipping because of failed dependencies
2016-02-24 03:07:27 +0000 /Stage[main]/Openstack::Mongo/Notify[mongodb finished] (notice): Dependency Mongodb_replset[ceilometer] has failures: true
2016-02-24 03:07:27 +0000 /Stage[main]/Openstack::Mongo/Notify[mongodb finished] (warning): Skipping because of failed dependencies
(see full puppet.log in attachment)
node-2 rs.status():
MongoDB shell version: 2.6.10
connecting to: test
ceilometer:SECONDARY> rs.status()
{
        "set" : "ceilometer",
        "date" : ISODate("2016-02-24T04:08:55Z"),
        "myState" : 2,
        "syncingTo" : "10.109.11.6:27017",
        "members" : [
                {
                        "_id" : 0,
                        "name" : "10.109.11.6:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 3690,
                        "optime" : Timestamp(1456283242, 1),
                        "optimeDate" : ISODate("2016-02-24T03:07:22Z"),
                        "lastHeartbeat" : ISODate("2016-02-24T04:08:54Z"),
                        "lastHeartbeatRecv" : ISODate("2016-02-24T04:08:54Z"),
                        "pingMs" : 1,
                        "electionTime" : Timestamp(1456283250, 1),
                        "electionDate" : ISODate("2016-02-24T03:07:30Z")
                },
                {
                        "_id" : 1,
                        "name" : "10.109.11.7:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 3703,
                        "optime" : Timestamp(1456283242, 1),
                        "optimeDate" : ISODate("2016-02-24T03:07:22Z"),
                        "self" : true
                }
        ],
        "ok" : 1
}

The another secondary mongo node (node-1, IP 10.109.11.4) contains this error:
2016-02-24 03:40:22 +0000 Scope(Class[main]) (notice): MODULAR: mongo.pp
2016-02-24 03:40:25 +0000 Puppet (notice): Compiled catalog for node-1.test.domain.local in environment production in 3.18 seconds
2016-02-24 03:40:33 +0000 Puppet (notice): MongoDB params: 127.0.0.110.109.11.4
2016-02-24 03:40:33 +0000 /Stage[main]/Openstack::Mongo/Notify[MongoDB params: 127.0.0.110.109.11.4]/message (notice): defined 'message' as 'MongoDB params: 127.0.0.110
.109.11.4'
2016-02-24 03:40:33 +0000 /Stage[main]/Mongodb::Replset/Mongodb_replset[ceilometer]/ensure (notice): created
2016-02-24 03:40:33 +0000 Puppet (warning): Host 10.109.11.6:27017 is available, but you are unauthorized because of authentication is enabled: true
2016-02-24 03:40:34 +0000 Puppet (warning): Host 10.109.11.7:27017 is available, but you are unauthorized because of authentication is enabled: true
2016-02-24 03:40:34 +0000 Puppet (warning): Host 10.109.11.6:27017 is available, but you are unauthorized because of authentication is enabled: true
2016-02-24 03:40:34 +0000 Puppet (warning): Host 10.109.11.7:27017 is available, but you are unauthorized because of authentication is enabled: true
2016-02-24 03:40:34 +0000 /Stage[main]/Mongodb::Replset/Mongodb_replset[ceilometer] (err): Could not evaluate: rs.initiate() failed for replicaset ceilometer: couldn't
initiate : member 10.109.11.6:27017 is already initiated
/etc/puppet/modules/mongodb/lib/puppet/provider/mongodb_replset/mongodb.rb:265:in `set_members'
/etc/puppet/modules/mongodb/lib/puppet/provider/mongodb_replset/mongodb.rb:62:in `flush'
...
2016-02-24 03:40:34 +0000 Puppet (warning): Impossible to get users info from slave
2016-02-24 03:40:34 +0000 /Stage[main]/Openstack::Mongo/Mongodb_user[admin] (notice): Dependency Mongodb_replset[ceilometer] has failures: true
2016-02-24 03:40:34 +0000 /Stage[main]/Openstack::Mongo/Mongodb_user[admin] (warning): Skipping because of failed dependencies
2016-02-24 03:40:34 +0000 /Stage[main]/Openstack::Mongo/Notify[mongodb configuring ceilometer database] (notice): Dependency Mongodb_replset[ceilometer] has failures: true
2016-02-24 03:40:34 +0000 /Stage[main]/Openstack::Mongo/Notify[mongodb configuring ceilometer database] (warning): Skipping because of failed dependencies

node-1 (ip 10.109.11.4) rs.status():
> rs.status()
{
        "startupStatus" : 3,
        "info" : "run rs.initiate(...) if not yet done for the set",
        "ok" : 0,
        "errmsg" : "can't get local.system.replset config from self or any seed (EMPTYCONFIG)"
}

At the same time there is no mongo-related errors in PRIMARY mongo node (node-3, IP 10.109.11.6) puppet.log
node-3 rs.status():
ceilometer:PRIMARY> rs.status()
{
        "set" : "ceilometer",
        "date" : ISODate("2016-02-24T04:10:19Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 0,
                        "name" : "10.109.11.6:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 3793,
                        "optime" : Timestamp(1456283242, 1),
                        "optimeDate" : ISODate("2016-02-24T03:07:22Z"),
                        "electionTime" : Timestamp(1456283250, 1),
                        "electionDate" : ISODate("2016-02-24T03:07:30Z"),
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "10.109.11.7:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 3777,
                        "optime" : Timestamp(1456283242, 1),
                        "optimeDate" : ISODate("2016-02-24T03:07:22Z"),
                        "lastHeartbeat" : ISODate("2016-02-24T04:10:18Z"),
                        "lastHeartbeatRecv" : ISODate("2016-02-24T04:10:18Z"),
                        "pingMs" : 1,
                        "syncingTo" : "10.109.11.6:27017"
                }
        ],
        "ok" : 1
}

NOTE: this bug is not reproduced if task-based deployment engine is disabled

Andrey Bubyr (abubyr) wrote :
Andrey Bubyr (abubyr) on 2016-02-24
summary: - [mongo] rs.initiate() failed on secondary node during task-based
+ [fuel-library] rs.initiate() failed on secondary node during task-based
deployment
Dmitry Klenov (dklenov) on 2016-02-24
tags: added: area-library
Changed in fuel:
milestone: none → 9.0
assignee: nobody → Fuel Library Team (fuel-library)
importance: Undecided → High
status: New → Confirmed
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Dmitry Bilunov (dbilunov)
tags: added: team-bugfixx
Andrey Bubyr (abubyr) on 2016-02-24
tags: added: feature-task-based
tags: added: team-bugfix
removed: team-bugfixx
Changed in fuel:
importance: High → Critical
Changed in fuel:
status: Confirmed → In Progress
Vladimir Kuklin (vkuklin) wrote :
Dmitry Bilunov (dbilunov) wrote :
Changed in fuel:
status: In Progress → Fix Committed
Vladimir (vushakov) on 2016-03-24
tags: added: on-verification
Vladimir (vushakov) wrote :

Verified on:
    Fuel 9.0 ISO #101

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers