Kolla mongodb error: can't find self in the replset config

Bug #1626339 reported by Matt McEuen
22
This bug affects 5 people
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
High
Jeffrey Zhang
Ocata
Fix Released
High
Jeffrey Zhang

Bug Description

In a multinode deployment using kolla-ansible master, deployment fails with a mongodb replication-related error: "can't find self in the replset config".

Logs:
...
TASK [mongodb : Copying over mongodb.conf] *************************************
changed: [192.168.1.1] => (item=mongodb.conf) => {"changed": true, "checksum": "980752ded7cd7eb6c4583ca8151e5b87e0d67be1", "dest": "/etc/kolla//mongodb/mongodb.conf", "gid": 0, "group": "root", "item": "mongodb.conf", "md5sum": "0c061ccb2a13cd5d05e0156bae2a055a", "mode": "0644", "owner": "root", "size": 305, "src": "/root/.ansible/tmp/ansible-tmp-1474464962.55-27417649920289/source", "state": "file", "uid": 0}

TASK [mongodb : Starting mongodb bootstrap container] **************************
changed: [192.168.1.1] => {"changed": true, "result": false}

TASK [mongodb : Starting mongodb container] ************************************
changed: [192.168.1.1] => {"changed": true, "result": false}

TASK [mongodb : Waiting for the mongodb startup] *******************************
ok: [192.168.1.1] => {"changed": false, "elapsed": 4, "path": null, "port": 27017, "search_regex": null, "state": "started"}

TASK [mongodb : Checking current replication status] ***************************
ok: [192.168.1.1 -> 192.168.1.1] => {"changed": false, "cmd": ["docker", "exec", "-t", "mongodb", "mongo", "192.168.0.221", "--quiet", "--eval", "rs.status().ok"], "delta": "0:00:00.281315", "end": "2016-09-21 08:36:22.924528", "rc": 0, "start": "2016-09-21 08:36:22.643213", "stderr": "", "stdout": "0", "stdout_lines": ["0"], "warnings": []}

TASK [mongodb : Copying the mongodb replication set bootstrap script] **********
changed: [192.168.1.1 -> localhost] => {"changed": true, "checksum": "c4df9fe1c7f76a2e7910051db00e9b0afeab0420", "dest": "/tmp/mongodb_bootstrap_replication_set.js", "gid": 0, "group": "root", "md5sum": "9c892a67480e445645deb087772aceeb", "mode": "0644", "owner": "root", "size": 179, "src": "/home/madgin/.ansible/tmp/ansible-tmp-1474464983.17-266716950686470/source", "state": "file", "uid": 0}

TASK [mongodb : Bootstraping the mongodb replication set] **********************
fatal: [192.168.1.1 -> 192.168.1.1]: FAILED! => {"changed": true, "cmd": ["docker", "exec", "-t", "mongodb", "mongo", "192.168.0.221", "--quiet", "--eval", "printjson(rs.initiate(\n {\n \"_id\" : \"rs0\",\n \"version\" : 1,\n \"members\" : [\n {\n \"_id\" : 1,\n \"host\" : \"192.168.1.1:27017\"\n } ]\n }\n))"], "delta": "0:00:00.267858", "end": "2016-09-21 08:36:24.310294", "failed": true, "failed_when_result": true, "rc": 0, "start": "2016-09-21 08:36:24.042436", "stderr": "", "stdout": "{\r\n\t\"ok\" : 0,\r\n\t\"errmsg\" : \"couldn't initiate : can't find self in the replset config\"\r\n}", "stdout_lines": ["{", "\t\"ok\" : 0,", "\t\"errmsg\" : \"couldn't initiate : can't find self in the replset config\"", "}"], "warnings": []}

Please let me know if there's additional information I can provide that would be helpful. Thanks!

Revision history for this message
Matt McEuen (mm9745) wrote :

Note: I'm not experienced enough with mongodb to interpret it, but the mongodb.conf file used by the (successfully started) mongodb container does end with:

replication:
  replSetName: rs0

Changed in kolla:
status: New → Triaged
Revision history for this message
Matt McEuen (mm9745) wrote :

I have reproduced this with the latest newton code, building from source on ubuntu. If there's any additional info I can provide, please let me know.

Changed in kolla:
importance: Undecided → High
Changed in kolla:
milestone: none → ocata-3
zhang kun (refrain)
Changed in kolla:
assignee: nobody → zhang kun (refrain)
zhang kun (refrain)
Changed in kolla:
status: Triaged → In Progress
Revision history for this message
zhang kun (refrain) wrote :

Fix Bootstraping the mongodb replication set

It will failed if you really set api_interface differ from storage_interface.We should bootstrap the mongodb replication set from the ip address of api_interface.

Review: https://review.openstack.org/422377

Changed in kolla-ansible:
status: New → In Progress
assignee: nobody → zhang kun (refrain)
Changed in kolla:
milestone: ocata-3 → ocata-rc1
Changed in kolla:
milestone: ocata-rc1 → pike-1
Revision history for this message
MarginHu (margin2017) wrote :

I met this issue, when this patch will be merged?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.openstack.org/441657

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (master)

Change abandoned by zhang kun (refrain@139.com) on branch: master
Review: https://review.openstack.org/441657
Reason: i had a parent commit

Revision history for this message
Jeffrey Zhang (jeffrey4l) wrote :

This should is not related to which interface should be used. but the replication set can not run in AIO env.

Revision history for this message
Jeffrey Zhang (jeffrey4l) wrote :

I got what's going on. will push a fix.

Changed in kolla-ansible:
importance: Undecided → High
no longer affects: kolla
Changed in kolla-ansible:
milestone: none → pike-1
assignee: zhang kun (refrain) → Jeffrey Zhang (jeffrey4l)
Revision history for this message
zhang kun (refrain) wrote :

Of course i run it in multiple node and multiple interface env.so that you know it,just do it.

Revision history for this message
Jeffrey Zhang (jeffrey4l) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.openstack.org/442999
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=8fc8d327b3a8d3b7577263ca88c63e2a011682b2
Submitter: Jenkins
Branch: master

commit 8fc8d327b3a8d3b7577263ca88c63e2a011682b2
Author: Jeffrey Zhang <email address hidden>
Date: Wed Mar 8 16:38:16 2017 +0800

    Fix mongodb cluster using wrong network interface

    MongoDB cluster only uses api_interface.

    Change-Id: If24a1c9bd685b268068cb676a28316636af52b78
    Closes-Bug: #1626339

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/450218

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/ocata)

Reviewed: https://review.openstack.org/450218
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=bc1af3a6e6357749e670e1e999a2d57b6d2778e0
Submitter: Jenkins
Branch: stable/ocata

commit bc1af3a6e6357749e670e1e999a2d57b6d2778e0
Author: Jeffrey Zhang <email address hidden>
Date: Wed Mar 8 16:38:16 2017 +0800

    Fix mongodb cluster using wrong network interface

    MongoDB cluster only uses api_interface.

    Change-Id: If24a1c9bd685b268068cb676a28316636af52b78
    Closes-Bug: #1626339
    (cherry picked from commit 8fc8d327b3a8d3b7577263ca88c63e2a011682b2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 4.0.1

This issue was fixed in the openstack/kolla-ansible 4.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 5.0.0.0b2

This issue was fixed in the openstack/kolla-ansible 5.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (master)

Change abandoned by Mark Goddard (<email address hidden>) on branch: master
Review: https://review.opendev.org/422377
Reason: Very old

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.