Adding MongoDB nodes

[Alexander Kislitsky]

Release 5.0

Steps:
- Create environment
- Add controller, compute, number of mongo nodes
- Deploy changes
- Add more mongo node(-s)
- Deploy changes

Expected:
First part of mongo nodes should be configured as primary-secondary.
Second part of mongo nodes should be properly added into the mongo primary-secondary structure. Primary-node should be reconfigured for using new nodes in replset.

Workaround:

1. Add new nodes and set 'mongo' role via CLI (adding additional mongo nodes is disallowed in UI) and deploy.

2. Find primary mongo node:
 - Open /etc/astute.yaml on any deployed node and find node with primary-mongo role
 - Retrieve db_password from ceilometer section in same file

3. Connect to the primary-mongo node and log into mongo
  # mongo -u admin -p <db_password> admin

4.For each new node
  ceilometer:PRIMARY> rs.add("management ip address of each new node")

5. On each controller node:
  Add new hosts to connection= line in /etc/ceilometer/ceilometer.conf

6. Restart ceilometer services

[Alexander Kislitsky (akislitsky)]

Depends on: https://bugs.launchpad.net/fuel/+bug/1308988, https://bugs.launchpad.net/fuel/+bug/1308982

[Max Mazur (mmaxur)]

I don't support adding mongo slave nodes to existing replicaset. Also it is necessary to reconfigure all ceilometer nodes and primary mongo node to do this.

[Dmitry Ilyin (idv1985)]

We are not going to support adding of Mongo nodes... at least for now. It will require implementation of complex reconfiguration logic.

[Vladimir Kuklin (vkuklin)]

this requires complex deployment mechanism addressed in fuel-modularization blueprint

[Meg McRoberts (dreidellhasa)]

Listed as "Known Issue" in 5.0.1 Release Notes.

[Meg McRoberts (dreidellhasa)]

Listed as "Known Issue" in 5.0 Release Notes.

[Matthew Mosesohn (raytrac3r)]

This is not targeted for 6.0. Moving to 'next' release.

[Dmitry Borodaenko (angdraug)]

We only use "next" milestone for blueprints, bugs should always be targeted to a real release milestone.

[Dmitry Borodaenko (angdraug)]

Vladimir,

https://blueprints.launchpad.net/fuel/+spec/fuel-modularization returns 404, which blueprint did you mean?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/146200

[Ryan Moe (rmoe)]

Vladimir,

Is this the blueprint you're talking about https://blueprints.launchpad.net/fuel/+spec/fuel-library-modularization? If so, will this issue be fixed in 6.1 as part of its implementation?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/146200
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=b200fc058a1d28b6d0d160de559b840e7227e6cd
Submitter: Jenkins
Branch: master

commit b200fc058a1d28b6d0d160de559b840e7227e6cd
Author: Meg McRoberts <email address hidden>
Date: Fri Jan 9 13:03:55 2015 -0800

    How to add MongoDB nodes using shell commands

    This should be merged to master and stable/6.0

    Added "Add MongoDB nodes" to Ops-Guide section about redeploying nodes

    Added link to that section from Release Notes; reworded Release Notes to
    say MongoDB nodes can not be added using the Fuel Web UI.

    Added information to the fuel-ref/astute.yaml page about
    the ceilometer/mongodb settings that are referenced from this procedure.

    I added the full contents of the Target astute.yaml file and did a quick
    pass to divide it into sections that can be discussed reasonably; most of
    the work on this reference page will be done in a separate CR.

    I did fix the file-ref doc so the TOC only shows the top-level section
    (the file name) rather than all the subsections.

    Change-Id: I237f15f7f57b2b4532566ea74fc6ca701124fee8
    Partial-Bug: 1308990

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/155076

[Dmitry Borodaenko (angdraug)]

Workaround documented, resetting status back to Triaged for 6.1 (leaving status for 6.0.x and earlier as Won't Fix assuming the fix will be too intrusive to backport).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/155076
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=a146195e2d3b824ecc1bf30aeb69f23006de6a87
Submitter: Jenkins
Branch: stable/6.0

commit a146195e2d3b824ecc1bf30aeb69f23006de6a87
Author: Meg McRoberts <email address hidden>
Date: Fri Jan 9 13:03:55 2015 -0800

    How to add MongoDB nodes using shell commands

    This should be merged to master and stable/6.0

    Added "Add MongoDB nodes" to Ops-Guide section about redeploying nodes

    Added link to that section from Release Notes; reworded Release Notes to
    say MongoDB nodes can not be added using the Fuel Web UI.

    Added information to the fuel-ref/astute.yaml page about
    the ceilometer/mongodb settings that are referenced from this procedure.

    I added the full contents of the Target astute.yaml file and did a quick
    pass to divide it into sections that can be discussed reasonably; most of
    the work on this reference page will be done in a separate CR.

    I did fix the file-ref doc so the TOC only shows the top-level section
    (the file name) rather than all the subsections.

    Change-Id: I237f15f7f57b2b4532566ea74fc6ca701124fee8
    Partial-Bug: 1308990
    (cherry picked from commit b200fc058a1d28b6d0d160de559b840e7227e6cd)

[Mike Scherbakov (mihgen)]

Didn't we fix this one already? If not, can we involve Ceilometer team here?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/179071

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/179075

[Aleksandr Didenko (adidenko)]

Mike, currently it's not allowed to add mongodb nodes via Fuel UI on a deployed cluster (operational env).

So to make this possible we need to:

1) always configure replica set in mongo (even when we deploy only one mongo node)
https://review.openstack.org/179071

2) allow adding mongodb nodes in UI
https://review.openstack.org/179075

3) fix adding nodes to replica set here (it requires authentication):
https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/mongodb/manifests/replset/install.pp#L18

Currently it gives: http://paste.openstack.org/show/213619/

4) trigger re-deploy on all controller and compute nodes to update ceilometer configs

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/180522

[Ivan Berezovskiy (iberezovskiy)]

To make adding mongodb nodes possible we should:

1. Always configure mongo with replica set. This patch https://review.openstack.org/179071 implements this.

2. Allow to add mongo nodes from UI. Thanks Aleksandr Didenko for the patch https://review.openstack.org/179075

3. When mongo is already deployed, we can't enter in mongo shell and can't add new mongo node to replica without authentication.
This patch allow to login as 'admin' user into mongo and add nodes: https://review.openstack.org/#/c/180522/

4. After deploying new mongo node we should notify primary-mongo node for redeploy (use https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/roles/mongo_primary.pp). We should determine which mongo is primary now in cluster and execute commands for adding nodes (which implemented in 3 step) from it.

5. Also we should have tool to avoid failures on new mongo node. For example, similar bug (with adding controller node) is https://bugs.launchpad.net/fuel/+bug/1394188

6. After redeploy of primary mongo node we should run ceilometer puppet module on each controller in cluster. This redeploy should change ceilometer configuration option - connection string to database, and restart all ceilometer services. Rerun of ceilometer manifests https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/ceilometer/controller.pp should do this.

Now, we don't have enough time for implementing 4-6 steps and testing. So, it should be implemented later, for 7.0.

Assign to fuel documentation team for 6.1 release.

[Bogdan Dobrelya (bogdando)]

Note, this is a known issue since the 6.0 release https://docs.mirantis.com/openstack/fuel/fuel-6.0/release-notes.html?#issues-in-other-components

[Aleksandr Didenko (adidenko)]

Ivan, please take a look at https://review.openstack.org/#/c/179071/ - I've updated it to use special puppet resource instead of set of execs. It should work better and we can improve resource provider further if we need to.

As for primary - since we have a "sticky" primary role in our logic (primary-mongo role) maybe we should try to use replica set members priorities? Here's the doc:

http://docs.mongodb.org/manual/tutorial/force-member-to-be-primary/#force-a-member-to-be-primary-by-setting-its-priority-high

So we'll just set highest priority for 'primary-mongo' role and it will always be primary in replica set, unless it's not down of course.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/186446

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Berezovskiy (<email address hidden>) on branch: master
Review: https://review.openstack.org/180522
Reason: https://review.openstack.org/#/c/186446/

[Dmitry Borodaenko (angdraug)]

Is there any more documentation work to be done on this in 6.1?

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Berezovskiy (<email address hidden>) on branch: master
Review: https://review.openstack.org/186446

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/6.1
Review: https://review.openstack.org/194961

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/194961
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=0e26e7d7cc153d179ec34985645dd23cdd239ddb
Submitter: Jenkins
Branch: stable/6.1

commit 5cc5f0c643aebecaf3bf4580535a3ea7c3334a6c
Author: Mike Scherbakov <email address hidden>
Date: Tue Jun 23 13:43:35 2015 -0700

    Removed streamlined patching backend pieces

    Change-Id: I955e76ccdbd12a9145f4e9b689f80bdf9fcaf929

commit 563c4b5c78ebfcb1f4f91047c2919f6270f9a1d4
Author: Mike Scherbakov <email address hidden>
Date: Tue Jun 23 13:30:30 2015 -0700

    Removed outdated patching guide

    Change-Id: I76180c277789ade9c5ebedd19fe2092847c0b7d9

commit 8d120c14bec1ab41d448683ad146a3053a57c4ee
Author: Irina Povolotskaya <email address hidden>
Date: Tue Jun 23 19:59:11 2015 +0300

    Add dual hypervisor ref arch into 6.1 docs

    Change-Id: I900c24c9de878eafadbfc995aa879b7f55737fac

commit feebd1592d3305b64bbdfd0bc5fe108190aef120
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 18:38:17 2015 +0300

    [OPs guide] Running Ceilometer section edits

    1. conf file extract is updated
    2. note is updated

    Closes-bug: 1467817
    Change-Id: I0217e164108e0ba6c1397045a5e57d13ff429223

commit 44a93f9dead7511a3461ec35248dbb689c81eafd
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 18:04:40 2015 +0300

    [RN6_1] Final changes

    1. capitalization
    2. 2014.2 to 2014.2.2
    3. general improvements

    Change-Id: I45057e90c90550559f66bc67ccdf97a559fd9000

commit bb41389cae58084285688853281516b659686422
Author: evkonstantinov <email address hidden>
Date: Tue Jun 23 16:45:35 2015 +0300

    Update patching decription

    Update patching description with
    the standard Linux commands.

    Change-Id: Ia1a8346639c468fdfce15a11d2430bf3a4731244

commit bf3018fae3f2e564413d33aba6cdebf8868f0b4e
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 15:55:49 2015 +0300

    [RN6_1] Clean up

    1. Rearranges sections
    2. Improves RST
    3. Changes titles order

    Change-Id: I6110bf515667d3d6ba08ad35ff5d593dbc96641e

commit 1c7e4457808e8f2d6c56fdf31252170972e444b9
Author: Maria Zlatkova <email address hidden>
Date: Tue Jun 23 15:26:28 2015 +0300

    Replaces VBOX screenshots

    This patch:
    - replaces VBOX screenshots
    - changes the link for Download Mirantis VirtualBox scripts
     to https://docs.mirantis.com/openstack/fuel/fuel-master/#downloads

    Change-Id: I58dede960c5c3355d39b07ff44b757403f6af02c
    Closes-Bug: #1467872

commit 0a568bf53fc0e25d1d692d5d74b4a7b4d983bbcc
Author: evkonstantinov <email address hidden>
Date: Tue Jun 23 14:01:55 2015 +0300

    6.1 --separate repos

    change wording and add links to the
    separate repos feature.

    Change-Id: Ib5d0778a0d8f1534f79ed2f553574cb69a3150b0

commit 95a188b21cbdd064d92696b7920e6a0105fe0c56
Author: Maria Zlatkova <email address hidden>
Date: Tue Jun 23 12:07:28 2015 +0300

    Corrects the output 'pcs status'

    Changes the example outputs to appropriate ones.

    Change-Id: Ib6d83...

[OpenStack Infra (hudson-openstack)]

Change abandoned by Aleksandr Didenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/179071
Reason: Superseded by I3b9cbd35bf46483782d7b7eb7ef030fd31fdd544

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/192597
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=ceeb76af18edf58f2bce1881eb1acffa59f8e7e7
Submitter: Jenkins
Branch: master

commit ceeb76af18edf58f2bce1881eb1acffa59f8e7e7
Author: iberezovskiy <email address hidden>
Date: Wed Jun 17 13:27:58 2015 +0300

    Adapt synced puppetlabs-mongodb module for Fuel usage

      Key moments:
        * Adds Support for Keyfile Key:
          upstream commit 7bbcff8024152b4ec9ce53dbba00024523707162
        * Remove useless openstack::mongo class
        * Add upstream changes for 217 pull request
          (https://github.com/puppetlabs/puppetlabs-mongodb/pull/217):
          - Implement retries for MongoDB shell commands
          - Initiate replica set creation from localhost if auth is enabled
        * Improve support for MongoDB authentication
          https://github.com/puppetlabs/puppetlabs-mongodb/pull/194
        * Fix lint errors (actually fixed in upstream)
        * Add noop tests for MongoDB parameters

      Implements: blueprint upgrade-openstack-puppet-modules

      Closes-bug: #1383639
      Related-bug: #1308990

    Change-Id: I3b9cbd35bf46483782d7b7eb7ef030fd31fdd544

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/207457
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b21a424da595339fc3f8a3836e9fb86b264939fd
Submitter: Jenkins
Branch: master

commit b21a424da595339fc3f8a3836e9fb86b264939fd
Author: Ivan Ponomarev <email address hidden>
Date: Thu Jul 30 15:48:25 2015 +0300

    Fix mongodb idempotence

      - Fix mongodb idempotence.
        1) Mongodb first run initiate mongo master replicas.
        2) The second run on other instances automatically initiate
        mongodb slave
      - Fix Puppet fails when primary-mongo role and master host for replicaset of
        mongoDB are placed on the different controller nodes
        Closes-bug: #1475296
      - Fix mongodb authentication problem after the first run
        Closes-bug: #1478871
      - Partially fix Adding MongoDB nodes
        Partial-bug: #1308990

        This patch set contains changes in upstream module which will be
        proposed to upstream in 8.0 timeline according to bug
        https://bugs.launchpad.net/fuel/+bug/1475948

    Change-Id: Ie34be251c2211d99afe81db3595b74867c548577

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/179075
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=f0223ea7d34ca2ac38bc7c17219c8acc57bc688a
Submitter: Jenkins
Branch: master

commit f0223ea7d34ca2ac38bc7c17219c8acc57bc688a
Author: Aleksandr Didenko <email address hidden>
Date: Thu Apr 30 16:44:45 2015 +0300

    Allow to add MongoDB nodes on operational env

    New MongoDB node has ability to add yourself
    to MongoDB cluster according to this change:
    Ie34be251c2211d99afe81db3595b74867c548577

    Closes-bug: #1308990

    Change-Id: I72b4145e74abb003d0974d5980a884574e26d770

[Alexander Arzhanov (aarzhanov)]

Verified on ISO #183:

api: '1.0'
astute_sha: e24ca066bf6160bc1e419aaa5d486cad1aaa937d
auth_required: true
build_id: 2015-08-17_09-09-46
build_number: '183'
feature_groups:
- mirantis
fuel-agent_sha: 57145b1d8804389304cd04322ba0fb3dc9d30327
fuel-library_sha: 9de2625d26c3b88d22082baecd789b6bd5ddf3fa
fuel-nailgun-agent_sha: e01693992d7a0304d926b922b43f3b747c35964c
fuel-ostf_sha: 17786b86b78e5b66d2b1c15500186648df10c63d
fuelmain_sha: d8c726645be087bc67e2eeca134f0f9747cfeacd
nailgun_sha: 4710801a2f4a6d61d652f8f1e64215d9dde37d2e
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 4c74a60aa60c06c136d9197c7d09fa4f8c8e2863
release: '7.0'
release_versions:
  2015.1.0-7.0:
    VERSION:
      api: '1.0'
      astute_sha: e24ca066bf6160bc1e419aaa5d486cad1aaa937d
      build_id: 2015-08-17_09-09-46
      build_number: '183'
      feature_groups:
      - mirantis
      fuel-agent_sha: 57145b1d8804389304cd04322ba0fb3dc9d30327
      fuel-library_sha: 9de2625d26c3b88d22082baecd789b6bd5ddf3fa
      fuel-nailgun-agent_sha: e01693992d7a0304d926b922b43f3b747c35964c
      fuel-ostf_sha: 17786b86b78e5b66d2b1c15500186648df10c63d
      fuelmain_sha: d8c726645be087bc67e2eeca134f0f9747cfeacd
      nailgun_sha: 4710801a2f4a6d61d652f8f1e64215d9dde37d2e
      openstack_version: 2015.1.0-7.0
      production: docker
      python-fuelclient_sha: 4c74a60aa60c06c136d9197c7d09fa4f8c8e2863
      release: '7.0'

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 0.10.0-mos
Change author: Denis Egorenko <email address hidden>
Review: https://review.fuel-infra.org/11063

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 0.10.0-mos
Change author: Denis Egorenko <email address hidden>
Review: https://review.fuel-infra.org/11064

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11063
Submitter: Ivan Berezovskiy <email address hidden>
Branch: 0.10.0-mos

Commit: f2846cddafdfe79b93f72959803c981b567c23e2
Author: Denis Egorenko <email address hidden>
Date: Mon Sep 21 10:08:06 2015

Adapt synced puppetlabs-mongodb module for Fuel usage

Key moments:
    * Adds Support for Keyfile Key:
      upstream commit 7bbcff8024152b4ec9ce53dbba00024523707162
    * Remove useless openstack::mongo class
    * Add upstream changes for 217 pull request
      (https://github.com/puppetlabs/puppetlabs-mongodb/pull/217):
      - Implement retries for MongoDB shell commands
      - Initiate replica set creation from localhost if auth is enabled
    * Improve support for MongoDB authentication
      https://github.com/puppetlabs/puppetlabs-mongodb/pull/194
    * Fix lint errors (actually fixed in upstream)
    * Add noop tests for MongoDB parameters
  Implements: blueprint upgrade-openstack-puppet-modules
  Closes-bug: #1383639
  Related-bug: #1308990

Change-Id: I3b9cbd35bf46483782d7b7eb7ef030fd31fdd544

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11064
Submitter: Ivan Berezovskiy <email address hidden>
Branch: 0.10.0-mos

Commit: e09da4c1105fb6cbf4bc7c909ee352d0cd5ca0c5
Author: Denis Egorenko <email address hidden>
Date: Mon Sep 21 10:40:31 2015

Fix mongodb idempotence

  - Fix mongodb idempotence.
    1) Mongodb first run initiate mongo master replicas.
    2) The second run on other instances automatically initiate
    mongodb slave
  - Fix Puppet fails when primary-mongo role and master host for replicaset of
    mongoDB are placed on the different controller nodes
    Closes-bug: #1475296
  - Fix mongodb authentication problem after the first run
    Closes-bug: #1478871
  - Partially fix Adding MongoDB nodes
    Partial-bug: #1308990

This patch set contains changes in upstream module which will be
proposed to upstream in 8.0 timeline according to bug
https://bugs.launchpad.net/fuel/+bug/1475948

Change-Id: Ie34be251c2211d99afe81db3595b74867c548577