Deployment fail after master mongo moved from primary controller

[Ivan Ponomarev]

When primary controller was unaccessible or deleted from cluster master mongo moved to the other controller.
Puppet fails when primary-mongo role and master host for replicaset of mongoDB are placed on the different controller nodes

The possible solution is update mongoDB from upstream:
https://github.com/puppetlabs/puppetlabs-mongodb/blob/master/lib/puppet/provider/mongodb_replset/mongo.rb#L212-L217

Reproduced in fuel 6.1 and 7.0

Step to reproduce:
1) Build cluster using controller+mongodb
2) Delete primary-controller.
3) Add new controller to cluster.

The last tested version of fuel:
 release: "7.0"
 openstack_version: "2014.2.2-7.0"
 api: "1.0"
 build_number: "141"
 build_id: "2015-06-30_19-02-39"
 nailgun_sha: "9ea8b2cf30966b4805d0655b516c5a5d67d05865"
 python-fuelclient_sha: "7f50043963bda38b7857cfcb61ece2bcc5b9aa1c"
 astute_sha: "8d11177f91aab90f8034e9f6a479f343e1a4ec7f"
 fuel-library_sha: "21131baa6dbe2f79fc14cdc2667c301f0be97a9c"
 fuel-ostf_sha: "2eff8e314d3024707f32cf539a7997bfc427f351"
 fuelmain_sha: "b6f98e6b31cee7f9b925c6cc24e47966fa63e8ec"

[Alexander Kislitsky (akislitsky)]

Moving to library team

[Aleksandr Didenko (adidenko)]

Critical since it blocks one of the features

[Aleksandr Didenko (adidenko)]

The problem is not only with migration of mongodb master, the problem is that /etc/puppet/modules/osnailyfacter/modular/roles/mongo_primary.pp could not be applied successfully after the first run. Simply upgrading mongodb module will not help since we have issues with authentication after the first run. It may require refactoring of our manifests and deeper investigation of how to deploy primary mongo and replica set in order to provide mongo_primary.pp idempotency.

In order to reproduce the issue, you can simply run this command on a deployed node with primary-mongo role:
 puppet apply -d -v --noop /etc/puppet/modules/osnailyfacter/modular/roles/mongo_primary.pp

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/207457

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/207457
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b21a424da595339fc3f8a3836e9fb86b264939fd
Submitter: Jenkins
Branch: master

commit b21a424da595339fc3f8a3836e9fb86b264939fd
Author: Ivan Ponomarev <email address hidden>
Date: Thu Jul 30 15:48:25 2015 +0300

    Fix mongodb idempotence

      - Fix mongodb idempotence.
        1) Mongodb first run initiate mongo master replicas.
        2) The second run on other instances automatically initiate
        mongodb slave
      - Fix Puppet fails when primary-mongo role and master host for replicaset of
        mongoDB are placed on the different controller nodes
        Closes-bug: #1475296
      - Fix mongodb authentication problem after the first run
        Closes-bug: #1478871
      - Partially fix Adding MongoDB nodes
        Partial-bug: #1308990

        This patch set contains changes in upstream module which will be
        proposed to upstream in 8.0 timeline according to bug
        https://bugs.launchpad.net/fuel/+bug/1475948

    Change-Id: Ie34be251c2211d99afe81db3595b74867c548577

[Alexander Arzhanov (aarzhanov)]

Verified on ISO #183:

api: '1.0'
astute_sha: e24ca066bf6160bc1e419aaa5d486cad1aaa937d
auth_required: true
build_id: 2015-08-17_09-09-46
build_number: '183'
feature_groups:
- mirantis
fuel-agent_sha: 57145b1d8804389304cd04322ba0fb3dc9d30327
fuel-library_sha: 9de2625d26c3b88d22082baecd789b6bd5ddf3fa
fuel-nailgun-agent_sha: e01693992d7a0304d926b922b43f3b747c35964c
fuel-ostf_sha: 17786b86b78e5b66d2b1c15500186648df10c63d
fuelmain_sha: d8c726645be087bc67e2eeca134f0f9747cfeacd
nailgun_sha: 4710801a2f4a6d61d652f8f1e64215d9dde37d2e
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 4c74a60aa60c06c136d9197c7d09fa4f8c8e2863
release: '7.0'
release_versions:
  2015.1.0-7.0:
    VERSION:
      api: '1.0'
      astute_sha: e24ca066bf6160bc1e419aaa5d486cad1aaa937d
      build_id: 2015-08-17_09-09-46
      build_number: '183'
      feature_groups:
      - mirantis
      fuel-agent_sha: 57145b1d8804389304cd04322ba0fb3dc9d30327
      fuel-library_sha: 9de2625d26c3b88d22082baecd789b6bd5ddf3fa
      fuel-nailgun-agent_sha: e01693992d7a0304d926b922b43f3b747c35964c
      fuel-ostf_sha: 17786b86b78e5b66d2b1c15500186648df10c63d
      fuelmain_sha: d8c726645be087bc67e2eeca134f0f9747cfeacd
      nailgun_sha: 4710801a2f4a6d61d652f8f1e64215d9dde37d2e
      openstack_version: 2015.1.0-7.0
      production: docker
      python-fuelclient_sha: 4c74a60aa60c06c136d9197c7d09fa4f8c8e2863
      release: '7.0'

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 0.10.0-mos
Change author: Denis Egorenko <email address hidden>
Review: https://review.fuel-infra.org/11064

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11064
Submitter: Ivan Berezovskiy <email address hidden>
Branch: 0.10.0-mos

Commit: e09da4c1105fb6cbf4bc7c909ee352d0cd5ca0c5
Author: Denis Egorenko <email address hidden>
Date: Mon Sep 21 10:40:31 2015

Fix mongodb idempotence

  - Fix mongodb idempotence.
    1) Mongodb first run initiate mongo master replicas.
    2) The second run on other instances automatically initiate
    mongodb slave
  - Fix Puppet fails when primary-mongo role and master host for replicaset of
    mongoDB are placed on the different controller nodes
    Closes-bug: #1475296
  - Fix mongodb authentication problem after the first run
    Closes-bug: #1478871
  - Partially fix Adding MongoDB nodes
    Partial-bug: #1308990

This patch set contains changes in upstream module which will be
proposed to upstream in 8.0 timeline according to bug
https://bugs.launchpad.net/fuel/+bug/1475948

Change-Id: Ie34be251c2211d99afe81db3595b74867c548577