Deployment failed due inability to connect to mongodb server (after partition preservation)

[Vasily Gorin]

CI: https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.partition_preservation/82/testReport/(root)/mongo_mysql_partition_preservation/mongo_mysql_partition_preservation/

Filtered Log:
node-3.test.domain.local 2016-04-19T01:16:02.917967 err: Unable to connect to mongodb server! (127.0.0.1:27017)

node-3.test.domain.local 2016-04-19T01:16:02.928107 err: (/Stage[main]/Mongodb::Server::Service/Mongodb_conn_validator[mongodb]/ensure) change from absent to present failed: Unable to connect to mongodb server! (127.0.0.1:27017)
node-3.test.domain.local 2016-04-19T01:16:15.213183 err: Could not prefetch mongodb_replset provider 'mongo': Could not evalute MongoDB shell command: load('/root/.mongorc.js'); printjson(rs.conf())
node-3.test.domain.local 2016-04-19T01:16:46.318352 err: Could not prefetch mongodb_database provider 'mongodb': Could not evalute MongoDB shell command: load('/root/.mongorc.js'); printjson(db.getMongo().getDBs())
node-3.test.domain.local 2016-04-19T01:16:46.376416 err: Could not prefetch mongodb_user provider 'mongodb': Execution of '/usr/bin/mongo admin --quiet --host 127.0.0.1:27017 --eval load('/root/.mongorc.js'); printjson(db.isMaster())' returned 1: 2016-04-19T01:16:46.379+0000 warning: Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
node-3.test.domain.local 2016-04-19T01:16:46.376526 err: 2016-04-19T01:16:46.380+0000 Error: couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed at src/mongo/shell/mongo.js:148
node-3.test.domain.local 2016-04-19T01:16:46.376526 err: exception: connect failed

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Maksim Malchuk (mmalchuk)]

actually deploy failed due:

2016-04-19 00:54:42 +0000 Puppet (err): Could not run: Could not find file /etc/puppet/modules/openstack_tasks/examples/swift/proxy.pp

[Maksim Malchuk (mmalchuk)]

looks like we have a regression added by: https://review.openstack.org/#/c/305329/
investigating

[Sergey Novikov (snovikov)]

These tests
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.full_cluster_reinstallation/82/testReport/(root)/full_cluster_reinstallation/full_cluster_reinstallation/

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ready_node_reinstallation/82/testReport/(root)/reinstall_single_regular_controller_node/

are affected by this bug too

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/308368

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/308373

[Maksim Malchuk (mmalchuk)]

fix proposed and passed CI, but temporarily blocked by another issue: https://bugs.launchpad.net/fuel/+bug/1573105

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/308368
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f1f838b6effddad69dbb993b92a628ceec630599
Submitter: Jenkins
Branch: master

commit f1f838b6effddad69dbb993b92a628ceec630599
Author: Maksim Malchuk <email address hidden>
Date: Wed Apr 20 17:32:26 2016 +0300

    Fix swift proxy task dependencies

    Change-Id: I9de36798dddb404c3a40474211281a416217cb9c
    Closes-Bug: #1572104
    Signed-off-by: Maksim Malchuk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/308373
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b65fbd5753a17b4d11880298ebaa0766ed9d4683
Submitter: Jenkins
Branch: stable/mitaka

commit b65fbd5753a17b4d11880298ebaa0766ed9d4683
Author: Maksim Malchuk <email address hidden>
Date: Wed Apr 20 17:32:26 2016 +0300

    Fix swift proxy task dependencies

    Change-Id: I9de36798dddb404c3a40474211281a416217cb9c
    Closes-Bug: #1572104
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit f1f838b6effddad69dbb993b92a628ceec630599)

[Maksim Malchuk (mmalchuk)]

RCA: a regression was added by https://review.openstack.org/#/c/305329/1/deployment/puppet/openstack_tasks/examples/swift/tasks.yaml because custom-ci scenario didn't used in upstream ci.

[Dmitriy Kruglov (dkruglov)]

There was an occurrence of the issue on the latest swarm ran against MOS 9.0, build 424 -
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.partition_preservation/126/testReport/(root)/mongo_mysql_partition_preservation/mongo_mysql_partition_preservation/

Traceback from puppet.log http://paste.openstack.org/show/506856/

[Michael Polenchuk (mpolenchuk)]

in logs: mongodb main process terminated with status 100

EXIT_UNCAUGHT = 100 , // top level exception that wasn't caught

possible reasons:
 * permission issue on the data directory (/var/lib/mongo)
 * lock file exists

[Michael Polenchuk (mpolenchuk)]

Mongodb::Server::Config/File[/var/lib/mongo/mongodb]/owner (notice): owner changed 'aodh' to 'mongodb'
Mongodb::Server::Config/File[/var/lib/mongo/mongodb]/group (notice): group changed 'aodh' to 'mongodb'

[Maksim Malchuk (mmalchuk)]

9.0 ETA - 3 Jun 2016

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/323966

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/323952
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4ed71cd20e2f38d7c1184492d889e705f29175b6
Submitter: Jenkins
Branch: master

commit 4ed71cd20e2f38d7c1184492d889e705f29175b6
Author: Maksim Malchuk <email address hidden>
Date: Wed Jun 1 17:36:07 2016 +0300

    Fix mongodb permissions after partition preservation

    This would solve the problem with incorrect permisions of mongodb
    files after the partition preservation feature.

    Change-Id: Id874889f0af4a97ca6a098ca9a7a0052ec9695a9
    Closes-Bug: #1572104
    Co-Authored-By: Michael Polenchuk <email address hidden>
    Signed-off-by: Maksim Malchuk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/323966
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=265b3de11aeb8215073e9a24da12bd3971056fa0
Submitter: Jenkins
Branch: stable/mitaka

commit 265b3de11aeb8215073e9a24da12bd3971056fa0
Author: Maksim Malchuk <email address hidden>
Date: Wed Jun 1 17:36:07 2016 +0300

    Fix mongodb permissions after partition preservation

    This would solve the problem with incorrect permisions of mongodb
    files after the partition preservation feature.

    Change-Id: Id874889f0af4a97ca6a098ca9a7a0052ec9695a9
    Closes-Bug: #1572104
    Co-Authored-By: Michael Polenchuk <email address hidden>
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit 4ed71cd20e2f38d7c1184492d889e705f29175b6)

[Maksim Malchuk (mmalchuk)]

Fix in upstream also merged: https://github.com/puppetlabs/puppetlabs-mongodb/pull/310

[Dmitriy Kruglov (dkruglov)]

The issue is reproduced on the latest swarm run (MOS 9.0 ISO build 453), although the committed patch is present in the ISO.

The puppet.log trace http://paste.openstack.org/show/509008/

[Michael Polenchuk (mpolenchuk)]

LOGS:
Could not look up qualified variable '::mongodb::server::user'; class ::mongodb::server has not been evaluated

FIX:
include ::mongodb::params
$user = $::mongodb::params::user
$group = $::mongodb::params::group

DESC:
mongodb module has really confused params getter:
server <-> globals <-> config <-> params

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/327548

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/327954

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/327548
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c53136f947001ded0509a8c6f21d24f75f6e40ca
Submitter: Jenkins
Branch: master

commit c53136f947001ded0509a8c6f21d24f75f6e40ca
Author: Maksim Malchuk <email address hidden>
Date: Thu Jun 9 12:49:08 2016 +0300

    Fix for look up qualified variables '::mongodb::server::...'

    Change-Id: Ib056ada75ddf140f600dfaeec81726c7687497f3
    Closes-Bug: #1572104
    Signed-off-by: Maksim Malchuk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/327954
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=925133b90cf400ecd230f9ab2e7c4d3f8d003671
Submitter: Jenkins
Branch: stable/mitaka

commit 925133b90cf400ecd230f9ab2e7c4d3f8d003671
Author: Maksim Malchuk <email address hidden>
Date: Thu Jun 9 12:49:08 2016 +0300

    Fix for look up qualified variables '::mongodb::server::...'

    Change-Id: Ib056ada75ddf140f600dfaeec81726c7687497f3
    Closes-Bug: #1572104
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit c53136f947001ded0509a8c6f21d24f75f6e40ca)

[Kyrylo Romanenko (kromanenko)]

Job https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.partition_preservation/ is still failing. I`d like to reopen issue for clarification.
Vasily Gorin, could you look at that and check: do we have regression or some another bug causes test job to fail?

[Maksim Malchuk (mmalchuk)]

the partition preservation test for mongodb and mysql is broken, the related bugs were created:
1. https://bugs.launchpad.net/fuel/mitaka/+bug/1591212
2. https://bugs.launchpad.net/fuel/+bug/1591271

[Maksim Malchuk (mmalchuk)]

tested on mos 9.0 iso #465. both fixes exists.