upgrade from newton to ocata: error with mongodb

[Javier Castillo]

Trying to upgrade from newton to ocata, ubuntu/source installation, mongodb deployed, upgrade task fails when

TASK [mongodb : Waiting for the mongodb startup] *******************************
fatal: [control01]: FAILED! => {"changed": false, "elapsed": 300, "failed": true, "msg": "Timeout when waiting for 10.20.0.11:27017"}

inspecting container logs:

2017-02-27T15:48:54.628+0100 [initandlisten] options: { command: [ "run" ], config: "/etc/mongodb.conf", net: { bindIp: "10.20.0.11", port: 27017, unixDomainSocket: { pathPrefix: "/var/run/mongodb" } }, replication: { replSetName: "rs0" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/kolla/mongodb/mongodb.log" } }
2017-02-27T15:48:54.629+0100 [initandlisten] exception in initAndListen: 10309 Unable to create/open lock file: /var/lib/mongodb/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating
2017-02-27T15:48:54.629+0100 [initandlisten] dbexit:

so, it looks like the upgrade procedure does not clear .lock files....

[Javier Castillo (javcasalc)]

Another note:

on newton, mongodb was running under UID 105. But on ocata, mongodb is running under UID 42432. So this bug might be related with uid/perms.

NEWTON:
root@control01:~# ls -l /var/lib/docker/volumes/mongodb/_data/mongod.lock
-rwxr-xr-x 1 _apt nogroup 2 Feb 27 12:39 /var/lib/docker/volumes/mongodb/_data/mongod.loc

OCATA:
root@control01:~# ls -lt /var/lib/docker/volumes/mongodb/_data/
total 1294352
-rwxr-xr-x 1 42432 nogroup 0 Feb 27 16:30 mongod.lock

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/443160
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=44e8419a5fc8053ce492451324f16367f462d3ac
Submitter: Jenkins
Branch: master

commit 44e8419a5fc8053ce492451324f16367f462d3ac
Author: Javier Castillo Alcíbar <email address hidden>
Date: Wed Mar 8 15:47:20 2017 +0100

    Fix dbPath typo in templates/mongodb.json.j2

    During mongodb upgrade process "/var/lib/mongodb" should have
    correct permissions, but a typo present in config.json was giving
    permissions to "/var/lib/kolla/mongodb"

    Change-Id: I916e7d831531131b2f141a2f7ac712d4a82f7e86
    Closes-Bug: #1668303

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/443180
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=a24acd730c4e0c073130d517f05b16c301c0bf80
Submitter: Jenkins
Branch: stable/ocata

commit a24acd730c4e0c073130d517f05b16c301c0bf80
Author: Javier Castillo Alcíbar <email address hidden>
Date: Wed Mar 8 15:47:20 2017 +0100

    Fix dbPath typo in templates/mongodb.json.j2

    During mongodb upgrade process "/var/lib/mongodb" should have
    correct permissions, but a typo present in config.json was giving
    permissions to "/var/lib/kolla/mongodb"

    Change-Id: I916e7d831531131b2f141a2f7ac712d4a82f7e86
    Closes-Bug: #1668303

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 4.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 5.0.0.0b2 development milestone.