Deployment with Ceilometer failed with error 'Unable to connect to mongodb server'

[Vitaly Gusev]

Bug reproduced here: https://product-ci.infra.mirantis.net/view/8.0_swarm/job/8.0.system_test.ubuntu.services_ha.ceilometer/66/console

Steps to reproduce:
1. Create env with parameters:
3 controller + mongo, 1 compute, 1 cinder; Cinder LVM; hypervisor KVM; Neutron VXLAN
2. Start deploy

Expected result:
Deployment finished successfully.

Actual result:
Deployment failed. In Astute log we can see these errors:
2015-12-06 23:35:09 ERR 2015-12-06T23:35:09.534+0000 Error: couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed at src/mongo/shell/mongo.js:148
2015-12-06 23:35:09 ERR Could not prefetch mongodb_replset provider 'mongodb': Execution of '/usr/bin/mongo --quiet --host 127.0.0.1:27017 --eval load('/root/.mongorc.js'); printjson(rs.conf())' returned 1: 2015-12-06T23:35:09.533+0000 warning: Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
2015-12-06 23:35:09 ERR (/Stage[main]/Mongodb::Server::Service/Mongodb_conn_validator[mongodb]/ensure) change from absent to present failed: Unable to connect to mongodb server! (127.0.0.1:27017)
https://paste.mirantis.net/show/1540/

In /var/log/mongod.log we can see this error:
<14>Dec 7 00:01:11 node-1 mongod.27017[15164]: invalid char in key file /etc/mongodb.key: '

Output of the /etc/mongodb.key file:
https://paste.mirantis.net/show/1541/

Looks like that root cause is in incorrect mongodb key.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "263"
  build_id: "263"
  fuel-nailgun_sha: "0e09dce510927f2cc490b898e5fe3f813bd791be"
  python-fuelclient_sha: "f033192b84263f0e699458a4274289a5198ae7e4"
  fuel-agent_sha: "660c6514caa8f5fcd482f1cc4008a6028243e009"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "48fd58676debcc85951db68df6d77c22daa55e52"
  fuel-library_sha: "ab7e51f345ffb7c256e0f61addcf86553d7c3867"
  fuel-ostf_sha: "23b7ae2a1a57de5a3e1861ffb7805394ca339cc2"
  fuel-mirror_sha: "6534117233a5bdc51d7d47361bc7d511e4b11e6f"
  fuelmenu_sha: "fcb15df4fd1a790b17dd78cf675c11c279040941"
  shotgun_sha: "a0bd06508067935f2ae9be2523ed0d1717b995ce"
  network-checker_sha: "a3534f8885246afb15609c54f91d3b23d599a5b1"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "26adf12c320936a97a9b0a84169a6e58c530e848"

[Timur Nurlygayanov (tnurlygayanov)]

Looks like it is invalid mongodb.key file

[Stanislaw Bogatkin (sbogatkin)]

Timur, you're right. It is related to OpenSSL problems, not to mongo ones. Mongo itself is okay, key is generated badly. I'll look how to avoid this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/254791

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/254791
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b7769459a0049df50aad254c8b7927ae5c1a0087
Submitter: Jenkins
Branch: master

commit b7769459a0049df50aad254c8b7927ae5c1a0087
Author: Stanislaw Bogatkin <email address hidden>
Date: Tue Dec 8 17:42:14 2015 +0300

    Declare RANDFILE explicitly

    Point to openssl where RANDFILE is to avoid cases when this file
    needed but not declared.

    Change-Id: I19620d9f31eadfb345812de81d50b0326067c143
    Closes-Bug: #1523449

[Vitaly Gusev (vgusev)]

Deployment was successfull in SWARM job https://product-ci.infra.mirantis.net/view/8.0_swarm/job/8.0.system_test.ubuntu.services_ha_one_controller/70/console

So, fix released

[Nastya Urlapova (aurlapova)]

@Vasiliy, could you attach full version, instead of link to job.

[Vitaly Gusev (vgusev)]

@Nastya, yes, of course

Verified on ISO
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "283"
  build_id: "283"
  fuel-nailgun_sha: "073beb9f02e103e8f78abe9425d4c9feb5fab6e4"
  python-fuelclient_sha: "f96659066e522e28b389de3cc685f6f2aacca3da"
  fuel-agent_sha: "2f18b7596bc7da79d2f28c34f42620b2090d8a35"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "f7df18ab6b2089649abcfa8cf667ffd6b332be1a"
  fuel-library_sha: "66fc43373039a23fb63c26e7919a60b3560e0db1"
  fuel-ostf_sha: "632730169e8c01afe7fd5d78a898f00d4646358b"
  fuel-mirror_sha: "31b9df814960ec69b644ca9b689dacec0c7e10a1"
  fuelmenu_sha: "680b720291ff577f4c058cee25f85e563c96312e"
  shotgun_sha: "a0bd06508067935f2ae9be2523ed0d1717b995ce"
  network-checker_sha: "a3534f8885246afb15609c54f91d3b23d599a5b1"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "1577a306c2c9e7bd12f28c0e16cf3652997da004"

[Nastya Urlapova (aurlapova)]

  @Vitaly, sorry for my mistake in comment #7. Thanks that you have attached full version :)