ceilometer is not installed. ceilometer-dbsync returned 1

[Vadim Rovachev]

{"build_id": "2014-04-29_17-00-21", "mirantis": "yes", "build_number": "160", "ostf_sha": "134765fcb5a07dce0cd1bb399b2290c988c3c63b", "nailgun_sha": "4f6208c617f4931314812091de1b64dfd18f9073", "production": "prod", "api": "1.0", "fuelmain_sha": "99c0830b8c6f679c36351e41d8555504c5660c2b", "astute_sha": "3cffebde1e5452f5dbf8f744c6525fc36c7afbf3", "release": "5.0", "fuellib_sha": "68b11620e7b0d42eafde375ef4992cadc8b0d10e"}

Step to reproduce:
1. install fuel master.
2. create env with parameters:
   virt_type=kvm
   config_mode=multinode
   release_name=Icehouse on CentOS 6.5
   net_provider=nova_network
   debug=true
   auto_assign_floating_ip=false
   nova_quota=true
   settings:{"volumes_lvm": false, "volumes_ceph": true, "images_ceph": true, "ephemeral_ceph": false,
    "objects_ceph": true, "osd_pool_size": 1, "sahara": true, "murano": false, "ceilometer": true}
   controller_role(kvm)=controller mongo
   compute_role(hardware)=compute ceph-osd
3. deploy env.
Expected result:
   Success. Deployment of environment '2' is done
Actual result:
   Error. Deployment has failed. Check these nodes:'NODE_controller_1'

-----------------------------------------------------------
controller console:

[root@node-1 ~]# ceilometer meter-list
Error communicating with http://172.18.92.84:8777 [Errno 111] Connection refused

[root@node-1 ~]# mongo
MongoDB shell version: 2.4.6
connecting to: test
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
 http://docs.mongodb.org/
Questions? Try the support group
 http://groups.google.com/group/mongodb-user
> show databases;
admin (empty)
ceilometer (empty)
local 0.078125GB
> exit;

-----------------------------------------------------------
ceilometer log(/var/log/ceilometer/api.log):
http://paste.openstack.org/show/77900/
-----------------------------------------------------------

[Dmitry Ilyin (idv1985)]

I reproduced it.

On the first run on 'primary-mongo' role package was installed, service have been started and firewall rule was added:

(/Stage[main]/Mongodb::Server::Install/Package[mongodb_server]/ensure) created
2014-04-30T14:14:01.737951+00:00 notice: (/Stage[main]/Mongodb::Server::Service/Service[mongodb]/ensure) ensure changed 'stopped' to 'running'
2014-04-30T14:14:01.932865+00:00 notice: (/Stage[main]/Mongodb::Firewall/Firewall[120 mongodb]/ensure) created

But then creation of ceilometer database failed:

2014-04-30T14:14:34.133064+00:00 err: (/Stage[main]/Openstack::Mongo_primary/Mongodb::Db[ceilometer]/Mongodb_database[ceilometer]) Could not evaluate: Execution of '/usr/bin/mongo --quiet --eval db.getMon
2014-04-30T14:14:34.133480+00:00 err: (/Stage[main]/Openstack::Mongo_primary/Mongodb::Db[ceilometer]/Mongodb_database[ceilometer]) exception: connect failed

Everything else mongo related failed too and Nailgun started the second try to install primary mongo role

2014-04-30T14:14:48.454325+00:00 notice: ***** Beginning deployment of node node-4 with role primary-mongo *****
2014-04-30T14:14:48.454719+00:00 notice: (/Stage[zero]/Begin_deployment/Notify[***** Beginning deployment of node node-4 with role primary-mongo *****]/message) defined 'message' as '***** Beginning deplo
2014-04-30T14:14:50.811954+00:00 notice: MongoDB params: 127.0.0.1192.168.0.1
2014-04-30T14:14:50.812272+00:00 notice: (/Stage[main]/Openstack::Mongo_primary/Notify[MongoDB params: 127.0.0.1192.168.0.1]/message) defined 'message' as 'MongoDB params: 127.0.0.1192.168.0.1'
2014-04-30T14:14:50.907577+00:00 notice: Installing package mongodb from 2.4.6-1.mira1 to installed
2014-04-30T14:14:51.247689+00:00 notice: (/Stage[main]/Mongodb::Client/Package[mongodb_client]/ensure) ensure changed '2.4.6-1.mira1' to 'installed'
2014-04-30T14:14:51.461579+00:00 notice: mongodb configuring databases
2014-04-30T14:14:51.462026+00:00 notice: (/Stage[main]/Openstack::Mongo_primary/Notify[mongodb configuring databases]/message) defined 'message' as 'mongodb configuring databases'
2014-04-30T14:14:51.776395+00:00 notice: (/Stage[main]/Openstack::Mongo_primary/Mongodb::Db[ceilometer]/Mongodb_database[ceilometer]/ensure) created
2014-04-30T14:14:52.036495+00:00 notice: (/Stage[main]/Openstack::Mongo_primary/Mongodb::Db[ceilometer]/Mongodb_user[ceilometer]/ensure) created
2014-04-30T14:14:52.291130+00:00 notice: (/Stage[main]/Openstack::Mongo_primary/Mongodb::Db[admin]/Mongodb_database[admin]/ensure) created
2014-04-30T14:14:52.540861+00:00 notice: (/Stage[main]/Openstack::Mongo_primary/Mongodb::Db[admin]/Mongodb_user[admin]/ensure) created
2014-04-30T14:14:52.548255+00:00 notice: mongodb primary finished

and everything went fine.

Then controller role was successfuly deployed successfully too and the entire environment is operational.

Why did mongo client failed to connect to the mongo service on the first deployment run? Maybe the was not enough time for mongo to warm up?

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/91409

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/91409
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=8981cd8006749400a2c953e2e7d7d4c0a94cae50
Submitter: Jenkins
Branch: master

commit 8981cd8006749400a2c953e2e7d7d4c0a94cae50
Author: Dmitry Ilyin <email address hidden>
Date: Mon May 5 18:59:01 2014 +0400

    Add mongodb wait_for_ready exec

    Change-Id: I9c6d72439cdb879c7bca56223360dc08f89c8f4f
    Closes-Bug: 1314514

[Andrey Sledzinskiy (asledzinskiy)]

verified on {"build_id": "2014-05-08_01-10-31", "mirantis": "yes", "build_number": "188", "ostf_sha": "fe718434f88f2ab167779770828a195f06eb29f8", "nailgun_sha": "82b9d42a7a5e9aa1caf6b2779c45ca045cad0ad2", "production": "docker", "api": "1.0", "fuelmain_sha": "97d7f6d5461db3afc27f58160cf9f6985230d255", "astute_sha": "9c83d3ecec69df03cd94620e2df92249ba4ec786", "release": "5.0", "fuellib_sha": "fd31d9a8f85136347b60377df00df6728eda14ca"}

[Leontii Istomin (listomin)]

Hello! I've reproduced this issue on 6.1 build:

[root@fuel ~]# fuel --fuel-version
api: '1.0'
astute_sha: d81ff53c2f467151ecde120d3a4d284e3b5b3dfc
auth_required: true
build_id: 2015-02-23_22-54-44
build_number: '140'
feature_groups:
- mirantis
fuellib_sha: 8b79d47ef41bff293210d2a7b1bb02843f70948d
fuelmain_sha: b975019fabdb429c1869047df18dd792d2163ecc
nailgun_sha: 3616ae9df4ac3e088157bb94f73743a521f76f1a
ostf_sha: 1a0b2c6618fac098473c2ed5a9af11d3a886a3bb
production: docker
python-fuelclient_sha: 5657dbf06fddb74adb61e9668eb579a1c57d8af8
release: '6.1'
release_versions:
  2014.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: d81ff53c2f467151ecde120d3a4d284e3b5b3dfc
      build_id: 2015-02-23_22-54-44
      build_number: '140'
      feature_groups:
      - mirantis
      fuellib_sha: 8b79d47ef41bff293210d2a7b1bb02843f70948d
      fuelmain_sha: b975019fabdb429c1869047df18dd792d2163ecc
      nailgun_sha: 3616ae9df4ac3e088157bb94f73743a521f76f1a
      ostf_sha: 1a0b2c6618fac098473c2ed5a9af11d3a886a3bb
      production: docker
      python-fuelclient_sha: 5657dbf06fddb74adb61e9668eb579a1c57d8af8
      release: '6.1'

Baremetal, Centos, HA, Neutron-gre,Ceilometer,Ceph-all, Debug, nova-quotas, 6.1_140
Controllers:3 Computes:47

Deployment has been failed.

From astute log:
[424] Task '{"priority"=>1100, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/openstack-controller/openstack-controller.pp", "timeout"=>3600, "cwd"=>"/"}}' on node 1 valid, but failed

From puppet log of node-1 (controller node):
2015-02-25 08:00:35 ERR
 (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]) Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0]

If I try execute this command:
[root@node-1 ~]# ceilometer-dbsync --debug --config-file=/etc/ceilometer/ceilometer.conf
2015-02-25 09:48:25.283 23642 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.metering.storage' get_connection /usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py:96
2015-02-25 09:48:25.360 23642 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('192.168.0.3', 27017), ('192.168.0.46', 27017), ('192.168.0.51', 27017)]
2015-02-25 09:48:25.365 23642 CRITICAL ceilometer [-] ConfigurationError: No suitable hosts found
2015-02-25 09:48:25.365 23642 TRACE ceilometer Traceback (most recent call last):
2015-02-25 09:48:25.365 23642 TRACE ceilometer File "/usr/bin/ceilometer-dbsync", line 10, in <module>
2015-02-25 09:48:25.365 23642 TRACE ceilometer sys.exit(dbsync())
2015-02-25 09:48:25.365 23642 TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/cmd/storage.py", line 32, in dbsync
2015-02-25 09:48:25.365 23642 TRACE ceilometer storage.get_connection_from_config(cfg.CONF, 'metering').upgrade()
2015-02-25 09:48:25.365 23642 TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py", line 86, in get_connection_from_config
2015-02-25 09:48:25.365 23642 TRACE ceilometer return get_connection(url, namespace)
2015-02-25 09:48:25.3...

[Leontii Istomin (listomin)]

DG is attached

[Leontii Istomin (listomin)]

6.1 has been failed because https://bugs.launchpad.net/fuel/+bug/1421283 has been occurred.
Fix already commited https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b2b6e0c47669fe5c770eec20a4b254aa543a0451