Ceilometer does not run

[Julia Aranovich]

{"build_id": "2014-05-16_01-10-31", "mirantis": "yes", "build_number": "208", "ostf_sha": "353f918197ec53a00127fd28b9151f248a2a2d30", "nailgun_sha": "89e7224f0b87284f75bf02b956b640768fabe352", "production": "docker", "api": "1.0", "fuelmain_sha": "861f4410bd07ecf40c1f87785dd0c86de387f2cf", "astute_sha": "c0418e1739cceb864fd2c548a7c6b9ad6a46ed86", "release": "5.0", "fuellib_sha": "68ef90fdc9fe5a3978f756a96a3fa29d5f1a2929"}

Environment settings: Ubuntu, Multi-node (simple), QEMU, Nova-network (VLAN), networks tagging - yes.
Nodes: 1 controller, 1 compute, 1 cinder, 1 mongo.
Additional services: Sahara, Ceilometer.

OSTF tests failed for the deployed env. Ceilometer didn't start.
Seems like it can't sync with data base and can't launch it's services: http://paste.openstack.org/show/albLMVMblktzMxTtdE4r/

[Bogdan Dobrelya (bogdando)]

Looks like the root cause is
2014-05-16T11:34:19.731655 node-6 ./node-6.domain.tld/puppet-apply.log:2014-05-16T11:34:19.731655+00:00 debug: (/Stage[main]/Mysql::Password/Exec[se
t_mysql_rootpw]/unless) error: 'Access denied for user 'root'@'localhost' (using password: YES)'

caused the db sync failure later
2014-05-16T11:36:01.545305 node-6 ./node-6.domain.tld/puppet-apply.log:2014-05-16T11:36:01.545305+00:00 err: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]) Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0]

[Bogdan Dobrelya (bogdando)]

Sorry, Ceilometer has nothing common to the MySQL. Wrong RC candidate... :)

[Alexei Kornienko (alexei-kornienko)]

Seems that MongoDB is not running when ceilometer is trying to start. There is a patch in upstream that would partially fix it (try to reconnect several times). It may be easier for you to fix it by applying a small timeout before ceilometer start or ensuring that mongodb is running in some other way

[Dina Belova (dbelova)]

https://review.openstack.org/#/c/86609/ - (try to reconnect several times) - here is this patch

[Dina Belova (dbelova)]

But this really looks like some deployment error - so the steps here are the following:

1) become sure that Mongo is up and configured *before* ceilometer is about to start
2) if it's not, do this else 3
3) commit quick fix - like proposed by Alexei

[Ivan Berezovskiy (iberezovskiy)]

I deployed such environment today and deploy was success, but in logs I could see erros like:

(/Stage[main]/Glance::Registry/Exec[glance-manage db_sync]) Failed to call refresh: glance-manage db_sync || (glance-manage version_control 0 && glance-manage db_sync) returned 2 instead of one of [0]

The same fail with db-sync was with cinder and heat too, but higher in logs these commands was successfully run and glance, heat and cinder worked well. Ceilometer worked well in my deploy too and without any errors.

As it was discussed with Bogdan, this problem with ceilometer db-sync might happen because of failed mysql dependency. Because in that env was a problem with setup root password in mysql.

[Mike Scherbakov (mihgen)]

If Ceilometer depends on MySQL, then it is an issue, and this dependency should be removed - we use MongoDB for Ceilometer.

[Dina Belova (dbelova)]

Ceilometer has the SQLalchemy and SQL-related libs as main dependency (due to the fact that SQL-based testing, etc. should be evadable by default,...), and Mongo deps are run-time ones - that you need to install separately. So yes, Ceilometer has SQLalchemy deps by default - https://github.com/openstack/ceilometer/blob/master/requirements.txt#L33 and Ceilo folks won't remove it from there as sql backends are the priority from the TC.

TL:DR

SQL stuff should be always installed, Mongo should be installed separately if needed

[Mike Scherbakov (mihgen)]

I'm talking about puppet resources dependencies, not about python.

[Bogdan Dobrelya (bogdando)]

Cannot reproduce at #212 with
a) Nodes: 1 controller, 1 compute, 1 mongo (cinder node was omitted):
- Mongo node was deployed first - OK
- Controller node was deployed second w/o issues
- Compute node was 3rd as well

[Bogdan Dobrelya (bogdando)]

2nd and 3rd Ceilometer OSTF tests for case (a) was OK. (1st one was skipped due to now cinder node was deployed)

[Bogdan Dobrelya (bogdando)]

Note: case (a) was deployed w/o additional services: Sahara
Now verifying the case (b) - the reported one:
Environment settings: Ubuntu, Multi-node (simple), *KVM*, Nova-network (VLAN), networks tagging - yes.
Nodes: 1 controller, 1 compute, 1 cinder, 1 mongo.
Additional services: Sahara, Ceilometer.

[Bogdan Dobrelya (bogdando)]

Cannot reproduce for the case (b) as well. Setting to invalid

[Bogdan Dobrelya (bogdando)]

Looks like the source of this issue was unconfigured mysql password https://bugs.launchpad.net/fuel/+bug/1320842