Ceilometer Mongo DB backend cannot failover

[Tatyanka]

{"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}

Steps to reproduce:
1. Ubuntu with nova
3 controllers with mongo + ceiloemeter
1 compute
2. Deploy
3. Run ostf
4. ssh on primary controller and drop traffic to the mongo port
iptables -I OUTPUT 1 -p tcp --dport 27017 -j DROP
iptables -I INPUT 1 -p tcp --dport 27017 -j DROP

5. Run ostf
6. ssh on other controller and request alarm-list

Actual result:
OSTF tests are failed. alarm list failed. Seems we do not monitor mongo at all

Expected:
mongo runs under crm - so in case of failover - user data will be available

INFO: the same result if I kill mongo pid , but it works fine in case if node was shut down

[Matthew Mosesohn (raytrac3r)]

This needs to be converted to a blueprint for next release.

[Bogdan Dobrelya (bogdando)]

could be related https://bugs.launchpad.net/mos/+bug/1380800

[Bogdan Dobrelya (bogdando)]

Matthew, which excatly blueprint do you mean? The issue looks like not a feature request or an improvement

[Ivan Berezovskiy (iberezovskiy)]

This patch https://review.fuel-infra.org/#/c/2168/ should fix this problem.

I have verified it on ISO:

{"build_id": "2015-02-01_22-55-01", "ostf_sha": "c9100263140008abfcc2704732e98fbdfd644068", "build_number": "92", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-02-01_22-55-01", "ostf_sha": "c9100263140008abfcc2704732e98fbdfd644068", "build_number": "92", "api": "1.0", "nailgun_sha": "c0932eb5c2aa7fd1e13a999cb1b4bf5aff101c3b", "production": "docker", "python-fuelclient_sha": "2ea7b3e91c1d2ff85110bf5abb161a6f4e537358", "astute_sha": "ed5270bf9c6c1234797e00bd7d4dd3213253a413", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "", "fuellib_sha": "c5e4a0410ba66f9e9911f62b3b71c0b9c29aed6e"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "c0932eb5c2aa7fd1e13a999cb1b4bf5aff101c3b", "production": "docker", "python-fuelclient_sha": "2ea7b3e91c1d2ff85110bf5abb161a6f4e537358", "astute_sha": "ed5270bf9c6c1234797e00bd7d4dd3213253a413", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "", "fuellib_sha": "c5e4a0410ba66f9e9911f62b3b71c0b9c29aed6e"}

[Anastasia Palkina (apalkina)]

Verified on ISO #202

"build_id": "2015-03-16_22-54-44", "ostf_sha": "e86c961ceacfa5a8398b6cbda7b70a5f06afb476", "build_number": "202", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-16_22-54-44", "ostf_sha": "e86c961ceacfa5a8398b6cbda7b70a5f06afb476", "build_number": "202", "api": "1.0", "nailgun_sha": "874df0d06e32f14db77746cfeb2dd74d4a6e528c", "production": "docker", "python-fuelclient_sha": "2509c9b72cdcdbe46c141685a99b03cd934803be", "astute_sha": "93e427ac49109fa3fd8b0e1d0bb3d14092be2e8c", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "608b72a6f79a719cf01c35a19d0091fe20c8288a", "fuellib_sha": "924d73ae4766646e1c3a44d7b59c4120985e45f0"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "874df0d06e32f14db77746cfeb2dd74d4a6e528c", "production": "docker", "python-fuelclient_sha": "2509c9b72cdcdbe46c141685a99b03cd934803be", "astute_sha": "93e427ac49109fa3fd8b0e1d0bb3d14092be2e8c", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "608b72a6f79a719cf01c35a19d0091fe20c8288a", "fuellib_sha": "924d73ae4766646e1c3a44d7b59c4120985e45f0"

After successful deployment I killed mongodb process on primary controller. Started OSTF tests. It was also successful.