Ceilometer Mongo DB backend cannot failover

Bug #1371799 reported by Tatyanka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Igor Degtiarov
5.0.x
Fix Committed
High
Unassigned
5.1.x
Fix Committed
High
Unassigned
6.0.x
Fix Committed
High
Unassigned

Bug Description

{"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}

Steps to reproduce:
1. Ubuntu with nova
3 controllers with mongo + ceiloemeter
1 compute
2. Deploy
3. Run ostf
4. ssh on primary controller and drop traffic to the mongo port
iptables -I OUTPUT 1 -p tcp --dport 27017 -j DROP
iptables -I INPUT 1 -p tcp --dport 27017 -j DROP

5. Run ostf
6. ssh on other controller and request alarm-list

Actual result:
OSTF tests are failed. alarm list failed. Seems we do not monitor mongo at all

Expected:
mongo runs under crm - so in case of failover - user data will be available

INFO: the same result if I kill mongo pid , but it works fine in case if node was shut down

Tags: ceilometer ha
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
summary: - Ceilometer randomply fails in ha if it is mongo was blocked on primary
+ Ceilometer randomply fails in ha if mongo was blocked on primary
controller
Changed in fuel:
status: New → Confirmed
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Sergii Golovatiuk (sgolovatiuk)
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote : Re: Ceilometer randomply fails in ha if mongo was blocked on primary controller

This needs to be converted to a blueprint for next release.

Changed in fuel:
milestone: 6.0 → 6.1
summary: - Ceilometer randomply fails in ha if mongo was blocked on primary
+ Ceilometer randomly fails in HA if mongo was blocked on primary
controller
Changed in fuel:
assignee: Sergii Golovatiuk (sgolovatiuk) → nobody
assignee: nobody → Fuel Library Team (fuel-library)
Revision history for this message
Bogdan Dobrelya (bogdando) wrote : Re: Ceilometer randomly fails in HA if mongo was blocked on primary controller
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Matthew, which excatly blueprint do you mean? The issue looks like not a feature request or an improvement

summary: - Ceilometer randomly fails in HA if mongo was blocked on primary
- controller
+ Ceilometer Mongo DB backend cannot failover
Revision history for this message
Ivan Berezovskiy (iberezovskiy) wrote :

This patch https://review.fuel-infra.org/#/c/2168/ should fix this problem.

I have verified it on ISO:

{"build_id": "2015-02-01_22-55-01", "ostf_sha": "c9100263140008abfcc2704732e98fbdfd644068", "build_number": "92", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-02-01_22-55-01", "ostf_sha": "c9100263140008abfcc2704732e98fbdfd644068", "build_number": "92", "api": "1.0", "nailgun_sha": "c0932eb5c2aa7fd1e13a999cb1b4bf5aff101c3b", "production": "docker", "python-fuelclient_sha": "2ea7b3e91c1d2ff85110bf5abb161a6f4e537358", "astute_sha": "ed5270bf9c6c1234797e00bd7d4dd3213253a413", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "", "fuellib_sha": "c5e4a0410ba66f9e9911f62b3b71c0b9c29aed6e"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "c0932eb5c2aa7fd1e13a999cb1b4bf5aff101c3b", "production": "docker", "python-fuelclient_sha": "2ea7b3e91c1d2ff85110bf5abb161a6f4e537358", "astute_sha": "ed5270bf9c6c1234797e00bd7d4dd3213253a413", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "", "fuellib_sha": "c5e4a0410ba66f9e9911f62b3b71c0b9c29aed6e"}

Changed in fuel:
status: Confirmed → Fix Committed
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Igor Degtiarov (idegtiarov)
Revision history for this message
Anastasia Palkina (apalkina) wrote :

Verified on ISO #202

"build_id": "2015-03-16_22-54-44", "ostf_sha": "e86c961ceacfa5a8398b6cbda7b70a5f06afb476", "build_number": "202", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-16_22-54-44", "ostf_sha": "e86c961ceacfa5a8398b6cbda7b70a5f06afb476", "build_number": "202", "api": "1.0", "nailgun_sha": "874df0d06e32f14db77746cfeb2dd74d4a6e528c", "production": "docker", "python-fuelclient_sha": "2509c9b72cdcdbe46c141685a99b03cd934803be", "astute_sha": "93e427ac49109fa3fd8b0e1d0bb3d14092be2e8c", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "608b72a6f79a719cf01c35a19d0091fe20c8288a", "fuellib_sha": "924d73ae4766646e1c3a44d7b59c4120985e45f0"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "874df0d06e32f14db77746cfeb2dd74d4a6e528c", "production": "docker", "python-fuelclient_sha": "2509c9b72cdcdbe46c141685a99b03cd934803be", "astute_sha": "93e427ac49109fa3fd8b0e1d0bb3d14092be2e8c", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "608b72a6f79a719cf01c35a19d0091fe20c8288a", "fuellib_sha": "924d73ae4766646e1c3a44d7b59c4120985e45f0"

After successful deployment I killed mongodb process on primary controller. Started OSTF tests. It was also successful.

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.