Mongo failed to choose primary controller when old primary controller was down.

[Vadim Rovachev]

{"build_id": "2014-11-20_21-01-00", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "28", "auth_required": true, "api": "1.0", "nailgun_sha": "7580f6341a726c2019f880ae23ff3f1c581fd850", "production": "docker", "fuelmain_sha": "eac9e2704424d1cb3f183c9f74567fd42a1fa6f3", "astute_sha": "51087c92a50be982071a074ff2bea01f1a5ddb76", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-11-20_21-01-00", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "28", "api": "1.0", "nailgun_sha": "7580f6341a726c2019f880ae23ff3f1c581fd850", "production": "docker", "fuelmain_sha": "eac9e2704424d1cb3f183c9f74567fd42a1fa6f3", "astute_sha": "51087c92a50be982071a074ff2bea01f1a5ddb76", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "b3d9f0e203f2f0faf3763e871a8dc31570777fed"}}}, "fuellib_sha": "b3d9f0e203f2f0faf3763e871a8dc31570777fed"}

Steps to reproduce:
1. Create and deploy env:
HA mode
CentOS
neutron GRE
nodes: 3 controller,mongo + 1 compute,cepth_osd

2. After deploy hard shutdown drimary controller

last lines on /var/log/ceilometer/agent-notification.log:
http://paste.openstack.org/show/135996/

cli request to ceilometer aftr shutdown primary controller:
http://paste.openstack.org/show/135997/

[Bogdan Dobrelya (bogdando)]

Please provide the diagnostic logs snapshot

[Vadim Rovachev (vrovachev)]

> rs.status()
{
 "startupStatus" : 3,
 "info" : "run rs.initiate(...) if not yet done for the set",
 "ok" : 0,
 "errmsg" : "can't get local.system.replset config from self or any seed (EMPTYCONFIG)"
}
>

[Ivan Berezovskiy (iberezovskiy)]

Trying to reproduce

[Vladimir Kuklin (vkuklin)]

Ivan, do you have any update on bug status?

[Ivan Berezovskiy (iberezovskiy)]

Can't reproduce on env with qemu. ISO:

{"build_id": "2014-11-26_00-05-06", "ostf_sha": "a35f516f1606b0d03d51ff63bfe3fbe23de4b622", "build_number": "8", "auth_required": true, "api": "1.0", "nailgun_sha": "cbe7b96943d43397dc608a2f6c9dc1af14dd9a48", "production": "docker", "fuelmain_sha": "7db74b9f80180bf3936db1edc4aebfae310d024a", "astute_sha": "c15623d05ccdf7ac10873e7a90df954de8726280", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-26_00-05-06", "ostf_sha": "a35f516f1606b0d03d51ff63bfe3fbe23de4b622", "build_number": "8", "api": "1.0", "nailgun_sha": "cbe7b96943d43397dc608a2f6c9dc1af14dd9a48", "production": "docker", "fuelmain_sha": "7db74b9f80180bf3936db1edc4aebfae310d024a", "astute_sha": "c15623d05ccdf7ac10873e7a90df954de8726280", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "8c7eec6225184e0391569b2b5371196ab3e3fa19"}}}, "fuellib_sha": "8c7eec6225184e0391569b2b5371196ab3e3fa19"}

[Ivan Berezovskiy (iberezovskiy)]

Can't reproduce on env with kvm. ISO:

{"build_id": "2014-11-24_16-40-53", "ostf_sha": "a35f516f1606b0d03d51ff63bfe3fbe23de4b622", "build_number": "3", "auth_required": true, "api": "1.0", "nailgun_sha": "603a8d438dc7a3cf6286eb9f16deb8137f47d703", "production": "docker", "fuelmain_sha": "45b21f7bdb061b59b80f8d126d9a6f6e50505a0d", "astute_sha": "c15623d05ccdf7ac10873e7a90df954de8726280", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-24_16-40-53", "ostf_sha": "a35f516f1606b0d03d51ff63bfe3fbe23de4b622", "build_number": "3", "api": "1.0", "nailgun_sha": "603a8d438dc7a3cf6286eb9f16deb8137f47d703", "production": "docker", "fuelmain_sha": "45b21f7bdb061b59b80f8d126d9a6f6e50505a0d", "astute_sha": "c15623d05ccdf7ac10873e7a90df954de8726280", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "893883f7fa8ffc5dde975b6806e538a11969a15b"}}}, "fuellib_sha": "893883f7fa8ffc5dde975b6806e538a11969a15b"}

[Ivan Berezovskiy (iberezovskiy)]

Also can't reproduce on 5.1.1 ISO:

{"build_id": "2014-11-26_21-01-00", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "42", "auth_required": true, "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "51e66db7750e9c856ba128f35cfb6724895bf479", "astute_sha": "dade74af41d4972fe05a1c16ae1db2a2e60c6715", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-11-26_21-01-00", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "42", "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "51e66db7750e9c856ba128f35cfb6724895bf479", "astute_sha": "dade74af41d4972fe05a1c16ae1db2a2e60c6715", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "db976a45be4f6606e7c439cbec237fe8482f4f80"}}}, "fuellib_sha": "db976a45be4f6606e7c439cbec237fe8482f4f80"}

[Bogdan Dobrelya (bogdando)]

This bug remained incomplete for more than a month. Setting its state to invalid.