Some openstack services doesn't work after primary controller shutdown when database is deployed on separate nodes

[Andrey Sledzinskiy]

Some openstack services doesn't work after primary controller shutdown when database is deployed on separate nodes

Steps:
1. Upload and install on master detach-database plugin
2. Create next cluster - Ubuntu, HA, Neutron Vlan, Cinder, detach-database plugin enabled, 3 controllers, 3 db node, 1 compute, 1 cinder
3. Deploy cluster
4. Run ostf
5. Shutdown primary controller
6. Wait for 15 minutes
7. Run ostf

Actual result - all nova, cinder related tests don't work because it's trying to connect to shut-down controller (10.109.2.5 - management ip of destroyed controller (node-10))

cinder --debug list
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://10.109.2.3:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
DEBUG:keystoneclient.session:RESP: [200] content-length: 336 vary: X-Auth-Token connection: close date: Mon, 27 Jul 2015 12:13:04 GMT content-type: application/json x-openstack-request-id: req-38a5adb8-fae2-477a-8965-60c3b7d98ad5
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.109.2.3:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://10.109.2.3:5000/v2.0/tokens
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://10.109.2.5:8776/v1/f9dfd95d65954e42a337b1cd395240fd/volumes/detail -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}1d8bdbd1be02c7579dddce8a8e204bfba0b5e3fb"
ERROR: Unable to establish connection to http://10.109.2.5:8776/v1/f9dfd95d65954e42a337b1cd395240fd/volumes/detail

Logs are attached

{

    "build_id": "2015-07-22_03-49-58",
    "build_number": "68",
    "release_versions":

{

    "2014.2.2-7.0":

{

    "VERSION":

{

    "build_id": "2015-07-22_03-49-58",
    "build_number": "68",
    "api": "1.0",
    "fuel-library_sha": "f0624481718913647c34bd65081e68e094a9119f",
    "nailgun_sha": "01368e907b34180268841a97657cc5e7e4db843f",
    "feature_groups":

            [
                "mirantis"
            ],
            "openstack_version": "2014.2.2-7.0",
            "fuel-agent_sha": "fd16b972dde3ffb1b1c69a29a54f8ccd1c36f47c",
            "production": "docker",
            "python-fuelclient_sha": "57713c96a30831d33609e6e812f8d4eb6054d04e",
            "astute_sha": "b1f37a988e097175cbbd14338286017b46b584c3",
            "fuel-ostf_sha": "bf94b765214e977df2ca2058cc8fb10bc073002b",
            "release": "7.0",
            "fuelmain_sha": "07bbcba2cb8ce8e81de8874b7957e0fe1664f455"
        }
    }

},
"auth_required": true,
"api": "1.0",
"fuel-library_sha": "f0624481718913647c34bd65081e68e094a9119f",
"nailgun_sha": "01368e907b34180268841a97657cc5e7e4db843f",
"feature_groups":

    [
        "mirantis"
    ],
    "openstack_version": "2014.2.2-7.0",
    "fuel-agent_sha": "fd16b972dde3ffb1b1c69a29a54f8ccd1c36f47c",
    "production": "docker",
    "python-fuelclient_sha": "57713c96a30831d33609e6e812f8d4eb6054d04e",
    "astute_sha": "b1f37a988e097175cbbd14338286017b46b584c3",
    "fuel-ostf_sha": "bf94b765214e977df2ca2058cc8fb10bc073002b",
    "release": "7.0",
    "fuelmain_sha": "07bbcba2cb8ce8e81de8874b7957e0fe1664f455"

}

[Vladimir Kuklin (vkuklin)]

Seems ARP address related

[Matthew Mosesohn (raytrac3r)]

i'm holding too many bugs, so passing back to fuel-library

[Michael Polenchuk (mpolenchuk)]

Not reproducible

[Andrey Sledzinskiy (asledzinskiy)]

It's not reproduced on
{

    "build_id": "2015-08-21_03-00-36",
    "build_number": "203",
    "release_versions":

{

    "2015.1.0-7.0":

{

    "VERSION":

{

    "build_id": "2015-08-21_03-00-36",
    "build_number": "203",
    "api": "1.0",
    "fuel-library_sha": "7aab6c2e53e5ee43333bce89ee406ae1db8c4063",
    "nailgun_sha": "1e5034274e92cda41ab6861585c396470fe08a0f",
    "feature_groups":

            [
                "mirantis"
            ],
            "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c",
            "openstack_version": "2015.1.0-7.0",
            "fuel-agent_sha": "f2df59bb3ff2b8a79a47be258421f603863a4bbc",
            "production": "docker",
            "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472",
            "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c",
            "fuel-ostf_sha": "16839cbf471b7142b04c0d2c2d94786bc486fefe",
            "release": "7.0",
            "fuelmain_sha": "c9dad194e82a60bf33060eae635fff867116a9ce"
        }
    }

},
"auth_required": true,
"api": "1.0",
"fuel-library_sha": "7aab6c2e53e5ee43333bce89ee406ae1db8c4063",
"nailgun_sha": "1e5034274e92cda41ab6861585c396470fe08a0f",
"feature_groups":

    [
        "mirantis"
    ],
    "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c",
    "openstack_version": "2015.1.0-7.0",
    "fuel-agent_sha": "f2df59bb3ff2b8a79a47be258421f603863a4bbc",
    "production": "docker",
    "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472",
    "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c",
    "fuel-ostf_sha": "16839cbf471b7142b04c0d2c2d94786bc486fefe",
    "release": "7.0",
    "fuelmain_sha": "c9dad194e82a60bf33060eae635fff867116a9ce"

}