Openstack operations periodically return 401 unaithorized error after controller simultaneous add/delete

[Andrey Sledzinskiy]

Openstack operations periodically return 401 unaithorized error after controller simultaneous add/delete

Steps:
1. Create cluster with Neutron GRE
            2. Add 3 node with controller role
            3. Add 1 node with compute
            3. Deploy cluster
            4. Remove one controller and add new controller
            5. Deploy changes
            6. Run OSTF

Actual result - openstack operations fails on different stages with 401, 500 from keystone
nova list
ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-859d6f43-1713-421e-a680-6395969b2172)

[Boris Bobrov (bbobrov)]

I was provided an env to debug on. Are you aware that time on your controllers is different? node-3 and node-4 have 1am and node-1 7am.

[Boris Bobrov (bbobrov)]

Asked Andrey to try to reproduce it, he will try to do it locally.

[Andrey Sledzinskiy (asledzinskiy)]

Issue wasn't reproduced locally and on CI
checked on
{

    "build_id": "2015-08-18_03-05-20",
    "build_number": "187",
    "release_versions":

{

    "2015.1.0-7.0":

{

    "VERSION":

{

    "build_id": "2015-08-18_03-05-20",
    "build_number": "187",
    "api": "1.0",
    "fuel-library_sha": "0062e69db17f8a63f85996039bdefa87aea498e1",
    "nailgun_sha": "4710801a2f4a6d61d652f8f1e64215d9dde37d2e",
    "feature_groups":

            [
                "mirantis"
            ],
            "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c",
            "openstack_version": "2015.1.0-7.0",
            "fuel-agent_sha": "57145b1d8804389304cd04322ba0fb3dc9d30327",
            "production": "docker",
            "python-fuelclient_sha": "4c74a60aa60c06c136d9197c7d09fa4f8c8e2863",
            "astute_sha": "e24ca066bf6160bc1e419aaa5d486cad1aaa937d",
            "fuel-ostf_sha": "17786b86b78e5b66d2b1c15500186648df10c63d",
            "release": "7.0",
            "fuelmain_sha": "c9dad194e82a60bf33060eae635fff867116a9ce"
        }
    }

},
"auth_required": true,
"api": "1.0",
"fuel-library_sha": "0062e69db17f8a63f85996039bdefa87aea498e1",
"nailgun_sha": "4710801a2f4a6d61d652f8f1e64215d9dde37d2e",
"feature_groups":

    [
        "mirantis"
    ],
    "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c",
    "openstack_version": "2015.1.0-7.0",
    "fuel-agent_sha": "57145b1d8804389304cd04322ba0fb3dc9d30327",
    "production": "docker",
    "python-fuelclient_sha": "4c74a60aa60c06c136d9197c7d09fa4f8c8e2863",
    "astute_sha": "e24ca066bf6160bc1e419aaa5d486cad1aaa937d",
    "fuel-ostf_sha": "17786b86b78e5b66d2b1c15500186648df10c63d",
    "release": "7.0",
    "fuelmain_sha": "c9dad194e82a60bf33060eae635fff867116a9ce"

}