[ostf] Use proxy on primary controller for keystone client

[Artem Panchenko]

In case of network connectivity problems on some controller other controller nodes in HA cluster mark it as offline, but if admin network works fine there, OSTF still uses proxy started on that node and returns 500 error:

AuthorizationFailure: Authorization Failed: An unexpected error prevented the server from fulfilling your request. (1047, 'Unknown command') (HTTP 500)

This issue can be reproduced in the following way:

1. Deploy HA environment with neutron
2. Login on controller which has the smallest id
3. Shut down management interface (emulate network connectivity failure e.g.: ovs-vsctl del-port br-mgmt br-mgmt--br-eth0)
4. Run health checks. All checks will fail with authorization failure

I propose to add an additional step there:

https://github.com/stackforge/fuel-ostf/blob/stable/5.0/fuel_health/config.py#L639

if len(self.compute.online_controllers) > 1 then check out which controller is primary and use it as proxy server. IMHO a method which should detect an id of primary controller can execute 'pcs status' command on each controller node and return ID of node, which has partition with quorum on most of controllers.

[Dima Shulyak (dshulyak)]

I strongly disagree with provided approach..
Ostf should not go to each node and execute pcs status or any other shell command to find out primary controller.

If we need this kind of functionality lets register blueprint for 6.0, where we will provide a way for node
to report this kind of data to nailgun. And ostf will fetch this data from nailgun api.

[Dima Shulyak (dshulyak)]

Artem, what if we will use another method for detecting working controller,
we can try to authorize on each of those controllers, and stick with one that is working

Will it cover your case?

[Artem Panchenko (apanchenko-8)]

Dima,

yeah, I believe it will be enough to avoid the issue I reported. But I'm not sure

[Mike Scherbakov (mihgen)]

Folks,
it seems to me that this issues blocks proper automated testing of HA failover. Because of this, I think it has be higher priority issue than it is now.

[Dennis Dmitriev (ddmitriev)]

Confirmed on CI for 5.1.1: http://jenkins-product.srt.mirantis.net:8080/view/5.1_swarm/job/5.1_fuelmain.system_test.ubuntu.ha_neutron_destructive/48/ , system test 'ha_disconnect_controllers'

{"build_id": "2014-12-03_01-07-36", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "48", "auth_required": true, "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-12-03_01-07-36", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "48", "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"}}}, "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/145016

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/145016
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=0b75defeb24e301e412ef28220c9b0ff50c5be16
Submitter: Jenkins
Branch: master

commit 0b75defeb24e301e412ef28220c9b0ff50c5be16
Author: Tatyana Leontovich <email address hidden>
Date: Mon Jan 5 17:32:58 2015 +0200

    Add verification if proxy works

    In case if controller is online, but connectivity
    on it is broken we receive false positive
    message like "Can not set proxy". To fix this:
    * Import of keystoneclient was added into config.py
    * find_proxy method was added
    * set proxy changed

    Change-Id: I77d3c0ec12bc071a7e490ddb3ea24cd2f7fea543
    Closes-Bug: #1355897

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/161233

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/161251

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/161233
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=e15d4ebc7da62cd6793e1fcd7ce9e8f2504f788e
Submitter: Jenkins
Branch: stable/6.0

commit e15d4ebc7da62cd6793e1fcd7ce9e8f2504f788e
Author: Tatyana Leontovich <email address hidden>
Date: Wed Mar 4 16:30:04 2015 +0200

    6.0 Add verification if proxy works

    In case if controller is online, but connectivity
    on it is broken we receive false positive
    message like "Can not set proxy". To fix this:
    * Import of keystoneclient was added into config.py
    * find_proxy method was added
    * set proxy changed

    Change-Id: I77d3c0ec12bc071a7e490ddb3ea24cd2f7fea543
    Closes-Bug: #1355897

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/161251
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=9d4392fb97252c84dee7d29ee33c706d78b5bbef
Submitter: Jenkins
Branch: stable/5.1

commit 9d4392fb97252c84dee7d29ee33c706d78b5bbef
Author: Tatyana Leontovich <email address hidden>
Date: Wed Mar 4 16:30:04 2015 +0200

    5.1 Add verification if proxy works

    In case if controller is online, but connectivity
    on it is broken we receive false positive
    message like "Can not set proxy". To fix this:
    * Import of keystoneclient was added into config.py
    * find_proxy method was added
    * set proxy changed

    Change-Id: I77d3c0ec12bc071a7e490ddb3ea24cd2f7fea543
    Closes-Bug: #1355897

[Tatyanka (tatyana-leontovich)]

{"build_id": "2015-03-26_09-08-29", "ostf_sha": "a4cf5f218c6aea98105b10c97a4aed8115c15867", "build_number": "231", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-26_09-08-29", "ostf_sha": "a4cf5f218c6aea98105b10c97a4aed8115c15867", "build_number": "231", "api": "1.0", "nailgun_sha": "7f0e0af1f54db840230745ee4f7aec6824dac9b9", "production": "docker", "python-fuelclient_sha": "e5e8389d8d481561a4d7107a99daae07c6ec5177", "astute_sha": "631f96d5a09cc48bfbddcbf056b946c8a80438f0", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "320b5f46fc1b2798f9e86ed7df51d3bda1686c10", "fuellib_sha": "345a98b34dd0cd450a45d405ac47a6a9fa48b6d8"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "7f0e0af1f54db840230745ee4f7aec6824dac9b9", "production": "docker", "python-fuelclient_sha": "e5e8389d8d481561a4d7107a99daae07c6ec5177", "astute_sha": "631f96d5a09cc48bfbddcbf056b946c8a80438f0", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "320b5f46fc1b2798f9e86ed7df51d3bda1686c10", "fuellib_sha": "345a98b34dd0cd450a45d405ac47a6a9fa48b6d8"} verified for 6.1

for other version move to fix committed according to comments above