[OSTF] Need to provide more descriptive messages in HA suite in case if failover was happened on one of controller

[Tatyanka]

Steps to repsroduce:
1. Deploy Ha (Centos, ubuntu)
2. shot down one of the controller
3. Run ostf

Actual result:
Ha tests failed:
Check amount of tables in databases is the same on each node -
Time limit exceeded while waiting for detect mysql node to finish. Please refer to OpenStack logs for more details.

Check data replication over mysql
- Time limit exceeded while waiting for detect mysql node to finish. Please refer to OpenStack logs for more details.
But this tests fails according test data was not replicated (SSHExecCommandFailed: Command 'mysql -e 'DROP DATABASE ost1484'', exit status: 1, Error:
ERROR 1008 (HY000) at line 1: Can't drop database 'ost1484'; database doesn't exist)

Expected:
1. Tests may fails but descriptive message must show
2. for those tests where we ssh on controller and examine status on mysql on each controller - we should use here only online nodes

[Andrew Woodward (xarses)]

We should first determine if a node is online or not and then if mysql is up. If there is a quorum between the remaining nodes, we should warn that all nodes aren't online instead of a vague failure message

[Tatyanka (tatyana-leontovich)]

Thank you Andrew, absolutely agree, that's why we've created the issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/80551

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/80551
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=be64966dd549ba6bf7792ca8a1929b46c4d77c75
Submitter: Jenkins
Branch: master

commit be64966dd549ba6bf7792ca8a1929b46c4d77c75
Author: Tatyana Leontovich <email address hidden>
Date: Fri Mar 14 14:52:54 2014 +0000

    Add parsing of node status before ha tests

    If some node fails(shutdown) ha tests fail
    on step when we try ssh to ofline node
    Add nodes state parsing to verify HA recovery
    only on onlines nodes

    Change-Id: Ia89824316bc2228fdefdb4d0cb1cd965bb58145e
    Closes-bug: #1288882

[Anastasia Palkina (apalkina)]

Verified on ISO #33
"build_id": "2014-03-18_01-01-16",
"mirantis": "yes",
"build_number": "33",
"nailgun_sha": "a5f0a430e4c0f3315d8ea160d4a7a7c83c7d1917",
"ostf_sha": "31c8316d4d9762f280da9e360d3d42e88464aac1",
"fuelmain_sha": "d494e5cf2b8399899f00b5079eec1c5f70c0b66f",
"astute_sha": "a80acd1312be70676f0dd936814b27ce01fb013d",
"release": "5.0",
"fuellib_sha": "0f2765736a2e4af7265eeb936161f0a90cdc9a2e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/83092

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/83092
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=1096bbff82b64a1a70490afd825ca592dd198c48
Submitter: Jenkins
Branch: stable/4.1

commit 1096bbff82b64a1a70490afd825ca592dd198c48
Author: Tatyana Leontovich <email address hidden>
Date: Fri Mar 14 14:52:54 2014 +0000

    4.1.1 Add parsing of node status before ha tests

    If some node fails(shutdown) ha tests fail
    on step when we try ssh to ofline node
    Add nodes state parsing to verify HA recovery
    only on onlines nodes

    Change-Id: Ia89824316bc2228fdefdb4d0cb1cd965bb58145e
    Closes-bug: #1288882

[Anastasia Palkina (apalkina)]

Verified on ISO #266
"build_id": "2014-04-03_04-17-26",
"mirantis": "yes",
"build_number": "266",
"nailgun_sha": "7a05e365240ab27c492b20585ef8ac8557102cc0",
"ostf_sha": "de0222fed646525d248dc6892eeceab139d5c469",
"fuelmain_sha": "16637e2ea0ae6fe9a773aceb9d76c6e3a75f6c3b",
"astute_sha": "5bcacc84cdaee3b31f1178d92a1c0681dc6ce520",
"release": "4.1",
"fuellib_sha": "52e7f57695f33bafa5d84d524d77f1bc3a2289b2"