[OSTF] Need to provide more descriptive messages in HA suite in case if failover was happened on one of controller

Bug #1288882 reported by Tatyanka on 2014-03-06
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Medium
Tatyanka

Bug Description

Steps to repsroduce:
1. Deploy Ha (Centos, ubuntu)
2. shot down one of the controller
3. Run ostf

Actual result:
Ha tests failed:
Check amount of tables in databases is the same on each node -
Time limit exceeded while waiting for detect mysql node to finish. Please refer to OpenStack logs for more details.

Check data replication over mysql
- Time limit exceeded while waiting for detect mysql node to finish. Please refer to OpenStack logs for more details.
But this tests fails according test data was not replicated (SSHExecCommandFailed: Command 'mysql -e 'DROP DATABASE ost1484'', exit status: 1, Error:
ERROR 1008 (HY000) at line 1: Can't drop database 'ost1484'; database doesn't exist)

Expected:
1. Tests may fails but descriptive message must show
2. for those tests where we ssh on controller and examine status on mysql on each controller - we should use here only online nodes

Tatyanka (tatyana-leontovich) wrote :
Changed in fuel:
importance: Undecided → Medium
Andrew Woodward (xarses) wrote :

We should first determine if a node is online or not and then if mysql is up. If there is a quorum between the remaining nodes, we should warn that all nodes aren't online instead of a vague failure message

Tatyanka (tatyana-leontovich) wrote :

Thank you Andrew, absolutely agree, that's why we've created the issue.

Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Tatyana (tatyana-leontovich)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/80551
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=be64966dd549ba6bf7792ca8a1929b46c4d77c75
Submitter: Jenkins
Branch: master

commit be64966dd549ba6bf7792ca8a1929b46c4d77c75
Author: Tatyana Leontovich <email address hidden>
Date: Fri Mar 14 14:52:54 2014 +0000

    Add parsing of node status before ha tests

    If some node fails(shutdown) ha tests fail
    on step when we try ssh to ofline node
    Add nodes state parsing to verify HA recovery
    only on onlines nodes

    Change-Id: Ia89824316bc2228fdefdb4d0cb1cd965bb58145e
    Closes-bug: #1288882

Changed in fuel:
status: In Progress → Fix Committed
tags: added: in progress
Anastasia Palkina (apalkina) wrote :

Verified on ISO #33
"build_id": "2014-03-18_01-01-16",
"mirantis": "yes",
"build_number": "33",
"nailgun_sha": "a5f0a430e4c0f3315d8ea160d4a7a7c83c7d1917",
"ostf_sha": "31c8316d4d9762f280da9e360d3d42e88464aac1",
"fuelmain_sha": "d494e5cf2b8399899f00b5079eec1c5f70c0b66f",
"astute_sha": "a80acd1312be70676f0dd936814b27ce01fb013d",
"release": "5.0",
"fuellib_sha": "0f2765736a2e4af7265eeb936161f0a90cdc9a2e

tags: removed: in progress
Changed in fuel:
status: Fix Committed → Fix Released
Changed in fuel:
status: Fix Released → In Progress
tags: added: backports-4.1.1

Reviewed: https://review.openstack.org/83092
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=1096bbff82b64a1a70490afd825ca592dd198c48
Submitter: Jenkins
Branch: stable/4.1

commit 1096bbff82b64a1a70490afd825ca592dd198c48
Author: Tatyana Leontovich <email address hidden>
Date: Fri Mar 14 14:52:54 2014 +0000

    4.1.1 Add parsing of node status before ha tests

    If some node fails(shutdown) ha tests fail
    on step when we try ssh to ofline node
    Add nodes state parsing to verify HA recovery
    only on onlines nodes

    Change-Id: Ia89824316bc2228fdefdb4d0cb1cd965bb58145e
    Closes-bug: #1288882

Changed in fuel:
status: In Progress → Fix Committed
tags: added: inprogress
Anastasia Palkina (apalkina) wrote :

Verified on ISO #266
"build_id": "2014-04-03_04-17-26",
"mirantis": "yes",
"build_number": "266",
"nailgun_sha": "7a05e365240ab27c492b20585ef8ac8557102cc0",
"ostf_sha": "de0222fed646525d248dc6892eeceab139d5c469",
"fuelmain_sha": "16637e2ea0ae6fe9a773aceb9d76c6e3a75f6c3b",
"astute_sha": "5bcacc84cdaee3b31f1178d92a1c0681dc6ce520",
"release": "4.1",
"fuellib_sha": "52e7f57695f33bafa5d84d524d77f1bc3a2289b2"

tags: removed: inprogress
Changed in fuel:
status: Fix Committed → Fix Released
Andrew Woodward (xarses) on 2014-04-04
tags: added: ha
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers