[system-tests] Implement Ha destructive tests that verify ovs and l3 agents recovery after failover

[Tatyanka]

For neutron deployment we should verify several cases:
I Recovery of neutron agents after fail over(pacemaker should restart it) . Base scenario:
1. Deploy Ha with neutron
2. pick up host where ovs is running
3. turn off host
4. wait some time
5. Verify if ovs has been restarted
6. create several vms to be sure that vms receive ip addresses

II. Kill network at all on one of the controller - verify is openstack operational

III. Block traffic in/out traffic on some of controller(50 %, 75% , 95%) - verify if openstack operational(Create vm, allocate floating ip, diff sec groups, check connectivity between vms for diff case, cidr, vms from diffs tenants, vm from one tenant etc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/80212
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=8ef62e5bacc9b2cf2ee00b093ff7f0fefd4a4a4c
Submitter: Jenkins
Branch: master

commit 8ef62e5bacc9b2cf2ee00b093ff7f0fefd4a4a4c
Author: nfedotov <email address hidden>
Date: Thu Mar 13 12:34:46 2014 +0400

    Destructive test for neutron l3 agent

    Pacemaker should restart neutron l3 agent
    Scenario:
    1. Deploy Ha with neutron
    2. Pick up host where l3 agent is running
    3. Turn off host
    4. Wait some time
    6. Create several vms to be sure that vms receive ip addresses
    (Performed by OSTF connectivity tests)

    Partially fix bug: https://bugs.launchpad.net/fuel/+bug/1269806
    Implements first scenario

    Related-bug: #1269806
    Change-Id: I30d06753b72842e0db03f96914029f5d2dec8502

[Andrew Woodward (xarses)]

set back to high, this is a HA issue, it should maybe be better with last couple of patches, please retest.

[Mike Scherbakov (mihgen)]

Please retest it manually. We don't have time to implement system tests for it in 5.0.

[Tatyanka (tatyana-leontovich)]

Actually we have implemented in 4.0 scenario I and need to verify II and III

[Tatyanka (tatyana-leontovich)]

move to medium according we have implemented 1 scenario and should add last ones

[Nastya Urlapova (aurlapova)]

We have to implement only one scenario.

[Tatyanka (tatyana-leontovich)]

l3 agent cases covered by test:
 neutron_l3_migration_after_reset
neutron_l3_migration
neutron_l3_migration_after_destroy
neutron_packets_drop_stat

other cases covers by tests:
disconnect and destroy controllers so move to fix commited

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/170568
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=5f7c3fb18ccc0f5cb860006289aa328cf31ad062
Submitter: Jenkins
Branch: master

commit 5f7c3fb18ccc0f5cb860006289aa328cf31ad062
Author: Tatyana Leontovich <email address hidden>
Date: Thu Apr 2 13:28:52 2015 +0300

    Add case with 75 % packages loss on br-mgmt

    Closes-Bug: #1269806
    Change-Id: I7fcba12d1cfc5d024289bcc357d5baba9579bcc6

[Tatyanka (tatyana-leontovich)]

last case was executed in swarm 6.1-329 iso