Swift rebalance returns code 1 after redeploy

[Tatyana Kuterina]

Detailed bug description:
Found on CI:
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ha_neutron_tun_scale/174/console

Test: Check HA mode on scalability
Test Group: neutron_tun_scalability

Steps to reproduce:
        1. Create cluster
        2. Add 1 controller node
        3. Deploy the cluster
        4. Check swift, and invoke swift-rings-rebalance.sh
           on primary controller if check failed
        5. Add 2 controller nodes
        6. Deploy changes
        7. Check swift, and invoke swift-rings-rebalance.sh
           on primary controller if check failed
Expected results:
 swift-rings-rebalance finished without errors with exit code 0.
Actual result:
 UnexpectedExitCode: Command '/usr/local/bin/swift-rings-rebalance.sh' returned unexpected exit code 1, while waiting for [0]stdout: Reassigned 1024 (100.00%) partitions. Balance is now 50.00. Dispersion is now 0.00
-------------------------------------------------------------------------------
NOTE: Balance of 50.00 indicates you should push this
      ring, wait at least 1 hours, and rebalance/repush.
-------------------------------------------------------------------------------
Reassigned 1024 (100.00%) partitions. Balance is now 50.29. Dispersion is now 0.00
-------------------------------------------------------------------------------
NOTE: Balance of 50.29 indicates you should push this
      ring, wait at least 1 hours, and rebalance/repush.
-------------------------------------------------------------------------------
Reassigned 1024 (100.00%) partitions. Balance is now 50.29. Dispersion is now 0.00
-------------------------------------------------------------------------------
NOTE: Balance of 50.29 indicates you should push this
      ring, wait at least 1 hours, and rebalance/repush.
-------------------------------------------------------------------------------
Reproducibility:
 frequently

Description of the environment:
snapshot 21
https://drive.google.com/a/mirantis.com/file/d/0Bz15vbpS5ZPNNXdXNW56ZDVxUGs/view?usp=sharing

[Oleksiy Molchanov (omolchanov)]

Lowering to medium, didn't reproduced on my local env.

[ElenaRossokhina (esolomina)]

reproduced again https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_tun_scale/3/testReport/(root)/neutron_tun_scalability/

[Nastya Urlapova (aurlapova)]

@Oleksey, you can find env after swarm run:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_tun_scale/8/testReport/(root)/neutron_tun_scalability/neutron_tun_scalability/

[Oleksiy Molchanov (omolchanov)]

Guys, this test case need to be updated.

You are getting "Balance of X.X indicates you should push this ring, wait at least 1 hours, and rebalance/repush.". It is not an error, it just indicates that you need to make rebalance in different way.

Procedure is described here: https://wikitech.wikimedia.org/wiki/Swift/How_To#Rebalance_the_rings

PS: we are doing such a rebalance and ring-copy using crontab.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/373094

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/373094
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=380d393e2fd2693d757526d620f1b9d4d643219c
Submitter: Jenkins
Branch: stable/mitaka

commit 380d393e2fd2693d757526d620f1b9d4d643219c
Author: Dmitriy Kruglov <email address hidden>
Date: Tue Sep 20 09:01:55 2016 +0000

    Fix helpers.common.rebalance_swift_ring method

    After recent refactoring of SSH manager executing
    a command on a remote by default raises exception in
    case the command exit code is any other than 0.
    This effected in sporadical fails of
    helpers.common.rebalance_swift_ring method, where
    rebalance used to be done several times in some tests
    and due to abovementioned change started to fail after
    the very first unsuccessful attempt.

    Change-Id: Ifbc832b1dad53b33ca223d3500306c386bab0d91
    Closes-Bug: #1604271

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/374589

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/374589
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=cfb610e520bd26cde8a236945969b3e66d371d32
Submitter: Jenkins
Branch: master

commit cfb610e520bd26cde8a236945969b3e66d371d32
Author: Dmitriy Kruglov <email address hidden>
Date: Tue Sep 20 09:01:55 2016 +0000

    Fix helpers.common.rebalance_swift_ring method

    After recent refactoring of SSH manager executing
    a command on a remote by default raises exception in
    case the command exit code is any other than 0.
    This effected in sporadical fails of
    helpers.common.rebalance_swift_ring method, where
    rebalance used to be done several times in some tests
    and due to abovementioned change started to fail after
    the very first unsuccessful attempt.

    Change-Id: Ifbc832b1dad53b33ca223d3500306c386bab0d91
    Closes-Bug: #1604271

[Dmitriy Kruglov (dkruglov)]

Verified on latest Swarm run https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_tun_scale/71/
The issue is fixed.