[system tests] Test nova_delete_vips failed by waiting timeout(vip is not recovered for 60 s)

[Tatyanka]

http://jenkins-product.srt.mirantis.net:8080/job/6.1.system_test.centos.thread_5/143/testReport/junit/(root)/ha_nova_delete_vips/ha_nova_delete_vips/

 Delete management and public VIPs 10 times.
        Verify that they are restored.
        Verify cluster by OSTF

        Scenario:
            1. Delete 10 time public and management VIPs
            2. Wait while it is being restored
            3. Verify it is restored
            4. Run OSTF

Test failed with timeout error. Resource has not been restored for a 60 sec.

Assign this issue to the QA team, to reproduce it manually, and try to understand what time we should waiting for recovered vip

[Nastya Urlapova (aurlapova)]

Looks like duplicate of https://bugs.launchpad.net/fuel/+bug/1460654

[Tatyanka (tatyana-leontovich)]

It is not duplicate of https://bugs.launchpad.net/fuel/+bug/1460654, in 1460654 ostf fail with 401 error, but here vip does not appear in ns, so we fail before ostf execution

[Bogdan Dobrelya (bogdando)]

I will reproduce this bug and investigate if this is an HA issue or the test case related only

[Bogdan Dobrelya (bogdando)]

I cannot reproduce this bug after 1500 delete vip iterations. The test script was http://pastebin.com/ftHKW75w
The failover time for removed IP address to reappear was ~10-18 seconds.

[Bogdan Dobrelya (bogdando)]

Marked as invalid as for this specific case the failure was caused by lost pacemaker quorum: http://pastebin.com/UaSFPHf5

[Bogdan Dobrelya (bogdando)]

The RC of the quorum lost seems multiple link issues on hapr* interfaces http://pastebin.com/TDHheZUt

[Bogdan Dobrelya (bogdando)]

Although, I can be wrong with the RCA as these "link lost" messages could be related to the test case activities - when it removes VIP from hapr-m interface

[Bogdan Dobrelya (bogdando)]

But anyway, this test does not expect a quorum loss in pacemaker, hence invalid case