Functional test test_keepalived_state_change_notification may fail due to race condition

[Slawek Kaplonski]

Functional test neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_keepalived_state_change_notification may fail due to setting initial status of keepalived_state during starting of IP monitor.

Some time ago patch https://review.opendev.org/#/c/642295/ introduced checking initial status of VIP in router's namespace during IPMonitor initialisation process.
It works fine but in this specific test, if keepalived will first set router to master and than IPMonitor will be spawned, enqueue_state_change() function will be called 4 instead of 3 times and test will fail.

Example of failure http://logs.openstack.org/69/668569/3/check/neutron-functional-python27/9d345e8/testr_results.html.gz

Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%2048%2C%20in%20test_keepalived_state_change_notification%5C%22

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/670815

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/670815
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7bb1bbba365ad1358d6e66f88a8a48b7261c024e
Submitter: Zuul
Branch: master

commit 7bb1bbba365ad1358d6e66f88a8a48b7261c024e
Author: Slawek Kaplonski <email address hidden>
Date: Mon Jul 15 13:03:30 2019 +0200

    Fix race in test_keepalived_state_change_notification

    In case if initial keepalived status find in
    keepalived_state_change.MonitorDaemon is "master"
    this test_keepalived_state_change_notification was failing
    because there was 4 calls to the mocked enqueue_state_change()
    method instead of 3.

    This patch changes test to wait until 3 or 4 calls to this
    method will be counted and it also changes assertions of
    what state should be set on which call.
    Before the patch test was expecting that calls are always like:
    backup, master, backup
    but if there are 4 calls it is like: backup, master, master, backup.
    As it doesn't matter if there was one or two calls with "master"
    state, test will now assert that the last call is always with
    "backup" state.

    Change-Id: I78c30ab32ffda37176a9c71348d83e17ab2c972a
    Closes-Bug: #1836565

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.