Functional test test_keepalived_state_change_notification may fail due to race condition

Bug #1836565 reported by Slawek Kaplonski on 2019-07-15
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Medium
Slawek Kaplonski

Bug Description

Functional test neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_keepalived_state_change_notification may fail due to setting initial status of keepalived_state during starting of IP monitor.

Some time ago patch https://review.opendev.org/#/c/642295/ introduced checking initial status of VIP in router's namespace during IPMonitor initialisation process.
It works fine but in this specific test, if keepalived will first set router to master and than IPMonitor will be spawned, enqueue_state_change() function will be called 4 instead of 3 times and test will fail.

Example of failure http://logs.openstack.org/69/668569/3/check/neutron-functional-python27/9d345e8/testr_results.html.gz

Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%2048%2C%20in%20test_keepalived_state_change_notification%5C%22

Fix proposed to branch: master
Review: https://review.opendev.org/670815

Changed in neutron:
status: Confirmed → In Progress

Reviewed: https://review.opendev.org/670815
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7bb1bbba365ad1358d6e66f88a8a48b7261c024e
Submitter: Zuul
Branch: master

commit 7bb1bbba365ad1358d6e66f88a8a48b7261c024e
Author: Slawek Kaplonski <email address hidden>
Date: Mon Jul 15 13:03:30 2019 +0200

    Fix race in test_keepalived_state_change_notification

    In case if initial keepalived status find in
    keepalived_state_change.MonitorDaemon is "master"
    this test_keepalived_state_change_notification was failing
    because there was 4 calls to the mocked enqueue_state_change()
    method instead of 3.

    This patch changes test to wait until 3 or 4 calls to this
    method will be counted and it also changes assertions of
    what state should be set on which call.
    Before the patch test was expecting that calls are always like:
    backup, master, backup
    but if there are 4 calls it is like: backup, master, master, backup.
    As it doesn't matter if there was one or two calls with "master"
    state, test will now assert that the last call is always with
    "backup" state.

    Change-Id: I78c30ab32ffda37176a9c71348d83e17ab2c972a
    Closes-Bug: #1836565

Changed in neutron:
status: In Progress → Fix Released
summary: - Functional test test_keepalived_state_change_notification may fail do to
- race condition
+ Functional test test_keepalived_state_change_notification may fail due
+ to race condition

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers