[regression] We lost 11 % of packages after OVS restart: test_ovs_restart_with_iperf_traffic

[Timur Nurlygayanov]

Detailed bug description:
Automated test for OVS restart failed because we lost too many traffic in iperf test during the restart of OVS agents.

Steps to reproduce:
1. Run iperf server on server2
2. Run iperf client on server 1
3. Check that packet losses < 1%
4. Disable ovs-agents on all controllers,
restart service neutron-plugin-openvswitch-agent
on all computes, and enable them back.
5. Check that all ovs-agents are in alive state
6. Check that iperf traffic wasn't interrupted during ovs restart,
and not more than 10% datagrams are lost

Expected results:
We shouldn't lost more than 10% of traffic

Actual result:
We lost more than 10%.

Reproducibility:
100 %

Workaround:
We don't know the workaround.

Impact:
Users will lost connectivity during the OVS restart and it takes more time to recover network services in case of any failures / maintenance.

Additional information:
AssertionError: 11.0% datagrams lost. Should be < 10% assert 11.0 < 10
Stacktrace

self = <mos_tests.neutron.python_tests.test_ovs_restart.TestOVSRestartWithIperfTraffic object at 0x7f0cafe04f50>

    @pytest.mark.testrail_id('542659')
    @pytest.mark.require_QCOW2_ubuntu_image_with_iperf
    def test_ovs_restart_with_iperf_traffic(self):
        """Checks that iperf traffic is not interrupted during ovs restart

            Steps:
                1. Run iperf server on server2
                2. Run iperf client on server 1
                3. Check that packet losses < 1%
                4. Disable ovs-agents on all controllers,
                    restart service neutron-plugin-openvswitch-agent
                    on all computes, and enable them back.
                5. Check that all ovs-agents are in alive state
                6. Check that iperf traffic wasn't interrupted during ovs restart,
                    and not more than 10% datagrams are lost
            """
        self._prepare_openstack()
        # Launch iperf server on server2
        res = self.launch_iperf_server(self.server2, self.instance_keypair,
                                       vm_login='ubuntu', vm_pwd='ubuntu')
        err_msg = 'Failed to start the iperf server on vm result: {}'.format(
            res)
        assert not res['exit_code'], err_msg

        # Launch iperf client on server1
        res = self.launch_iperf_client(self.server1, self.server2,
                                       self.instance_keypair,
                                       vm_login='ubuntu', vm_pwd='ubuntu')
        err_msg = 'Failed to start the iperf client on vm result: {}'.format(
            res)
        assert not res['exit_code'], err_msg

        # Check iperf traffic before restart
        lost = self.get_lost_percentage(res['stdout'])
        err_msg = "Packet losses more than 0%. Actual value is {0}%".format(
            lost)
        assert lost == 0, err_msg

        self.os_conn.wait_agents_alive(self.ovs_agent_ids)

        # Launch client in background and restart agents
        res = self.launch_iperf_client(self.server1, self.server2,
                                       self.instance_keypair,
                                       vm_login='ubuntu', vm_pwd='ubuntu',
                                       background=True)
        err_msg = 'Failed to start the iperf client on vm result: {}'.format(
            res)
        assert not res['exit_code'], err_msg

        self.disable_ovs_agents_on_controller()
        self.os_conn.wait_agents_down(self.ovs_conroller_agents)
        self.restart_ovs_agents_on_computes()
        self.enable_ovs_agents_on_controllers()
        self.os_conn.wait_agents_alive(self.ovs_agent_ids)

        cmd = 'cat ~/iperf_client.log'

        def get_lost():
            result = self.run_on_vm(self.server1, self.instance_keypair, cmd,
                                    vm_login='ubuntu', vm_password='ubuntu')
            return self.get_lost_percentage(result['stdout'])

        lost = wait(get_lost, timeout_seconds=5 * 60, sleep_seconds=5,
                    waiting_for='interrupt iperf traffic')

        err_msg = "{0}% datagrams lost. Should be < 10%".format(lost)
> assert lost < 10, err_msg
E AssertionError: 11.0% datagrams lost. Should be < 10%
E assert 11.0 < 10

mos_tests/neutron/python_tests/test_ovs_restart.py:832: AssertionError

[Timur Nurlygayanov (tnurlygayanov)]

Kristina, could you please fix the test case?

Thank you!

[Timur Nurlygayanov (tnurlygayanov)]

It looks like the issue was successfully fixed, here the result of the latest test run:

https://mirantis.testrail.com/index.php?/runs/view/10558&group_by=cases:section_id&group_order=asc&group_id=23055