neutron.tests.unit.scheduler.test_dhcp_agent_scheduler.TestNetworksFailover.test_filter_bindings test can fail depending on generated UUIDs

Bug #1839595 reported by Bernard Cafarelli on 2019-08-09
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Undecided
Bernard Cafarelli

Bug Description

On my CentOS system, this test can locally fail 30-50% of the time - tested from queens to master:
==============================
Failed 1 tests - output below:
==============================

neutron.tests.unit.scheduler.test_dhcp_agent_scheduler.TestNetworksFailover.test_filter_bindings
------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "neutron/tests/base.py", line 177, in func
        return f(self, *args, **kwargs)
      File "neutron/tests/unit/scheduler/test_dhcp_agent_scheduler.py", line 527, in test_filter_bindings
        self.assertIn(network_ids[2], res_ids)
      File "/home/stack/neutron/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 417, in assertIn
        self.assertThat(haystack, Contains(needle), message)
      File "/home/stack/neutron/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: '8a6b1ce0-fc2e-40b0-bc98-a59722d96a3f' not in ['6f6c1774-b351-4763-a8d8-184c3bdf01de', '8cee4a38-63e8-4c71-99e9-611b61aa4c90']

The test creates 4 networks (random UUIDs), then 4 NetworkDhcpAgentBindings,
It then gets a list of these bindings with NetworkDhcpAgentBinding.get_objects() which uses a sort based on network_id by default.
This list is then used with _filter_bindings:
          with mock.patch.object(self, 'agent_starting_up',
                                 side_effect=[True, False]):
              res = [b for b in self._filter_bindings(None, bindings_objs)]
[...]
              res_ids = [b.network_id for b in res]
              self.assertIn(network_ids[2], res_ids)
              self.assertIn(network_ids[3], res_ids)

But as network_ids is not sorted, this can fail depending on the generated UUIDs. One example on my system:
network_ids:
['6f6c1774-b351-4763-a8d8-184c3bdf01de', '8cee4a38-63e8-4c71-99e9-611b61aa4c90', '8a6b1ce0-fc2e-40b0-bc98-a59722d96a3f', '4f476b11-085f-47bf-a27f-e300eb9a85b4']

binding_objs:
NetworkDhcpAgentBinding(dhcp_agent_id=1d6b3b74-afa1-410b-880e-be4f87ce7c6d,network_id=4f476b11-085f-47bf-a27f-e300eb9a85b4)
NetworkDhcpAgentBinding(dhcp_agent_id=07030f55-35cd-4386-853f-70777cdcae2b,network_id=6f6c1774-b351-4763-a8d8-184c3bdf01de)
NetworkDhcpAgentBinding(dhcp_agent_id=1d6b3b74-afa1-410b-880e-be4f87ce7c6d,network_id=8a6b1ce0-fc2e-40b0-bc98-a59722d96a3f)
NetworkDhcpAgentBinding(dhcp_agent_id=07030f55-35cd-4386-853f-70777cdcae2b,network_id=8cee4a38-63e8-4c71-99e9-611b61aa4c90)

which will give a (failing the test) res_ids: ['6f6c1774-b351-4763-a8d8-184c3bdf01de', '8cee4a38-63e8-4c71-99e9-611b61aa4c90']

I am not sure why upstream gates never seem to have the problem (as far as I have checked), but sorting the network_ids makes the test passing all the time

Fix proposed to branch: master
Review: https://review.opendev.org/675556

Changed in neutron:
status: New → In Progress
Bernard Cafarelli (bcafarel) wrote :

Thanks to Rodolfo and Slawek for pointing it out, the get_objects() has no guarantee on order of returned objects, so we should have that list and the network_ids list sorted out the same.

This also explains why changing sqlite version would help with the test passing

Reviewed: https://review.opendev.org/675556
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f59b6a470620eaf53c98f4d98801789c95ed9a26
Submitter: Zuul
Branch: master

commit f59b6a470620eaf53c98f4d98801789c95ed9a26
Author: Bernard Cafarelli <email address hidden>
Date: Fri Aug 9 10:26:13 2019 +0200

    Fix sort issue in test_dhcp_agent_scheduler.test_filter_bindings

    The test creates a list of networks, and then acts on a list of
    NetworkDhcpAgentBindings obtained from get_objects() not guaranteed to
    follow the original build order (based on the network_ids list)

    Make sure that returned list is sorted on network_id, and network_ids
    itself sorted so both lists match

    Change-Id: I9b07255988f7ba6609af1961b3429c3ce12d5186
    Closes-Bug: #1839595

Changed in neutron:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/675839
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=750a8e8327388225f9bea549a7de61741739e937
Submitter: Zuul
Branch: stable/stein

commit 750a8e8327388225f9bea549a7de61741739e937
Author: Bernard Cafarelli <email address hidden>
Date: Fri Aug 9 10:26:13 2019 +0200

    Fix sort issue in test_dhcp_agent_scheduler.test_filter_bindings

    The test creates a list of networks, and then acts on a list of
    NetworkDhcpAgentBindings obtained from get_objects() not guaranteed to
    follow the original build order (based on the network_ids list)

    Make sure that returned list is sorted on network_id, and network_ids
    itself sorted so both lists match

    Change-Id: I9b07255988f7ba6609af1961b3429c3ce12d5186
    Closes-Bug: #1839595
    (cherry picked from commit f59b6a470620eaf53c98f4d98801789c95ed9a26)

tags: added: in-stable-stein

Reviewed: https://review.opendev.org/675841
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9aa7b045d78248f745cef888dd01676a33b311d2
Submitter: Zuul
Branch: stable/queens

commit 9aa7b045d78248f745cef888dd01676a33b311d2
Author: Bernard Cafarelli <email address hidden>
Date: Fri Aug 9 10:26:13 2019 +0200

    Fix sort issue in test_dhcp_agent_scheduler.test_filter_bindings

    The test creates a list of networks, and then acts on a list of
    NetworkDhcpAgentBindings obtained from get_objects() not guaranteed to
    follow the original build order (based on the network_ids list)

    Make sure that returned list is sorted on network_id, and network_ids
    itself sorted so both lists match

    Change-Id: I9b07255988f7ba6609af1961b3429c3ce12d5186
    Closes-Bug: #1839595
    (cherry picked from commit f59b6a470620eaf53c98f4d98801789c95ed9a26)

tags: added: in-stable-queens
tags: added: in-stable-rocky

Reviewed: https://review.opendev.org/675840
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9fea420c2d9891ed46c1e55ea15f2828fa4b6c89
Submitter: Zuul
Branch: stable/rocky

commit 9fea420c2d9891ed46c1e55ea15f2828fa4b6c89
Author: Bernard Cafarelli <email address hidden>
Date: Fri Aug 9 10:26:13 2019 +0200

    Fix sort issue in test_dhcp_agent_scheduler.test_filter_bindings

    The test creates a list of networks, and then acts on a list of
    NetworkDhcpAgentBindings obtained from get_objects() not guaranteed to
    follow the original build order (based on the network_ids list)

    Make sure that returned list is sorted on network_id, and network_ids
    itself sorted so both lists match

    Change-Id: I9b07255988f7ba6609af1961b3429c3ce12d5186
    Closes-Bug: #1839595
    (cherry picked from commit f59b6a470620eaf53c98f4d98801789c95ed9a26)

tags: added: neutron-proactive-backport-potential

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

This issue was fixed in the openstack/neutron 14.0.3 release.

This issue was fixed in the openstack/neutron 13.0.5 release.

This issue was fixed in the openstack/neutron 12.1.1 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers