neutron.tests.unit.scheduler.test_dhcp_agent_scheduler.TestNetworksFailover.test_filter_bindings test can fail depending on generated UUIDs

Bug #1839595 reported by Bernard Cafarelli
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Undecided
Bernard Cafarelli

Bug Description

On my CentOS system, this test can locally fail 30-50% of the time - tested from queens to master:
==============================
Failed 1 tests - output below:
==============================

neutron.tests.unit.scheduler.test_dhcp_agent_scheduler.TestNetworksFailover.test_filter_bindings
------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "neutron/tests/base.py", line 177, in func
        return f(self, *args, **kwargs)
      File "neutron/tests/unit/scheduler/test_dhcp_agent_scheduler.py", line 527, in test_filter_bindings
        self.assertIn(network_ids[2], res_ids)
      File "/home/stack/neutron/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 417, in assertIn
        self.assertThat(haystack, Contains(needle), message)
      File "/home/stack/neutron/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: '8a6b1ce0-fc2e-40b0-bc98-a59722d96a3f' not in ['6f6c1774-b351-4763-a8d8-184c3bdf01de', '8cee4a38-63e8-4c71-99e9-611b61aa4c90']

The test creates 4 networks (random UUIDs), then 4 NetworkDhcpAgentBindings,
It then gets a list of these bindings with NetworkDhcpAgentBinding.get_objects() which uses a sort based on network_id by default.
This list is then used with _filter_bindings:
          with mock.patch.object(self, 'agent_starting_up',
                                 side_effect=[True, False]):
              res = [b for b in self._filter_bindings(None, bindings_objs)]
[...]
              res_ids = [b.network_id for b in res]
              self.assertIn(network_ids[2], res_ids)
              self.assertIn(network_ids[3], res_ids)

But as network_ids is not sorted, this can fail depending on the generated UUIDs. One example on my system:
network_ids:
['6f6c1774-b351-4763-a8d8-184c3bdf01de', '8cee4a38-63e8-4c71-99e9-611b61aa4c90', '8a6b1ce0-fc2e-40b0-bc98-a59722d96a3f', '4f476b11-085f-47bf-a27f-e300eb9a85b4']

binding_objs:
NetworkDhcpAgentBinding(dhcp_agent_id=1d6b3b74-afa1-410b-880e-be4f87ce7c6d,network_id=4f476b11-085f-47bf-a27f-e300eb9a85b4)
NetworkDhcpAgentBinding(dhcp_agent_id=07030f55-35cd-4386-853f-70777cdcae2b,network_id=6f6c1774-b351-4763-a8d8-184c3bdf01de)
NetworkDhcpAgentBinding(dhcp_agent_id=1d6b3b74-afa1-410b-880e-be4f87ce7c6d,network_id=8a6b1ce0-fc2e-40b0-bc98-a59722d96a3f)
NetworkDhcpAgentBinding(dhcp_agent_id=07030f55-35cd-4386-853f-70777cdcae2b,network_id=8cee4a38-63e8-4c71-99e9-611b61aa4c90)

which will give a (failing the test) res_ids: ['6f6c1774-b351-4763-a8d8-184c3bdf01de', '8cee4a38-63e8-4c71-99e9-611b61aa4c90']

I am not sure why upstream gates never seem to have the problem (as far as I have checked), but sorting the network_ids makes the test passing all the time

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/675556

Changed in neutron:
status: New → In Progress
Revision history for this message
Bernard Cafarelli (bcafarel) wrote :

Thanks to Rodolfo and Slawek for pointing it out, the get_objects() has no guarantee on order of returned objects, so we should have that list and the network_ids list sorted out the same.

This also explains why changing sqlite version would help with the test passing

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/675556
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f59b6a470620eaf53c98f4d98801789c95ed9a26
Submitter: Zuul
Branch: master

commit f59b6a470620eaf53c98f4d98801789c95ed9a26
Author: Bernard Cafarelli <email address hidden>
Date: Fri Aug 9 10:26:13 2019 +0200

    Fix sort issue in test_dhcp_agent_scheduler.test_filter_bindings

    The test creates a list of networks, and then acts on a list of
    NetworkDhcpAgentBindings obtained from get_objects() not guaranteed to
    follow the original build order (based on the network_ids list)

    Make sure that returned list is sorted on network_id, and network_ids
    itself sorted so both lists match

    Change-Id: I9b07255988f7ba6609af1961b3429c3ce12d5186
    Closes-Bug: #1839595

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/675839

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/675840

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/675841

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/675839
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=750a8e8327388225f9bea549a7de61741739e937
Submitter: Zuul
Branch: stable/stein

commit 750a8e8327388225f9bea549a7de61741739e937
Author: Bernard Cafarelli <email address hidden>
Date: Fri Aug 9 10:26:13 2019 +0200

    Fix sort issue in test_dhcp_agent_scheduler.test_filter_bindings

    The test creates a list of networks, and then acts on a list of
    NetworkDhcpAgentBindings obtained from get_objects() not guaranteed to
    follow the original build order (based on the network_ids list)

    Make sure that returned list is sorted on network_id, and network_ids
    itself sorted so both lists match

    Change-Id: I9b07255988f7ba6609af1961b3429c3ce12d5186
    Closes-Bug: #1839595
    (cherry picked from commit f59b6a470620eaf53c98f4d98801789c95ed9a26)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/675841
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9aa7b045d78248f745cef888dd01676a33b311d2
Submitter: Zuul
Branch: stable/queens

commit 9aa7b045d78248f745cef888dd01676a33b311d2
Author: Bernard Cafarelli <email address hidden>
Date: Fri Aug 9 10:26:13 2019 +0200

    Fix sort issue in test_dhcp_agent_scheduler.test_filter_bindings

    The test creates a list of networks, and then acts on a list of
    NetworkDhcpAgentBindings obtained from get_objects() not guaranteed to
    follow the original build order (based on the network_ids list)

    Make sure that returned list is sorted on network_id, and network_ids
    itself sorted so both lists match

    Change-Id: I9b07255988f7ba6609af1961b3429c3ce12d5186
    Closes-Bug: #1839595
    (cherry picked from commit f59b6a470620eaf53c98f4d98801789c95ed9a26)

tags: added: in-stable-queens
tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/675840
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9fea420c2d9891ed46c1e55ea15f2828fa4b6c89
Submitter: Zuul
Branch: stable/rocky

commit 9fea420c2d9891ed46c1e55ea15f2828fa4b6c89
Author: Bernard Cafarelli <email address hidden>
Date: Fri Aug 9 10:26:13 2019 +0200

    Fix sort issue in test_dhcp_agent_scheduler.test_filter_bindings

    The test creates a list of networks, and then acts on a list of
    NetworkDhcpAgentBindings obtained from get_objects() not guaranteed to
    follow the original build order (based on the network_ids list)

    Make sure that returned list is sorted on network_id, and network_ids
    itself sorted so both lists match

    Change-Id: I9b07255988f7ba6609af1961b3429c3ce12d5186
    Closes-Bug: #1839595
    (cherry picked from commit f59b6a470620eaf53c98f4d98801789c95ed9a26)

tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 15.0.0.0b1

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 14.0.3

This issue was fixed in the openstack/neutron 14.0.3 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 13.0.5

This issue was fixed in the openstack/neutron 13.0.5 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 12.1.1

This issue was fixed in the openstack/neutron 12.1.1 release.

tags: removed: neutron-proactive-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.