Functional tests are failing intermittently

[Numan Siddique]

Failure 1
-----------

2017-04-24 09:52:37.383170 | 2017-04-24 09:52:37.382 |
2017-04-24 09:52:37.385186 | 2017-04-24 09:52:37.384 | Captured traceback:
2017-04-24 09:52:37.387186 | 2017-04-24 09:52:37.386 | ~~~~~~~~~~~~~~~~~~~
2017-04-24 09:52:37.389274 | 2017-04-24 09:52:37.388 | Traceback (most recent call last):
2017-04-24 09:52:37.391355 | 2017-04-24 09:52:37.390 | File "/opt/stack/new/neutron/neutron/tests/base.py", line 115, in func
2017-04-24 09:52:37.393341 | 2017-04-24 09:52:37.392 | return f(self, *args, **kwargs)
2017-04-24 09:52:37.395598 | 2017-04-24 09:52:37.395 | File "/opt/stack/new/neutron/neutron/tests/base.py", line 115, in func
2017-04-24 09:52:37.397701 | 2017-04-24 09:52:37.397 | return f(self, *args, **kwargs)
2017-04-24 09:52:37.400006 | 2017-04-24 09:52:37.399 | File "networking_ovn/tests/functional/test_ovsdb_monitor.py", line 84, in test_port_up_down_events
2017-04-24 09:52:37.402161 | 2017-04-24 09:52:37.401 | self._test_port_up_down_helper(p, self.mech_driver)
2017-04-24 09:52:37.404203 | 2017-04-24 09:52:37.403 | File "networking_ovn/tests/functional/test_ovsdb_monitor.py", line 40, in _test_port_up_down_helper
2017-04-24 09:52:37.406521 | 2017-04-24 09:52:37.406 | ovn_mech_driver.set_port_status_up.assert_called_once_with(port['id'])
2017-04-24 09:52:37.408740 | 2017-04-24 09:52:37.408 | File "/opt/stack/new/networking-ovn/.tox/dsvm-functional/local/lib/python2.7/site-packages/mock/mock.py", line 947, in assert_called_once_with
2017-04-24 09:52:37.411108 | 2017-04-24 09:52:37.410 | raise AssertionError(msg)
2017-04-24 09:52:37.413087 | 2017-04-24 09:52:37.412 | AssertionError: Expected 'mock' to be called once. Called 0 times.

Failure 2
---------

networking_ovn.tests.unit.l3.test_l3_ovn.OVNL3RouterPlugin.test_disable_snat
----------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
    b'Traceback (most recent call last):'
    b' File "/home/jenkins/workspace/gate-networking-ovn-python35/.tmp/tmp.4qLmOL284Q/openstack/neutron/neutron/tests/base.py", line 115, in func'
    b' return f(self, *args, **kwargs)'
    b' File "/home/jenkins/workspace/gate-networking-ovn-python35/.tox/py35/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched'
    b' return func(*args, **keywargs)'
    b' File "/home/jenkins/workspace/gate-networking-ovn-python35/networking_ovn/tests/unit/l3/test_l3_ovn.py", line 567, in test_disable_snat'
    b' self.l3_plugin._ovn.delete_static_route.assert_not_called()'
    b' File "/home/jenkins/workspace/gate-networking-ovn-python35/.tox/py35/lib/python3.5/site-packages/mock/mock.py", line 897, in assert_not_called'
    b' raise AssertionError(msg)'
    b"AssertionError: Expected 'mock' to not have been called. Called 1 times."

Both the failures are seen independently

[Lucas Alvares Gomes (lucasagomes)]

I've proposed a patch to add a query to elastic recheck for this problem: https://review.openstack.org/#/c/459622/

I think we have a race condition somewhere in the unit-tests

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/459757
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=0628e125eb3070afcbaf7a02ae5e2271676e984a
Submitter: Jenkins
Branch: master

commit 0628e125eb3070afcbaf7a02ae5e2271676e984a
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Apr 25 15:30:18 2017 +0100

    Fix the L3 unit tests race condition

    This patch is an attempt to fix up the race condition we have seem in
    the L3 unit tests. This patch does two things:

    * We make sure that all mocks that are started at the setUp() method are
      also stopped (self.addCleanup)

    * Remove the name collision with the "l3_plugin" attribute in the test
      class. The base class already contains a class attribute called
      "l3_plugin" which holds a path (string) as the value but it was being
      overritten in the setUp() method with an instance of the L3 plugin.

    Change-Id: I2c25bd0c31958c0ea769f53b88d6dcc92669a630
    Partial-Bug: #1685844

[Dong Jun (dongj)]

I've submitted a patch to fix two NAt failure
https://review.openstack.org/#/c/460402/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/460402
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=00bb39fb69cf9f1f6e8b487667e1163b0b74bb9e
Submitter: Jenkins
Branch: master

commit 00bb39fb69cf9f1f6e8b487667e1163b0b74bb9e
Author: Dong Jun <email address hidden>
Date: Thu Apr 27 13:33:56 2017 +0800

    Fix intermittent failure of nat unit test cases

    This patch fixes intermittent failure of two unit test cases,
    test_enable_snat and test_disable_snat.
    The reason is, str(dict) has a stable output in python 2 rather than in
    python 3. I copy the checking logic from _check_for_external_ip_change

    Change-Id: I3b532e875906d38d2d4f4ffb85ba4fb7886dfa7b
    Partial-Bug: #1685844
    Signed-off-by: Dong Jun <email address hidden>