Unit tests failing due to recent Neutron patch

[Daniel Alvarez]

Since Nov 7th we have unit tests failing. I've doing git bisect on neutron and found that [0] is the culprit. Digging further I checked that it's not actually that MAX_RETRIES changed from 10 (neutron code) to 20 (neutron-lib) but the fact that "from neutron.db import api as db_api" is no longer imported.

[0] https://github.com/openstack/neutron/commit/3316b45665a99b0f61e45a8c7facf538618861bf

[Daniel Alvarez (dalvarezs)]

Failures:

ft1.10: networking_ovn.tests.unit.ml2.test_mech_driver.TestOVNMechansimDriverSubnetsV2.test_create_subnet_gw_values_StringException: Traceback (most recent call last):
  File "/home/zuul/src/git.openstack.org/openstack/networking-ovn/.tox/py27/local/lib/python2.7/site-packages/neutron/tests/base.py", line 151, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/git.openstack.org/openstack/networking-ovn/.tox/py27/local/lib/python2.7/site-packages/neutron/tests/base.py", line 151, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/git.openstack.org/openstack/networking-ovn/.tox/py27/local/lib/python2.7/site-packages/neutron/tests/base.py", line 151, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/git.openstack.org/openstack/networking-ovn/.tox/py27/local/lib/python2.7/site-packages/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 3781, in test_create_subnet_gw_values
    self._test_create_subnet(expected=expected, gateway_ip=gateway)
  File "/home/zuul/src/git.openstack.org/openstack/networking-ovn/.tox/py27/local/lib/python2.7/site-packages/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 3127, in _test_create_subnet
    self._compare_resource(subnet, expected, 'subnet')
  File "/home/zuul/src/git.openstack.org/openstack/networking-ovn/.tox/py27/local/lib/python2.7/site-packages/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 787, in _compare_resource
    sorted(observed_res[res_name][k]))
  File "/home/zuul/src/git.openstack.org/openstack/networking-ovn/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/home/zuul/src/git.openstack.org/openstack/networking-ovn/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: [{'end': '10.0.0.253', 'start': '10.0.0.1'}] != []

Workaround: https://review.openstack.org/#/c/616668/

[Daniel Alvarez (dalvarezs)]

There's tons of errors like:

RuntimeError: Method <function create_network at 0x7f88b9d5d5f0> cannot be called within a transaction.

RuntimeError: Method <function create_port at 0x7f076516f500> cannot be called within a transaction.

...

[Daniel Alvarez (dalvarezs)]

When importing that module, these event listeners are created:

https://github.com/openstack/neutron/blob/master/neutron/db/api.py#L110 and https://github.com/openstack/neutron/blob/master/neutron/db/api.py#L134

Adding them manually fixed the issue. So far the workaround imports the file to get the listeners imported but this is something that Neutron folks need to confirm now and perhaps move them somewhere else to get them imported either way.

[Slawek Kaplonski (slaweq)]

So maybe we should import neutron.db.api module in neutron/common/utils.py again (with "noqa" to make pep8 happy).
Or any other, better solutions?

[Daniel Alvarez (dalvarezs)]

Yes, this is what I did here for networking-ovn: https://review.openstack.org/#/c/616668/

Perhaps it's needed in Neutron too? Apparently it didn't fail in neutron gate

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/617119

[OpenStack Infra (hudson-openstack)]

Change abandoned by Daniel Alvarez (<email address hidden>) on branch: master
Review: https://review.openstack.org/616668
Reason: https://review.openstack.org/#/c/617307/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/617307
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=605ad27f053a1ef9ffb3601c6023987ddb6a22e1
Submitter: Zuul
Branch: master

commit 605ad27f053a1ef9ffb3601c6023987ddb6a22e1
Author: Daniel Alvarez <email address hidden>
Date: Mon Nov 12 13:21:41 2018 +0100

    Fix Tempest and Unit tests

    This patch is a squash of two commits that fix both tempest and
    unit tests:

    1. Port Groups: Fix bug when updating a port to trusted

    When a normal port with port security enabled (ie. belonging to
    default drop Port Group) gets updated to a trusted port (ie.
    router port), the existing code won't remove the port from the
    Port Group so all traffic gets dropped.

    This patch fixes the issue by checking this condition in the
    update_port method of the ovn_client module.

    2. UT: Fix unit tests failures

    After 3316b45665a99b0f61e45a8c7facf538618861bf got merged in
    Neutron, our gate is blocked due to unit tests failing.
    After some debugging, it looks like no longer importing
    neutron.db.api is the culprit. This patch is importing it
    from our unit tests to unbreak the gate while the actual
    root case is determined.

    Change-Id: I3d30ec18f0c9df256c8f5846b52ab619835b5e32
    Closes-Bug: #1802373
    Partial-Bug: #1802369
    Signed-off-by: Daniel Alvarez <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/617979

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/617979
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=e83edbc171a33fa5f495b5b9fe483988419f2bfd
Submitter: Zuul
Branch: stable/rocky

commit e83edbc171a33fa5f495b5b9fe483988419f2bfd
Author: Daniel Alvarez <email address hidden>
Date: Mon Nov 12 13:21:41 2018 +0100

    Fix Tempest and Unit tests

    This patch is a squash of two commits that fix both tempest and
    unit tests:

    1. Port Groups: Fix bug when updating a port to trusted

    When a normal port with port security enabled (ie. belonging to
    default drop Port Group) gets updated to a trusted port (ie.
    router port), the existing code won't remove the port from the
    Port Group so all traffic gets dropped.

    This patch fixes the issue by checking this condition in the
    update_port method of the ovn_client module.

    2. UT: Fix unit tests failures

    After 3316b45665a99b0f61e45a8c7facf538618861bf got merged in
    Neutron, our gate is blocked due to unit tests failing.
    After some debugging, it looks like no longer importing
    neutron.db.api is the culprit. This patch is importing it
    from our unit tests to unbreak the gate while the actual
    root case is determined.

    Change-Id: I3d30ec18f0c9df256c8f5846b52ab619835b5e32
    Closes-Bug: #1802373
    Partial-Bug: #1802369
    Signed-off-by: Daniel Alvarez <email address hidden>
    (cherry picked from commit 605ad27f053a1ef9ffb3601c6023987ddb6a22e1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/617119
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=593c04d10d9d67cc6f864ff37416951dd28584f5
Submitter: Zuul
Branch: master

commit 593c04d10d9d67cc6f864ff37416951dd28584f5
Author: Slawek Kaplonski <email address hidden>
Date: Sat Nov 10 15:48:20 2018 +0100

    Import neutron.db.api module in common.utils

    Patch [1] removed import of neutron.db.api module in
    neutron.common.utils.
    This cause some issues e.g. in networking-ovn projects because
    listeners:

    @event.listens_for(orm.session.Session, "persistent_to_deleted")
    on _persistent_to_deleted() function [2] and

    @event.listens_for(model_base.BASEV2, "attribute_instrument", propagate=True)
    on _listen_for_changes() function [3]

    aren't created as they where before.

    This patch adds import of this neutron.db.api module again just to
    ensure that those listeners are created properly.

    [1] https://review.openstack.org/#/c/614252/
    [2] https://github.com/openstack/neutron/blob/master/neutron/db/api.py#L110
    [3] https://github.com/openstack/neutron/blob/master/neutron/db/api.py#L134

    Closes-Bug: #1802369

    Change-Id: I3e427ee5c904c34115aaf1afa67e1109ea58d603

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/625332

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0b1 development milestone.

[Slawek Kaplonski (slaweq)]

This patch https://review.openstack.org/#/c/625332/ to neutron-lib should fix such issues in proper way :)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/625332
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=14bc0d940841de31f055fa3170289e0fa671b7bb
Submitter: Zuul
Branch: master

commit 14bc0d940841de31f055fa3170289e0fa671b7bb
Author: Boden R <email address hidden>
Date: Fri Dec 14 12:05:29 2018 -0700

    rehome db api orm event listener functions

    The rehome/consumption of the db api caused some errors in consumer
    projects related to the ORM event listeners no longer getting
    initialized [1]. While the short term fix [1] was to import neutron's
    db api elsewhere, this doesn't work longer term as consumers need to
    decouple from neutron, thus not importing neutron modules.

    This patch rehomes the db api ORM event listeners into neutron-lib and
    initializes them upon import of neutron_lib (top-level). This change
    will allow consumers to load the event listeners by importing anything
    from neutron-lib, thus breaking the dependency on neutron.

    This patch also bumps the requirement for SQLAlchemy to match neutrons.

    [1] https://bugs.launchpad.net/neutron/+bug/1802369

    Related-Bug: 1802369

    Change-Id: I3e702b99fd5084e8090f93c384aa1f704edceaff

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/681568

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/681568
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=49e6e141f254762f63ba8347ead84e28eefca838
Submitter: Zuul
Branch: stable/queens

commit 49e6e141f254762f63ba8347ead84e28eefca838
Author: Daniel Alvarez <email address hidden>
Date: Mon Nov 12 13:21:41 2018 +0100

    Fix Tempest and Unit tests

    This patch is a squash of two commits that fix both tempest and
    unit tests:

    1. Port Groups: Fix bug when updating a port to trusted

    When a normal port with port security enabled (ie. belonging to
    default drop Port Group) gets updated to a trusted port (ie.
    router port), the existing code won't remove the port from the
    Port Group so all traffic gets dropped.

    This patch fixes the issue by checking this condition in the
    update_port method of the ovn_client module.

    2. UT: Fix unit tests failures

    After 3316b45665a99b0f61e45a8c7facf538618861bf got merged in
    Neutron, our gate is blocked due to unit tests failing.
    After some debugging, it looks like no longer importing
    neutron.db.api is the culprit. This patch is importing it
    from our unit tests to unbreak the gate while the actual
    root case is determined.

    Change-Id: I3d30ec18f0c9df256c8f5846b52ab619835b5e32
    Closes-Bug: #1802373
    Partial-Bug: #1802369
    Signed-off-by: Daniel Alvarez <email address hidden>
    (cherry picked from commit 605ad27f053a1ef9ffb3601c6023987ddb6a22e1)