Extensions unit tests failing after neutron change

[Bernard Cafarelli]

Recent gate runs fail on unit tests, example:
http://logs.openstack.org/54/449754/3/check/gate-networking-sfc-python35/86e22ce/console.html#_2017-03-27_17_35_01_740640

networking_sfc.tests.unit.extensions.test_sfc.SfcExtensionTestCase.test_create_port_chain_none_chain_parameters
Captured traceback:
~~~~~~~~~~~~~~~~~~~
    b'Traceback (most recent call last):'
    b' File "/tmp/openstack/neutron/neutron/tests/base.py", line 116, in func'
    b' return f(self, *args, **kwargs)'
    b' File "/home/jenkins/workspace/gate-networking-sfc-python35/networking_sfc/tests/unit/extensions/test_sfc.py", line 130, in test_create_port_chain_none_chain_parameters'
    b' self._test_create_port_chain(chain_parameters=None)'
    b' File "/home/jenkins/workspace/gate-networking-sfc-python35/networking_sfc/tests/unit/extensions/test_sfc.py", line 103, in _test_create_port_chain'
    b" content_type='application/%s' % self.fmt)"
    b' File "/home/jenkins/workspace/gate-networking-sfc-python35/.tox/py35/lib/python3.5/site-packages/webtest/app.py", line 378, in post'
    b' content_type=content_type)'
    b' File "/home/jenkins/workspace/gate-networking-sfc-python35/.tox/py35/lib/python3.5/site-packages/webtest/app.py", line 747, in _gen_request'
    b' expect_errors=expect_errors)'
    b' File "/home/jenkins/workspace/gate-networking-sfc-python35/.tox/py35/lib/python3.5/site-packages/webtest/app.py", line 643, in do_request'
    b' self._check_status(status, res)'
    b' File "/home/jenkins/workspace/gate-networking-sfc-python35/.tox/py35/lib/python3.5/site-packages/webtest/app.py", line 675, in _check_status'
    b' res)'
    b'webtest.app.AppError: Bad response: 500 Internal Server Error (not 200 OK or 3xx redirect for http://localhost/sfc/port_chains.json)'
    b'b\'{"NeutronError": {"message": "Request Failed: internal server error while processing your request.", "type": "HTTPInternalServerError", "detail": ""}}\''
    b''

After some research and bisect, these are broken since 93d8376c44cdba58b1db88137b3dcb3a1ad1b2e6 (OVO for Quotas and Reservation ) landed in Neutron:
https://github.com/openstack/neutron/commit/93d8376c44cdba58b1db88137b3dcb3a1ad1b2e6

Fast way to reproduce on single failing test:
tox -e py27 networking_sfc.tests.unit.extensions.test_flowclassifier.FlowClassifierExtensionTestCase.test_create_flow_classifier_all_fields

[Thomas Morin (tmmorin-orange)]

Here is a clean traceback of a failed bgpvpn create test (after commenting out the retry decorators):

This is with networking_bgpvpn.tests.unit.extensions.test_bgpvpn.BgpvpnExtensionTestCase.test_bgpvpn_create_with_valid_route_targets .

Traceback (most recent call last):
  File "/home/teom7365/prog/openstack/networking-bgpvpn/.tmp/tmp.FxNzLbmRqx/openstack/neutron/neutron/api/v2/resource.py", line 79, in resource
    result = method(request=request, **args)
  File "/home/teom7365/prog/openstack/networking-bgpvpn/.tmp/tmp.FxNzLbmRqx/openstack/neutron/neutron/api/v2/base.py", line 438, in create
    return self._create(request, body, **kwargs)
  File "/home/teom7365/prog/openstack/networking-bgpvpn/.tmp/tmp.FxNzLbmRqx/openstack/neutron/neutron/api/v2/base.py", line 482, in _create
    self._plugin)
  File "/home/teom7365/prog/openstack/networking-bgpvpn/.tmp/tmp.FxNzLbmRqx/openstack/neutron/neutron/quota/__init__.py", line 232, in make_reservation
    context,
  File "/home/teom7365/prog/openstack/networking-bgpvpn/.tmp/tmp.FxNzLbmRqx/openstack/neutron/neutron/db/quota/driver.py", line 176, in make_reservation
    context, resources, tenant_id)
  File "/home/teom7365/prog/openstack/networking-bgpvpn/.tmp/tmp.FxNzLbmRqx/openstack/neutron/neutron/db/quota/driver.py", line 68, in get_tenant_quotas
    quota_objs = quota_obj.Quota.get_objects(context, project_id=tenant_id)
  File "/home/teom7365/prog/openstack/networking-bgpvpn/.tmp/tmp.FxNzLbmRqx/openstack/neutron/neutron/objects/base.py", line 449, in get_objects
    **cls.modify_fields_to_db(kwargs)
  File "/home/teom7365/prog/openstack/networking-bgpvpn/.tmp/tmp.FxNzLbmRqx/openstack/neutron/neutron/objects/db/api.py", line 49, in get_objects
    return plugin._get_collection(
  File "/home/teom7365/prog/openstack/networking-bgpvpn/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 698, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute '_get_collection'

The issue arises because the core plugin is actually a NonCallableMagicMock :

> /home/teom7365/prog/openstack/networking-bgpvpn/.tmp/tmp.FxNzLbmRqx/openstack/neutron/neutron/objects/db/api.py(46)get_objects()
-> filters = _kwargs_to_filters(**kwargs)
(Pdb) l
 41 for k, v in kwargs.items()}
 42
 43
 44 def get_objects(context, model, _pager=None, **kwargs):
 45 with context.session.begin(subtransactions=True):
 46 -> filters = _kwargs_to_filters(**kwargs)
 47 # TODO(ihrachys): decompose _get_collection from plugin instance
 48 plugin = directory.get_plugin()
 49 return plugin._get_collection(
 50 context, model,
 51 dict_func=None, # return all the data
(Pdb) directory.get_plugin()
<weakproxy at 0x7f97cfeeff18 to NonCallableMagicMock at 0x7f97cff7ff90>

I suspect that we aren't using the unit test framework in a way that creates a real core plugin, but I haven't investigated further yet.

A surprising thing is that this test, run with "python -m testtools.run networking_bgpvpn.tests.unit.extensions.test_bgpvpn.BgpvpnExtensionTestCase.test_bgpvpn_create_with_va...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/450865

[Thomas Morin (tmmorin-orange)]

I have this workaround for bgpvpn: https://review.openstack.org/450865

[OpenStack Infra (hudson-openstack)]

Change abandoned by Thomas Morin (<email address hidden>) on branch: master
Review: https://review.openstack.org/450865
Reason: happens in parallel with another gate breakage => wrap into I8570441a0b8d5ee3ad7f88e07affac2f1b782021

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/450873

[Bernard Cafarelli (bcafarel)]

Enabling quotas in extensions unit test make them pass again (review sent to test the gate).

We do have quotas in both extensions. But tests worked with use_quota=False (the default setting) before

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/425933
Committed: https://git.openstack.org/cgit/openstack/networking-bgpvpn/commit/?id=2bbebda405b6a504c07825f8681011de9698dc4d
Submitter: Jenkins
Branch: master

commit 2bbebda405b6a504c07825f8681011de9698dc4d
Author: Thomas Morin <email address hidden>
Date: Thu Jan 26 22:13:26 2017 +0100

    bagpipe driver: cleanup, remove OVSInterceptBridge

    With I55835a34d8fba7a139dce93f99cbff54584d695c, the OVSBridgeIntercept
    workaround can now be removed.

    This change also includes a workaround for a gate issue:
    It seems that enabling the use of quotas works around bug
    1676656, by triggering the use of neutron.quota.ConfDriver which
    does not trigger the bug.

    Change-Id: I8570441a0b8d5ee3ad7f88e07affac2f1b782021
    Related-Bug: 1676656

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/450923

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/450923
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bae6555816d7b5784a09bf261958025c52df433f
Submitter: Jenkins
Branch: master

commit bae6555816d7b5784a09bf261958025c52df433f
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Mar 28 01:57:51 2017 +0000

    ExtensionTestCase: register correct core plugin

    We should register the passed plugin as core only if service_type is
    False. Otherwise, the passed plugin is a service plugin, in which case
    we should register some other core plugin for the test. The easiest
    thing is to use the base db plugin for that matter.

    This fixes subproject unit tests because now the core plugin will always
    have _get_collection function that is needed by some underlying db api
    functions.

    Change-Id: Ica421831763eb2f5d25fc068287305b402227b53
    Closes-Bug: #1676656

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bernard Cafarelli (<email address hidden>) on branch: master
Review: https://review.openstack.org/450873
Reason: For this specific failure, fixed in neutron with https://review.openstack.org/#/c/450923/
For the OVSCookieBridge, review in progress at https://review.openstack.org/#/c/450463/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/450924
Committed: https://git.openstack.org/cgit/openstack/networking-bgpvpn/commit/?id=4a3801273245bef1f062794f4f6d706d68ee81dc
Submitter: Jenkins
Branch: master

commit 4a3801273245bef1f062794f4f6d706d68ee81dc
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Mar 28 02:01:00 2017 +0000

    Stop using config backed quota engine in unit tests

    It was enabled as part of gate fix
    I8570441a0b8d5ee3ad7f88e07affac2f1b782021 but the right fix for the
    issue is to register a proper core plugin for each test (see the
    dependency).

    Related-Bug: #1676656
    Change-Id: I2eb49a5371f5c9ea0ef4a50e6d95667b782f8842
    Depends-On: Ica421831763eb2f5d25fc068287305b402227b53

[Bernard Cafarelli (bcafarel)]

Fixed in neutron

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.0.0b1 development milestone.