Additional port list / get_ports() failures when filtering and limiting at the same time

Bug #1827363 reported by Sebastian Lohff
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Gabriele Cerami

Bug Description

When doing a openstack port list that filters for a fixed-ip/subnet and at the same time limits the amount of results neutron returns a 500 internal server error. This was already addressed in https://bugs.launchpad.net/neutron/+bug/1826186 but this bug is also present in other places.

While running tempest against a Neutron Queens installation I came across another _get_ports_query() in neutron/plugins/ml2/plugin.py where filter is again called onto the result of an already limited query.

See https://github.com/openstack/neutron/blob/6f4962dcf89aebf2552ee8ec0993c6389a953024/neutron/plugins/ml2/plugin.py#L2206

InvalidRequestError: Query.filter() being called on a Query which already has LIMIT or OFFSET applied. To modify the row-limited results of a Query, call from_self() first. Otherwise, call filter() before limit() or offset() are applied.
  File "pecan/core.py", line 683, in __call__
    self.invoke_controller(controller, args, kwargs, state)
[...]
  File "neutron/db/db_base_plugin_v2.py", line 1417, in get_ports
    page_reverse=page_reverse)
  File "neutron/plugins/ml2/plugin.py", line 1941, in _get_ports_query
    query = query.filter(substr_filter)
  File "<string>", line 2, in filter
  File "sqlalchemy/orm/base.py", line 200, in generate
    assertion(self, fn.__name__)
  File "sqlalchemy/orm/query.py", line 435, in _no_limit_offset
    % (meth, meth)

I applied a patch similar to the one Gabriele Cerami proposed in https://review.opendev.org/#/c/656066/ on our production setup and this seems to have fixed the bug there as well.

When doing a grep for _get_ports_query() in the neutron codebase I find a function with this name being called in neutron/db/dvr_mac_db.py in get_ports_on_host_by_subnet(), I do not have a stacktrace or test for that though.

See https://github.com/openstack/neutron/blob/6f4962dcf89aebf2552ee8ec0993c6389a953024/neutron/db/dvr_mac_db.py#L162

description: updated
Changed in neutron:
status: New → Confirmed
importance: Undecided → Medium
Changed in neutron:
importance: Medium → High
Changed in neutron:
assignee: nobody → Gabriele Cerami (gcerami)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/658777

Changed in neutron:
status: Confirmed → In Progress
tags: added: neutron-proactive-backport-potential
Revision history for this message
Gabriele Cerami (gcerami) wrote :

In the second case, the filters are hard coded, and not limit is used. I think it's safe not to handle that.

In the first case, ironically, the only place that seems to use that _get_ports_query method is the function in the second case, and the method in ML2plugin is also calling its parent in db_base, handled in the first patch. So we are handling limit in many places even if the usage of that method seems so ... limited.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/658777
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=94bc403078b1e5d2cddc697fc59b7f1d2d8a6045
Submitter: Zuul
Branch: master

commit 94bc403078b1e5d2cddc697fc59b7f1d2d8a6045
Author: Gabriele Cerami <email address hidden>
Date: Mon May 13 10:38:50 2019 +0100

    ML2 plugin: extract and postpone limit in port query

    as with https://review.opendev.org/#/c/656066/ if limit is applied in
    any place other than at the end of the filters, sql alchemy will return
    an error, and possibily we could return less result than intended.

    Change-Id: I9a54ae99d2d5dfda63cb0061bcf3d727ed7cc992
    Closes-Bug: #1827363

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/680665

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/680666

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/680666
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e3133d8ae9d5fdd94da024343ba8edde7cb4b63c
Submitter: Zuul
Branch: stable/rocky

commit e3133d8ae9d5fdd94da024343ba8edde7cb4b63c
Author: Gabriele Cerami <email address hidden>
Date: Mon May 13 10:38:50 2019 +0100

    ML2 plugin: extract and postpone limit in port query

    as with https://review.opendev.org/#/c/656066/ if limit is applied in
    any place other than at the end of the filters, sql alchemy will return
    an error, and possibily we could return less result than intended.

    Change-Id: I9a54ae99d2d5dfda63cb0061bcf3d727ed7cc992
    Closes-Bug: #1827363
    (cherry picked from commit 94bc403078b1e5d2cddc697fc59b7f1d2d8a6045)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/680665
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=677d1f5d03718b669b03f4d8c17f75ae00109304
Submitter: Zuul
Branch: stable/stein

commit 677d1f5d03718b669b03f4d8c17f75ae00109304
Author: Gabriele Cerami <email address hidden>
Date: Mon May 13 10:38:50 2019 +0100

    ML2 plugin: extract and postpone limit in port query

    as with https://review.opendev.org/#/c/656066/ if limit is applied in
    any place other than at the end of the filters, sql alchemy will return
    an error, and possibily we could return less result than intended.

    Change-Id: I9a54ae99d2d5dfda63cb0061bcf3d727ed7cc992
    Closes-Bug: #1827363
    (cherry picked from commit 94bc403078b1e5d2cddc697fc59b7f1d2d8a6045)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 15.0.0.0b1

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 14.0.3

This issue was fixed in the openstack/neutron 14.0.3 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 13.0.5

This issue was fixed in the openstack/neutron 13.0.5 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.