[RFE] Support sub-string matching when filtering port with IP address

[Zhenyu Zheng]

Problem
=======
The GET /ports API only support exact matching when filtering by IP address.
Providing support for more powerful regex matching filtering could be very
useful.

Usecase
=======
Nova allows filtering instance by IP address and it is in regex matching
manner, but it now has serious performance issue [1]. How to enhance the
performance of this filter has been discussed in Queens PTG [2]. When
filtering instance by IP addresses, first calling Neutron port list API
with IP address filtering, and then retrieve the instance UUID in Nova
side could be a possible solution to improve the performance.

As Nova allows regex manner filtering, it could be better to make Neutron
also support regex matching.

Proposal
========
Adding regex matching support in DB layer using "LIKE" SQL phrase.

References
==========
[1] https://bugs.launchpad.net/nova/+bug/1711303
[2] http://lists.openstack.org/pipermail/openstack-dev/2017-September/122258.html

[Armando Migliaccio (armando-migliaccio)]

This sounds reasonable. So long as we make sure to sanitize the query to avoid SQL injection attacks, I am fine with this :)

[Armando Migliaccio (armando-migliaccio)]

Let's bring this to the attention of the drivers team.

[Matt Riedemann (mriedem)]

Will a new API extension be added to Neutron to show whether or not this capability is available? Just thinking from the nova-api as client perspective, we'll need to know if we can filter ports by IP regex or not. Otherwise nova-api listing ports with an IP regex filter on an older neutron installation won't return anything and nova will return nothing which would be potentially wrong.

[Miguel Lavalle (minsel)]

During the latest drivers meeting (http://eavesdrop.openstack.org/meetings/neutron_drivers/2017/neutron_drivers.2017-10-19-22.00.log.html) it was realized that Nova supports a full regex that is not compatible with with a LIKE query to the DB. It was agreed that the next step is to ask in the operators and development mailing list how the IP filtering is used in Nova, whether is is actually used as a full regex or rather as sub-strings

[Matt Riedemann (mriedem)]

http://lists.openstack.org/pipermail/openstack-operators/2017-October/014459.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/521683

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/525284

[Miguel Lavalle (minsel)]

I spoke to Matt Riedemann today about this. The conclusion from the ML thread http://lists.openstack.org/pipermail/openstack-operators/2017-October/014459.html is that Nova will be happy with Neutron supporting sub-string matching, not full blown regex. POCs have been submitted in Neutron (https://review.openstack.org/#/c/521683) and Nova (https://review.openstack.org/#/c/525505/) with sub-string matching. Tests were conducted (by Huawei) without and with these patches deployed. 2000 VMs and their corresponding ports (1 per instance) were created and then the instances are filtered using a specific ip address:

GET compute/v2.1/servers/details?all_tenants=1&ip=192.168.1.254

Without the POC patches, this query takes 4000ms. With the patches, the time goes down to 400ms. With substring:

GET compute/v2.1/servers/details?all_tenants=1&ip=192.168.1.2

It took 900ms

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/525284
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=7b43461c7abd8ecd486b01b1575b1c9ee7b264a6
Submitter: Zuul
Branch: master

commit 7b43461c7abd8ecd486b01b1575b1c9ee7b264a6
Author: Hongbin Lu <email address hidden>
Date: Mon Dec 4 19:18:09 2017 +0000

    Add the IP substring filter extension

    Change-Id: I97259b85a2dce5a54bb6ea2cb9d9779ec0a25504
    Related-Bug: #1718605

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/521683
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5c601bebeb01876f1ea744e0ae95f83d1c30120d
Submitter: Zuul
Branch: master

commit 5c601bebeb01876f1ea744e0ae95f83d1c30120d
Author: Hongbin Lu <email address hidden>
Date: Mon Nov 20 23:19:31 2017 +0000

    Support filtering port with IP address substring

    Neutron currently supports filtering ports by matching the exact
    IP address. This patch adds support for substring matching using
    "LIKE" SQL operator.

    This patch also added a new API extension to show whether or not
    the substring matching capability is available.

    APIImpact add IP address substring filtering on listing ports
    API-ref: I97259b85a2dce5a54bb6ea2cb9d9779ec0a25504

    Co-Authored-By: Zhenyu Zheng <email address hidden>

    Change-Id: I9549b2ba676e1bad0812682c3f3f3c97de15f5f6
    Closes-Bug: #1718605

[OpenStack Infra (hudson-openstack)]

Change abandoned by Zhenyu Zheng (<email address hidden>) on branch: master
Review: https://review.openstack.org/518865
Reason: Do not need this anymore

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.0.0b3 development milestone.