neutron agent. list API lacks sort and page feature

[yong sheng gong]

We have hundreds of neutron agents deployed, but currently, the agent list API does not support sorting and paging, which is very bad.

[Slawek Kaplonski (slaweq)]

What version are You using? I just checked on RHOSP-16 which is based on stable/train and the result is:

(overcloud) [stack@undercloud-0 ~]$ neutron agent-list --sort-column agent_type
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------+---------------------------+-------------------+-------+----------------+---------------------------+
| id | agent_type | host | availability_zone | alive | admin_state_up | binary |
+--------------------------------------+--------------------+---------------------------+-------------------+-------+----------------+---------------------------+
| 643a494d-ad06-4455-9c6b-6ad35f7b2f5e | DHCP agent | controller-2.redhat.local | nova | :-) | True | neutron-dhcp-agent |
| 8a092f5b-f415-4473-ac8e-9ca333b037bc | DHCP agent | controller-0.redhat.local | nova | :-) | True | neutron-dhcp-agent |
| 8a384ede-4ccb-4321-b170-2448344b92fd | DHCP agent | controller-1.redhat.local | nova | :-) | True | neutron-dhcp-agent |
| 2ec9c0d6-57d8-4803-9b44-942542e53d6e | L3 agent | controller-0.redhat.local | nova | :-) | True | neutron-l3-agent |
| 758c19ef-0bb3-48bb-a757-90956b830914 | L3 agent | controller-1.redhat.local | nova | :-) | True | neutron-l3-agent |
| 9cec9036-0d0f-432a-a0ea-4420f1e88921 | L3 agent | controller-2.redhat.local | nova | :-) | True | neutron-l3-agent |
| b0d7fb33-607a-4903-9bf3-987743f237a7 | L3 agent | compute-1.redhat.local | nova | :-) | True | neutron-l3-agent |
| bf84115b-fc33-41c6-8734-2cbe1c9a7a5c | L3 agent | compute-0.redhat.local | nova | :-) | True | neutron-l3-agent |
| 386111f1-563c-4a2c-9b41-b292a1aa876e | Metadata agent | compute-0.redhat.local | | :-) | True | neutron-metadata-agent |
| 43d33cd1-c817-4755-8628-e759402acc5b | Metadata agent | controller-0.redhat.local | | :-) | True | neutron-metadata-agent |
| ad765ce7-f2e2-4db3-8ff7-1765e5e5bb93 | Metadata agent | controller-2.redhat.local | | :-) | True | neutron-metadata-agent |
| c832081f-d598-483c-bb57-4f616e631fbb | Metadata agent | controller-1.redhat.local | | :-) | True | neutron-metadata-agent |
| dbd82bdb-43d3-46de-99bc-a8364f260457 | Metadata agent | compute-1.redhat.local | | :-) | True | neutron-metadata-agent |
| 274076b0-56b1-407a-9903-2a6529d6be77 | Open vSwitch agent | compute-1.redhat.local | | :-) | True | neutron-openvswitch-agent |
| 98ed6974-b379-4eb6-8267-d76cd38f9379 | Open vSwitch agent | controller-0.redhat.local | | :-) | True | neutron-openvswitch-...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/726497

[yangkai (yangkaiopenstack)]

Check on stable/train and the result is:

`neutron agent-list --sort-column binary` is sorted by neutronclient

`neutron agent-list --sort-key binnary` will be KO

DEBUG: keystoneauth.session REQ: curl -g -i -X GET http://overcloud.internal.vim1.local:9696/v2.0/agents?sort_key=binnary&sort_dir=asc -H "Accept:
 application/json" -H "User-Agent: python-neutronclient" -H "X-Auth-Token: {SHA256}186f3682c2ce0e17e2e3e74077285d319e5347ef6fca7fb78ccd6554fd21473
e"

DEBUG: neutronclient.v2_0.client Error message: {"NeutronError": {"message": "[u'binnary'] is invalid attribute for sort_keys", "type": "HTTPBadRequest", "detail": ""}}

So we need implement it in neutron-server

[Miguel Lavalle (minsel)]

I am testing in near master (built it about 6 weeks ago). I can see that the API behavior is inconsistent. If I do:

$ neutron net-list --sort_key name --sort_dir asc

or

$ neutron net-list --sort_key name --sort_dir desc

I get the networks sorted by name correctly, as indicated by the direction argument

If I do:

$ neutron net-list --sort_key name --debug

I get:

DEBUG: keystoneauth.session RESP: [400] Connection: keep-alive Content-Length: 123 Content-Type: application/json Date: Sun, 10 May 2020 17:42:41 GMT X-Openstack-Request-Id: req-3aa8373d-1d00-4a89-b5a5-a14782aa3a46
DEBUG: keystoneauth.session RESP BODY: {"NeutronError": {"type": "HTTPBadRequest", "message": "The number of sort_keys and sort_dirs must be same", "detail": ""}}
DEBUG: keystoneauth.session GET call to network for http://192.168.33.12:9696/v2.0/networks?sort_key=name used request id req-3aa8373d-1d00-4a89-b5a5-a14782aa3a46

Now, in the case of:

$ neutron agent-list --sort_key binary --sort_dir asc

$ neutron agent-list --sort_key binary --sort_dir desc

$ neutron agent-list --sort_key binary

$ neutron agent-list --sort_key binnary

$ neutron agent-list --sort_key binnary --sort_dir asc

I get the exam same list all the time. So the API is just ignoring the sort arguments.

This bug is related to https://bugs.launchpad.net/neutron/+bug/1749820, which was marked as closed. So let's keep this one and continue the conversation here

[Miguel Lavalle (minsel)]

Here's previous work related to this bug:

https://review.opendev.org/#/c/567033/
https://review.opendev.org/#/c/554368/
https://review.opendev.org/#/c/581870/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/726724

[Slawek Kaplonski (slaweq)]

I marked this bug as low-hanging-fruit bug as IMO it is something which should be easy to fix. If someone wants to work on it, feel free to join #openstack-neutron channel @freenode and ask about any help You need (You can ping e.g. slaweq or mlavalle there).
Also I think that https://github.com/openstack/neutron/blob/master/neutron/db/agents_db.py#L298 is probably place to start looking for a fix as it probably should accepts also sorts, limits, marker and page_reverse parameters.

[Slawek Kaplonski (slaweq)]

Related patch is proposed https://review.opendev.org/#/c/726724/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/738471

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/738475

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738471
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=5b035972e567e1426657218c52b0ebc06860d3ef
Submitter: Zuul
Branch: master

commit 5b035972e567e1426657218c52b0ebc06860d3ef
Author: vinay_m <email address hidden>
Date: Mon Jun 29 21:22:54 2020 +0530

    Neutron agent-list API added with sort_key keyword

    Defining a API Extention for enabling sorting in agent
    -list attributes.

    Added "is_sort_key" keyword for attributes of agent.
    Neutron API can use "sort_key" and "sort_dir" for sorting on resulted
    list.

    Change-Id: I3155a4f1de664e4ff8e32ccc4314927ea462048c
    Related-Bug: #1877254

[OpenStack Infra (hudson-openstack)]

Change abandoned by vinay harsha mitta (<email address hidden>) on branch: master
Review: https://review.opendev.org/738475

[Abhishek Mahajan (mahajan-abhishek)]

Related fix proposed to branch: master
https://review.opendev.org/#/c/743953/

[Slawek Kaplonski (slaweq)]

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/743953
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/791044

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/726497
Reason: This patch has not been attended in 1 year, if needed, you can restore it again.

[Heather Lemon (hypothetical-lemon)]

I am currently still working on this. There are failing zuul tests that need fixed first.
https://review.opendev.org/c/openstack/neutron/+/791044

[Slawek Kaplonski (slaweq)]

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/743953
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Slawek Kaplonski (slaweq)]

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/791044
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/726724
Reason: Depends-on patch was abandoned. Please feel free to restore this patch and the neutron-lib one.