Enable sorting and pagination by default

[Ihar Hrachyshka]

Currently those features are controlled by configuration options: allow_sorting, allow_pagination, and they are disabled by default. There are multiple issues with that:

- those useful API features are not available for any default installation of neutron;
- it's not great when API behaviour is not consistent, depending on local configuration;
- we don't have a way of detecting whether those features are enabled.

Base controller already supports both native and generic implementations for those features: if a plugin claims native support, then plugin calls are populated with corresponding sorting/pagination parameters; otherwise the base controller 'emulates' those features for the plugin. It seems that this fallback approach already covers all cases, and we should be safe to enable those features for all setups.

We need to make sure that testing coverage for the features is adequate (API tests), that we test it in gate; then we should consider enabling the features by default, deprecating those options and eventually removing them.

[Henry Gessau (gessau)]

I doubt this needs an RFE but we can discuss it.

[Ihar Hrachyshka (ihar-hrachyshka)]

If you think it's not worth and RFE, let's just move it to Wishlist. I need a bug to attach the work I plan to enable it.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/306272

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: master
Review: https://review.openstack.org/301634
Reason: Included in another patch.

[Ihar Hrachyshka (ihar-hrachyshka)]

It blocks some work for oslo.versionedobjects adoption for 1st class resources. We need to introduce extensive API layer testing for those features before we are able to switch resources like subnet or network to using the objects, to avoid regressions.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/306272
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0cf4ddc6ee35250fdfbce5f19bc62b8905f10fea
Submitter: Jenkins
Branch: master

commit 0cf4ddc6ee35250fdfbce5f19bc62b8905f10fea
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 5 15:54:33 2016 +0200

    tests: validate sorting and pagination for networks

    These are the very first API tests in the tree to cover those features.
    The test is implemented in a generic way that will hopefully ease its
    adoption for other resources.

    Sadly, those tests cannot pass on every neutron cloud, at least until we
    enable those API features by default and remove options to disable the
    features.

    There is no way to determine, via neutron API, whether cloud supports
    those features. To work around that, we introduce two new tempest
    options that determine whether tests that rely on sorting or pagination
    should be executed. Those options are set in post-extra phase because
    configure_tempest resets any configuration made during post-config.

    Also bump resource quotas x10 times since default quotas are now not
    enough, at least for network resource that is now under sorting and
    pagination testing.

    Related-Bug: #1566514
    Change-Id: I5e68f471a641a34100aba31cb2c4a815c7220014

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/320980

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/320980
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=99e9d5c0efc46729a6f14f66a8845305b7667876
Submitter: Jenkins
Branch: master

commit 99e9d5c0efc46729a6f14f66a8845305b7667876
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 24 14:50:58 2016 +0200

    tests: cover port with existing sorting/pagination api tests

    This is based on existing network api tests for the features that were
    implemented in an extensible way.

    Existing list_kwargs is decoupled from the base test class because they
    are network test specific. Also removed generic handling for
    create_method because each resource has its own signature for creation
    methods, so it's easier to implement it per resource than trying to come
    up with a generic solution.

    Related-Bug: #1566514
    Change-Id: I648851b48d0481c97054e1280b60a119b42dfd38

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/328273

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/329340

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/329350

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/329351

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/329474

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/329481

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/330058

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/329340
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e3b0803152766699a8d72dbb31fc273397d4495d
Submitter: Jenkins
Branch: master

commit e3b0803152766699a8d72dbb31fc273397d4495d
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Jun 14 11:40:19 2016 +0200

    Added sorting/pagination tests for subnets

    Change-Id: Ic04cd125dd715ae0694fbf24d8193abe3151ffd1
    Related-Bug: #1566514

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/330482

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/329350
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e923457403f08182096dbcf1235e291b8df5a71a
Submitter: Jenkins
Branch: master

commit e923457403f08182096dbcf1235e291b8df5a71a
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Jun 14 11:51:37 2016 +0200

    Added sorting/pagination tests for routers

    Change-Id: Ib3709d15fd87b93285dadf78cce73ee37e8898e2
    Depends-On: I8b553ab22846122dde22372f9901e46a5276ee8e
    Related-Bug: #1566514

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/328259
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6809a1aa24652689c32874673d596feb73030505
Submitter: Jenkins
Branch: master

commit 6809a1aa24652689c32874673d596feb73030505
Author: Ihar Hrachyshka <email address hidden>
Date: Fri Jun 10 13:44:22 2016 +0200

    qos: added api sorting/pagination tests for policies

    API tests were modified to support admin listing mode. This is needed
    for QoS resources because the default policy file does not allow to
    create policies, and we should avoid using shared policies due to races.

    Related-Bug: #1566514
    Depends-On: I8b553ab22846122dde22372f9901e46a5276ee8e
    Change-Id: Ie3a66d2a7c05143df3da1c56c8e8a5308d4bae91

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/328273
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0144c67d98f87605bfa3232a2b0df711804704be
Submitter: Jenkins
Branch: master

commit 0144c67d98f87605bfa3232a2b0df711804704be
Author: Ihar Hrachyshka <email address hidden>
Date: Fri Jun 10 14:44:36 2016 +0200

    qos: support native sorting/pagination for policies and rules

    Tests are already present to validate that no regression happens here,
    committed as part of Ie3a66d2a7c05143df3da1c56c8e8a5308d4bae91

    Change-Id: I85895ea3f2475433942fcdc1c13b211cff3dd477
    Related-Bug: #1566514

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/329351
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0cc77d24a34096f65ff7b1860bb085cf21ffa0e4
Submitter: Jenkins
Branch: master

commit 0cc77d24a34096f65ff7b1860bb085cf21ffa0e4
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Jun 14 11:52:26 2016 +0200

    l3: support native sorting/pagination for routers

    Tests are already present to validate that no regression happens here,
    committed as part of Ib3709d15fd87b93285dadf78cce73ee37e8898e2

    Change-Id: I4cdb1c4b8ee233d9d5bc78b0d3bd2ce28e725535
    Related-Bug: #1566514

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/329474
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5e0878f476d7116bc4e0c29c7929718f379b36c2
Submitter: Jenkins
Branch: master

commit 5e0878f476d7116bc4e0c29c7929718f379b36c2
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Jun 14 16:16:06 2016 +0200

    Added API extensions to detect sorting/pagination features

    Those features are available only when allow_sorting and
    allow_pagination options are enabled (the current default is False).

    They don't depend on plugin support, because when plugins don't
    implement them natively, emulated mode is applied by API router itself.
    So to make it plugin agnostic, we introduce a way to register custom
    per-extension checks to override support detection for cases like that
    one.

    Now that we have a way to detect support for those features via API,
    there is little reason to keep tempest configuration options to enable
    those features. Instead, just inspect [network-feature-enabled]
    api_extensions option in tempest.conf.

    Now that DEFAULT_ALLOW_SORTING/DEFAULT_ALLOW_PAGINATION constants are
    used in a single place only (in allow_sorting/allow_pagination
    definitions), removed them and replaced with a literal.

    Added first in-tree API tests for /extensions entry point.

    DocImpact Update API documentation to cover new extensions.
    APIImpact Document the new extensions.
    Related-Bug: #1566514
    Change-Id: I0aaaa037a8ad52060a68dd75c0a1accc6add238e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/329481
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9733cc25522b1cef83d090617a425efc5364d35b
Submitter: Jenkins
Branch: master

commit 9733cc25522b1cef83d090617a425efc5364d35b
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Jun 14 16:27:51 2016 +0200

    Enable sorting and pagination by default

    Now that we have a decent api test coverage for those features, as well
    as a way to detect the features via REST API, we should be safe and
    thrilled to enable them by default in all installations.

    Depends-On: I0aaaa037a8ad52060a68dd75c0a1accc6add238e
    Depends-On: I5e68f471a641a34100aba31cb2c4a815c7220014
    Depends-On: I648851b48d0481c97054e1280b60a119b42dfd38
    Depends-On: Ic04cd125dd715ae0694fbf24d8193abe3151ffd1
    Depends-On: Ib3709d15fd87b93285dadf78cce73ee37e8898e2
    Depends-On: Ie3a66d2a7c05143df3da1c56c8e8a5308d4bae91

    DocImpact Update configuration documentation to cover new defaults.
    UpgradeImpact To retain the original default behaviour, set
                  allow_sorting and allow_pagination to False.

    Closes-Bug: #1566514
    Change-Id: I3820bec029b3895913b034b718116c95b4942ed0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/330058
Committed: https://git.openstack.org/cgit/openstack/neutron-specs/commit/?id=2fe778edfd27ebd472c350a775c611dca55afef2
Submitter: Jenkins
Branch: master

commit 2fe778edfd27ebd472c350a775c611dca55afef2
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Jun 15 18:32:07 2016 +0200

    Document sorting/pagination extension existance

    We now have a way to detect if those features are available to API
    users, by checking if 'sorting' and 'pagination' extensions are present
    when fetching from /extensions API endpoint.

    Change-Id: I2f64a3f1dc34d9ffbc450ef1670ad5d2889cd7c9
    Related-Bug: #1566514
    Depends-On: I0aaaa037a8ad52060a68dd75c0a1accc6add238e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/330482
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fdf66024d38276f6eed111973127e1008ca68326
Submitter: Jenkins
Branch: master

commit fdf66024d38276f6eed111973127e1008ca68326
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Jun 16 13:41:23 2016 +0200

    Deprecate allow_sorting and allow_pagination options

    There is no technical reason to disable sorting and pagination, except
    maybe performance concerns. But there are other ways to DoS-attack
    neutron-server services, like... any other type of request. So the
    concern is moot.

    Keeping those options available for configuration reduces compatibility
    between Neutron installations. It's better to do the right thing, and do
    it right, implementing the feature for the benefit of API users.

    DocImpact Update configuration documentation to reflect deprecation.

    Change-Id: Iad26221a58589b10381f1a0d69a88c92fb5adeee
    Related-Bug: #1566514

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 9.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/404731

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/404731
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=61eb74af31125c7cb68f2d198602b15e505b2ee3
Submitter: Jenkins
Branch: master

commit 61eb74af31125c7cb68f2d198602b15e505b2ee3
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Nov 29 06:34:59 2016 +0000

    Remove allow_pagination and allow_sorting config options

    They were deprecated in Newton. This patch cleans them up.

    Note: it does not mean that the features will not work anymore. On the
    contrary, now API will consistently sort and paginate for plugins that
    honour the relevant sorting/pagination parameters.

    Note2: base resource controller still allows to pass
    allow_pagination=False and allow_sorting=False parameters to disable the
    features if a registered plugin does not support the features yet.

    Change-Id: I5fd30b20f645846d9366740372c4815c4b33e2eb
    Related-Bug: #1566514