Logging on list operations slows down operation by about 500%

[Salvatore Orlando]

On list operations, every attribute for every item returned by the plugin is checked by the policy engine to see whether the user has rights to see it [*], assuming the attribute map specifies enforce_policy=True for that attribute

However, in most cases no policies are actually defined for those attributes; in this case the neutron server emits a log.debug statement for the specific attribute saying no policy was found.
This statement is then repeated for every attribute without policy times the number of items in the response.
Simple instrumenation shows list operations take in the API layer more than 500% the time they take in the plugin.
The time is wasted after the plugin call completes and more precisely while doing policy checks.

While other optimisations might be performed (such as not repeating checks already performed, and not triggering init() for each check), the biggest offender is the log statement.

Some measurements are available here: http://paste.openstack.org/show/75069/

* I believe this is wrong as well, but that's another story

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/85299

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/85299
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ef01fb553a141c012c8d8600edd6b709c5807111
Submitter: Jenkins
Branch: master

commit ef01fb553a141c012c8d8600edd6b709c5807111
Author: Salvatore Orlando <email address hidden>
Date: Fri Apr 4 03:30:12 2014 -0700

    Don't emit log for missing attribute check policy

    A missing policy for validating whether an attribute should
    be visible or not is a perfectly normal condition, and
    there is no value in logging it.

    Writing this information to the log is also having a
    considerable performance impact on list operations.

    Change-Id: Ie9d3ae6b249a5c49f6d81c1e56915174d8479a07
    Closes-Bug: 1302467

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/85512

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/85512
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3940b430f1e72b70a758d348fb5447f0fb0aa17f
Submitter: Jenkins
Branch: milestone-proposed

commit 3940b430f1e72b70a758d348fb5447f0fb0aa17f
Author: Salvatore Orlando <email address hidden>
Date: Fri Apr 4 03:30:12 2014 -0700

    Don't emit log for missing attribute check policy

    A missing policy for validating whether an attribute should
    be visible or not is a perfectly normal condition, and
    there is no value in logging it.

    Writing this information to the log is also having a
    considerable performance impact on list operations.

    Change-Id: Ie9d3ae6b249a5c49f6d81c1e56915174d8479a07
    Closes-Bug: 1302467
    (cherry picked from commit ef01fb553a141c012c8d8600edd6b709c5807111)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/85652

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/85652
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a76c7c4de9bd2f6d473d3bbf55570c16cc747c35
Submitter: Jenkins
Branch: stable/havana

commit a76c7c4de9bd2f6d473d3bbf55570c16cc747c35
Author: Salvatore Orlando <email address hidden>
Date: Fri Apr 4 03:30:12 2014 -0700

    Don't emit log for missing attribute check policy

    A missing policy for validating whether an attribute should
    be visible or not is a perfectly normal condition, and
    there is no value in logging it.

    Writing this information to the log is also having a
    considerable performance impact on list operations.

    Change-Id: Ie9d3ae6b249a5c49f6d81c1e56915174d8479a07
    Closes-Bug: 1302467
    (cherry picked from commit ef01fb553a141c012c8d8600edd6b709c5807111)