Microversion 2.47 punches nova in its special place

[Dan Smith]

Testing with 500 instances in ACTIVE, and 500 in ERROR state, using curl to pull all 1000 instances ten times in a row, 2.47 clearly shows a knee in the curve on average response time:

https://imgur.com/a/2lmiw

We should...fix that and stuff.

[Matt Riedemann (mriedem)]

We were always joining on flavor in the database:

https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L58

And we were always pulling the embedded instance.flavor:

https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L263

So it would probably be something in here:

https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L248

Could it be the policy check on all 1000 instances?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/507965

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/507948
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b1591d1080e4dbfbaf5351e009efa3abd11d88d4
Submitter: Jenkins
Branch: master

commit b1591d1080e4dbfbaf5351e009efa3abd11d88d4
Author: Dan Smith <email address hidden>
Date: Wed Sep 27 11:46:34 2017 -0700

    Fix policy check performance in 2.47+

    In 2.47 we introduced inlined flavors, which include extra_specs if so
    authorized. The code was doing a policy check deep in the per-instance
    handler that generated this blob. For queries with a lot of instances,
    this policy check is a lot of overhead, especially as it introduces a
    check of the policy file on disk, in case it has changed.

    This patch makes us calculate the permission flag once per list operation
    and pass it down to the lower layers to honor.

    Closes-Bug: #1719966
    Change-Id: I036623ae7409c2e6c6a754b4c8d5c9120f777774

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/507965
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bdd8b58351e9ebe69f1c5ab52bb5085e03be5202
Submitter: Jenkins
Branch: stable/pike

commit bdd8b58351e9ebe69f1c5ab52bb5085e03be5202
Author: Dan Smith <email address hidden>
Date: Wed Sep 27 11:46:34 2017 -0700

    Fix policy check performance in 2.47+

    In 2.47 we introduced inlined flavors, which include extra_specs if so
    authorized. The code was doing a policy check deep in the per-instance
    handler that generated this blob. For queries with a lot of instances,
    this policy check is a lot of overhead, especially as it introduces a
    check of the policy file on disk, in case it has changed.

    This patch makes us calculate the permission flag once per list operation
    and pass it down to the lower layers to honor.

    Closes-Bug: #1719966
    Change-Id: I036623ae7409c2e6c6a754b4c8d5c9120f777774
    (cherry picked from commit b1591d1080e4dbfbaf5351e009efa3abd11d88d4)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.2 release.