/v2.1/servers/detail?tenant_id=XX returns unpredictable results

[Jordan Pittier]

Found this while investigating why the tempest test test_list_servers_by_admin_with_specified_tenant randomly fails on Py35.

This test produces the following API call:

GET /v2.1/servers/detail?tenant_id=XXX

Which calls this method in Nova:

nova.api.openstack.compute.servers.ServersController.detail()

In the _get_servers() method that detail() calls, for some reason (bug ? [1]), the 'project_id' is added to the search options. So now, we see that get_all() is called with both tenant_id and project_id:

Searching by: {'project_id': '74e1044b53de44d1bac80cded5146504', 'deleted': False, 'tenant_id': '153d2038e0bc4ea99819a21a55cb66ea'} get_all /opt/stack/new/nova/nova/compute/api.py:2336

Now, in nova/nova/compute/api.py in the get_all() method, there's a dict called filter_mapping, that is iterated upon. I believe, based on which order the dict (i.e the search options) is iterated upon, either tenant_id or project_id gets rewritten. This leads to random return values.

[1] : https://github.com/openstack/nova/blob/cba26a6e561c18fa4659efac8ddc0b3c139023fe/nova/api/openstack/compute/servers.py#L322

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/426259

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/426259
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f702762934e0a46449091243a39ad5d57108b23f
Submitter: Jenkins
Branch: master

commit f702762934e0a46449091243a39ad5d57108b23f
Author: Jordan Pittier <email address hidden>
Date: Fri Jan 27 15:22:25 2017 +0100

    Fix unspecified bahavior on GET /servers/detail?tenant_id=X as admin

    When an admin calls /v2.1/servers/detail?tenant_id=XX, then the
    `get_all` method of nova.compute.api.API is called with 2
    conflicting search options: {'tenant_id': XX, 'project_id': YY}.
    But because, latter on, in that `get_all` method we define a dict
    called filter_mapping, on which we iter upon, which value takes
    precedence depends on the order in which the dict is iterated upon.

    As the order in unpredictable and varies between Py2 and Py3, this
    is problematic. Especially, on Python 2 hash randomization is
    disabled by default but it's enabled by default on Py3 (unless
    the PYTHONHASHSEED env var is set to a fixed value) [0]

    The (unreliable) order we iterate on items of that `filter_mapping`
    dict is why the Tempest test_list_servers_by_admin_with_specified_tenant
    test randomly fails on Py35.

    This patch ensures that, if the all_tenants search option is
    not set, then the `tenant_id` search option is ignored. Note that
    this *is* the current behavior on Py27, as documented in lp:#1185290
    and tested by Tempest here [1].

    [0] https://docs.python.org/dev/using/cmdline.html#cmdoption-R
    [1] https://github.com/openstack/tempest/blob/fe1a8e289c2d79df29beaa6b3603afe5feb60fb3/tempest/api/compute/admin/test_servers.py#L96

    Change-Id: I1a74ef0f16da14444029c0f184b433df367ffb41
    Closes-Bug: #1659811

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/430504

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/430504
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1c992c7a172af0890c9fea147e018257ee018f2d
Submitter: Jenkins
Branch: stable/ocata

commit 1c992c7a172af0890c9fea147e018257ee018f2d
Author: Jordan Pittier <email address hidden>
Date: Fri Jan 27 15:22:25 2017 +0100

    Fix unspecified bahavior on GET /servers/detail?tenant_id=X as admin

    When an admin calls /v2.1/servers/detail?tenant_id=XX, then the
    `get_all` method of nova.compute.api.API is called with 2
    conflicting search options: {'tenant_id': XX, 'project_id': YY}.
    But because, latter on, in that `get_all` method we define a dict
    called filter_mapping, on which we iter upon, which value takes
    precedence depends on the order in which the dict is iterated upon.

    As the order in unpredictable and varies between Py2 and Py3, this
    is problematic. Especially, on Python 2 hash randomization is
    disabled by default but it's enabled by default on Py3 (unless
    the PYTHONHASHSEED env var is set to a fixed value) [0]

    The (unreliable) order we iterate on items of that `filter_mapping`
    dict is why the Tempest test_list_servers_by_admin_with_specified_tenant
    test randomly fails on Py35.

    This patch ensures that, if the all_tenants search option is
    not set, then the `tenant_id` search option is ignored. Note that
    this *is* the current behavior on Py27, as documented in lp:#1185290
    and tested by Tempest here [1].

    [0] https://docs.python.org/dev/using/cmdline.html#cmdoption-R
    [1] https://github.com/openstack/tempest/blob/fe1a8e289c2d79df29beaa6b3603afe5feb60fb3/tempest/api/compute/admin/test_servers.py#L96

    Change-Id: I1a74ef0f16da14444029c0f184b433df367ffb41
    Closes-Bug: #1659811
    (cherry picked from commit f702762934e0a46449091243a39ad5d57108b23f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.0.0b1 development milestone.