/v2.1/servers/detail?tenant_id=XX returns unpredictable results

Bug #1659811 reported by Jordan Pittier
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Jordan Pittier
Ocata
Fix Committed
Medium
Matt Riedemann

Bug Description

Found this while investigating why the tempest test test_list_servers_by_admin_with_specified_tenant randomly fails on Py35.

This test produces the following API call:

GET /v2.1/servers/detail?tenant_id=XXX

Which calls this method in Nova:

nova.api.openstack.compute.servers.ServersController.detail()

In the _get_servers() method that detail() calls, for some reason (bug ? [1]), the 'project_id' is added to the search options. So now, we see that get_all() is called with both tenant_id and project_id:

Searching by: {'project_id': '74e1044b53de44d1bac80cded5146504', 'deleted': False, 'tenant_id': '153d2038e0bc4ea99819a21a55cb66ea'} get_all /opt/stack/new/nova/nova/compute/api.py:2336

Now, in nova/nova/compute/api.py in the get_all() method, there's a dict called filter_mapping, that is iterated upon. I believe, based on which order the dict (i.e the search options) is iterated upon, either tenant_id or project_id gets rewritten. This leads to random return values.

[1] : https://github.com/openstack/nova/blob/cba26a6e561c18fa4659efac8ddc0b3c139023fe/nova/api/openstack/compute/servers.py#L322

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/426259

Changed in nova:
assignee: nobody → Jordan Pittier (jordan-pittier)
status: New → In Progress
Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/426259
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f702762934e0a46449091243a39ad5d57108b23f
Submitter: Jenkins
Branch: master

commit f702762934e0a46449091243a39ad5d57108b23f
Author: Jordan Pittier <email address hidden>
Date: Fri Jan 27 15:22:25 2017 +0100

    Fix unspecified bahavior on GET /servers/detail?tenant_id=X as admin

    When an admin calls /v2.1/servers/detail?tenant_id=XX, then the
    `get_all` method of nova.compute.api.API is called with 2
    conflicting search options: {'tenant_id': XX, 'project_id': YY}.
    But because, latter on, in that `get_all` method we define a dict
    called filter_mapping, on which we iter upon, which value takes
    precedence depends on the order in which the dict is iterated upon.

    As the order in unpredictable and varies between Py2 and Py3, this
    is problematic. Especially, on Python 2 hash randomization is
    disabled by default but it's enabled by default on Py3 (unless
    the PYTHONHASHSEED env var is set to a fixed value) [0]

    The (unreliable) order we iterate on items of that `filter_mapping`
    dict is why the Tempest test_list_servers_by_admin_with_specified_tenant
    test randomly fails on Py35.

    This patch ensures that, if the all_tenants search option is
    not set, then the `tenant_id` search option is ignored. Note that
    this *is* the current behavior on Py27, as documented in lp:#1185290
    and tested by Tempest here [1].

    [0] https://docs.python.org/dev/using/cmdline.html#cmdoption-R
    [1] https://github.com/openstack/tempest/blob/fe1a8e289c2d79df29beaa6b3603afe5feb60fb3/tempest/api/compute/admin/test_servers.py#L96

    Change-Id: I1a74ef0f16da14444029c0f184b433df367ffb41
    Closes-Bug: #1659811

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/430504

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/430504
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1c992c7a172af0890c9fea147e018257ee018f2d
Submitter: Jenkins
Branch: stable/ocata

commit 1c992c7a172af0890c9fea147e018257ee018f2d
Author: Jordan Pittier <email address hidden>
Date: Fri Jan 27 15:22:25 2017 +0100

    Fix unspecified bahavior on GET /servers/detail?tenant_id=X as admin

    When an admin calls /v2.1/servers/detail?tenant_id=XX, then the
    `get_all` method of nova.compute.api.API is called with 2
    conflicting search options: {'tenant_id': XX, 'project_id': YY}.
    But because, latter on, in that `get_all` method we define a dict
    called filter_mapping, on which we iter upon, which value takes
    precedence depends on the order in which the dict is iterated upon.

    As the order in unpredictable and varies between Py2 and Py3, this
    is problematic. Especially, on Python 2 hash randomization is
    disabled by default but it's enabled by default on Py3 (unless
    the PYTHONHASHSEED env var is set to a fixed value) [0]

    The (unreliable) order we iterate on items of that `filter_mapping`
    dict is why the Tempest test_list_servers_by_admin_with_specified_tenant
    test randomly fails on Py35.

    This patch ensures that, if the all_tenants search option is
    not set, then the `tenant_id` search option is ignored. Note that
    this *is* the current behavior on Py27, as documented in lp:#1185290
    and tested by Tempest here [1].

    [0] https://docs.python.org/dev/using/cmdline.html#cmdoption-R
    [1] https://github.com/openstack/tempest/blob/fe1a8e289c2d79df29beaa6b3603afe5feb60fb3/tempest/api/compute/admin/test_servers.py#L96

    Change-Id: I1a74ef0f16da14444029c0f184b433df367ffb41
    Closes-Bug: #1659811
    (cherry picked from commit f702762934e0a46449091243a39ad5d57108b23f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.1

This issue was fixed in the openstack/nova 15.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.0.0b1

This issue was fixed in the openstack/nova 16.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.