Broken instances quota check in Liberty

[Andrew Bogott]

I have recently upgraded my cluster to Liberty for all projects. Now, when I create new instances, I frequently get an incorrect quota warning from the instance creation workflow, despite having plenty of available quota:

"The requested instance cannot be launched as you only have 0 of your quota available."

The issue is happening within the _get_tenant_compute_usages() check. That function determines (correctly) that I am able to list instances in all projects, so passes 'all_tenants' to the nova api. That results in an api call that looks like this:

"GET /v2/<tenantid>/servers/detail?all_tenants=True&tenant_id=<tenantid> HTTP/1.1"

Nova replies with a list of every instance in my entire cloud, 719 at last count. The call is very slow and, of course, 719 is many more than my instance quota for the current project, so Horizon determines that I am over quota.

Note that this issue didn't appear when I was running Liberty Horizon with Kilo Nova (so maybe this is a bug or change in the nova-api) Best I can tell, the offending code in Horizon is still present in the git head.

[Andrew Bogott (andrewbogott)]

It appears that this was broken due to this change in nova:

https://github.com/openstack/nova/commit/55e63f83a7caec5a8d85b7532c501e1b846295ba

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/358790

[Rob Cresswell (robcresswell-deactivatedaccount)]

I'd really like some more input on this. It seems correct to my eyes, but I'm wondering what the original motivation was. Also we won't backport this to stable, but at least its one less patch to carry when you upgrade :)

[Andrew Bogott (andrewbogott)]

Here is how I'm demonstrating the issue:

1) I have a user who is a member of several projects, some of which contain many servers
2) My policy file includes a default rule "default": [["rule:admin_or_projectadmin"]] and my test user is a projectadmin.
3) debug lines:

def _get_tenant_compute_usages(request, usages, disabled_quotas, tenant_id):
    LOG.warn("ANDREW: _get_tenant_compute_usages")
    if tenant_id:
        # determine if the user has permission to view across projects
        # there are cases where an administrator wants to check the quotas
        # on a project they are not scoped to
        all_tenants = policy.check((("compute", "compute:get_all_tenants"),),
                                   request)
        LOG.warn("ANDREW: all_tenants is %s" % all_tenants)
        instances, has_more = nova.server_list(
            request, search_opts={'tenant_id': tenant_id},
            all_tenants=all_tenants)
    else:
        instances, has_more = nova.server_list(request)
    LOG.warn("ANDREW: instances is %s" % instances)

Output looks like this:

[Fri Nov 04 16:07:09.319728 2016] [:error] [pid 29436] ANDREW: _get_tenant_compute_usages
[Fri Nov 04 16:07:09.320121 2016] [:error] [pid 29436] ANDREW: all_tenants is True
[Fri Nov 04 16:08:09.003508 2016] [:error] [pid 29436] ANDREW: instances is <super long list>

[Andrew Bogott (andrewbogott)]

Note that the policy check itself might be insane:

> policy.check((("compute", "compute:get_all_tenants"),),request)

get_all_tenants on nova? I suspect that testers can't reproduce this because they aren't passing that check.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/358790
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=d7d59b85969acb757f9871535758d75592aae660
Submitter: Jenkins
Branch: master

commit d7d59b85969acb757f9871535758d75592aae660
Author: andrewbogott <email address hidden>
Date: Mon Aug 22 10:50:38 2016 -0500

    Remove all_tenants flag in quota check

    I don't really know what this was meant to do, but what it actually
    does doesn't make any sense -- when checking a quota for project
    'foo' we really don't care about how many instances are used
    in the entire cloud, only in project 'foo'.

    Until Liberty, the all_tenant flag was broken in the nova
    api, which probably explains why this was harmless until
    Liberty.

    Change-Id: I9e12c10951e7518ab1c622a4daac962bad332da5
    Closes-Bug: #1610693

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 11.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/422760

[Rob Cresswell (robcresswell-deactivatedaccount)]

I cannot recall why I said this couldnt be backported. I think I meant to say not to stable/kilo. Anyway, definitely Newton, maybe Mitaka.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/422760
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=62c89a38c06a67e3fc29be3d25f2aead16e80980
Submitter: Jenkins
Branch: stable/newton

commit 62c89a38c06a67e3fc29be3d25f2aead16e80980
Author: andrewbogott <email address hidden>
Date: Mon Aug 22 10:50:38 2016 -0500

    Remove all_tenants flag in quota check

    I don't really know what this was meant to do, but what it actually
    does doesn't make any sense -- when checking a quota for project
    'foo' we really don't care about how many instances are used
    in the entire cloud, only in project 'foo'.

    Until Liberty, the all_tenant flag was broken in the nova
    api, which probably explains why this was harmless until
    Liberty.

    Change-Id: I9e12c10951e7518ab1c622a4daac962bad332da5
    Closes-Bug: #1610693
    (cherry picked from commit d7d59b85969acb757f9871535758d75592aae660)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 10.0.2 release.

[Karim Boumedhel (karmab)]

weirdly enough, this bug seems to only affect user going with keystone v3, right ?

[Karim Boumedhel (karmab)]

we re getting this error again on liberty ( no keystone v3 in the setup).anu chance to backport this to liberty ?