Openstack usage list not showing all projects

[Giuseppe Petralia]

In a customer environment running nova 2:17.0.5-0ubuntu1~cloud0

when querying projects usage list most recent projects are not listed in the reply.

Example:

$ openstack usage list --print-empty --start 2019-01-01 --end 2019-02-01

Not showing any information about project a897ea83f01c436e82e13a4306fa5ef0

But querying for the usage of the specific project we can retrieve the results:

openstack usage show --project a897ea83f01c436e82e13a4306fa5ef0 --start 2019-01-01 --end 2019-02-01
Usage from 2019-01-01 to 2019-02-01 on project a897ea83f01c436e82e13a4306fa5ef0:
+---------------+------------+
| Field | Value |
+---------------+------------+
| CPU Hours | 528.3 |
| Disk GB-Hours | 10566.07 |
| RAM MB-Hours | 2163930.45 |
| Servers | 43 |
+---------------+------------+

As a workaround we are able to get projects_uuid like this:
projects_uuid=$(openstack project list | grep -v ID | awk '{print $2}')

And iterate over them and get individuals usage:

for prog in $projects_uuid; do openstack project show $prog; openstack usage show --project $prog --start 2019-01-01 --end 2019-02-01; done

[melanie witt (melwitt)]

I had a look through the code and don't see anything that jumps out that would cause this behavior. I need a bit more information from you to understand what the problem could be.

Are you call the API with the admin role or a non-admin role? How is your policy.json configured for the usage API? The default [1] is like this:

  # List per tenant usage statistics for all tenants
  # GET /os-simple-tenant-usage
  #"os_compute_api:os-simple-tenant-usage:list": "rule:admin_api"

which means that by default, only users with the admin role will be able to list usage for all projects.

Please let us know what role and policy you have configured, so we may investigate further.

[1] https://docs.openstack.org/nova/latest/configuration/sample-policy.html

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

[Xav Paice (xavpaice)]

Seeing this in two difference clouds, when operating as the admin user.

Client versions: Tested with both 3.14.0 and 3.18.0

Nova is version 17.0.9 (package 17.0.9-0ubuntu1~cloud0) and 17.0.5 (package 17.0.5-0ubuntu1~cloud0).

Note that we get a bunch of results, but not a complete list - it's as if there's some pagination, but no message to tell us, nor a logical point at which it splits. One site, we get 15 results, another 508.

One of these clouds has two sites, and have the same projects in each site. One site returns 24 rows, one 15. The site that returns more rows runs Nova 17.0.7 and client 3.16.2 (although I've tried this client elsewhere and it doesn't seem to make any difference).

[Brin Zhang (zhangbailin)]

I deployed the master branch (now in trai), that command execute ok, like:
[root@localhost nova-specs]# nova usage-list
Usage from 2019-08-21 to 2019-09-19:
+----------------------------------+---------+---------------+-----------+----------------+
| Tenant ID | Servers | RAM MiB-Hours | CPU Hours | Disk GiB-Hours |
+----------------------------------+---------+---------------+-----------+----------------+
| 40d622ea71704a11a37115947b250b83 | 8 | 151681.87 | 296.25 | 296.25 |
+----------------------------------+---------+---------------+-----------+----------------+
[root@localhost nova-specs]# openstack usage list
Usage from 2019-08-21 to 2019-09-19:
+---------+---------+--------------+-----------+---------------+
| Project | Servers | RAM MB-Hours | CPU Hours | Disk GB-Hours |
+---------+---------+--------------+-----------+---------------+
| admin | 8 | 151685.29 | 296.26 | 296.26 |
+---------+---------+--------------+-----------+---------------+

[Brin Zhang (zhangbailin)]

Nova/OpenStack Train version.
[root@localhost nova-specs]# nova-api --version
19.1.0
[root@localhost nova-specs]# openstack --version
openstack 3.19.0

I think it works ok now.

[Rodrigo Barbieri (rodrigo-barbieri2010)]

I have this issue happening on two environments, one running 17.0.10 and the other running 17.0.11.

I have investigated this in more detail, and found out that the reason it does not show all projects is because there are more than 1000 instances to show. The Nova Tenant Usage API will not return more than 1000 results unless specified in nova.conf.

Increasing this value in nova.conf is not the best alternative for us as it will use more bandwidth and can hit the limit again. The ideal approach is to use pagination, since it is already implemented in this API for this purpose.

To use pagination, OpenStack CLI 3.19 or later needs to be used, and run the command:

$ openstack usage list --start 2019-08-01 --end 2019-08-31 --os-compute-api-version 2.40

However, running the above command hangs. Running again with --debug results in infinite loop doing queries, as follows:

1) The first query is

REQ: curl -g -i --cacert "xxx.crt" -X GET https://xxx:8774/v2/<admin-project-id>/os-simple-tenant-usage?start=2019-08-01T00:00:00&end=2019-08-31T00:00:00&detailed=1 -H "Accept: application/json" -H "OpenStack-API-Version: compute 2.40" -H "User-Agent: python-novaclient" -H "X-Auth-Token: xxx" -H "X-OpenStack-Nova-API-Version: 2.40"

2) it returns approximately 960 instances, the last instance id is fb23858a-b004-4367-a262-7adef90efac0 and has status "active".

3) The second query is

REQ: curl -g -i --cacert "xxx.crt" -X GET https://xxx:8774/v2/<admin-project-id>/os-simple-tenant-usage?start=2019-08-01T00:00:00&end=2019-08-31T00:00:00&marker=fb23858a-b004-4367-a262-7adef90efac0&detailed=1 -H "Accept: application/json" -H "OpenStack-API-Version: compute 2.40" -H "User-Agent: python-novaclient" -H "X-Auth-Token: xxx" -H "X-OpenStack-Nova-API-Version: 2.40"

4) The response is exactly the same as the first response.

5) Steps #3 and #4 go on forever.

The problem seems to be server-side as the CLI is behaving accordingly, but the API is ignoring the marker completely.

[Balazs Gibizer (balazs-gibizer)]

I tried with 3 servers and with limit=1. And I cannot reproduce the problem. I was able to iterate on the 3 servers one by one using the marker provided in the response [1]. Similarly with limit=2. I also tried using the next link in the response for pagination and that worked too. Then I also tried to set the [api]/max_limit in the nova.conf to 2 and removed the limit form the request URI. Still pagination work.

So I think this works on a current master. I checked the git history and found a relevant bug[2]. It was fixed in 17.0.10 [3] and according to the bug report the reporter uses 17.0.5. so that would explain it. But later comment reports it on 17.0.10 and 17.0.11. So I'm puzzled.

[1] http://paste.openstack.org/show/793054/
[2] https://bugs.launchpad.net/nova/+bug/1796689
[3] https://review.opendev.org/#/c/632516/

[melanie witt (melwitt)]

I too am puzzled for the same reasons gibi described.

I am only aware of one bug having to do with an infinite pagination [1] that is merged to stable/queens but not yet released, BUT as far as I can tell, that code is not part of the path used by the the os-simple-tenant-usage API and thus seems unrelated.

[1] https://bugs.launchpad.net/nova/+bug/1849409

[Balazs Gibizer (balazs-gibizer)]

Without a local reproduction I cannot really move forward with this bug. I'm marking this Incomplete. If you can reproduce this on a recent OpenStack version then please set it back to New and provide information about the reproduction.

[Rodrigo Barbieri (rodrigo-barbieri2010)]

This issue has been further investigated and diagnosed to be a configuration mistake.