nova boot specifying flavor by name fails intermittently past 1000 flavors

[rick jones]

While running a script which:

1) creates a project
2) creates a user in that project
3) creates a flavor
4) tweaks some nova and neutron quotas for that project
5) switches to that project
6) creates an image
7) tweaks some security group rules
8) creates a network/subnet/router tuple
9) creates a port
10) creates a floating IP with that port id
and finally
11) nova boots with that port, image (by name) and flavor (again by name)

once one gets above 1000 iterations, the nova boot will start to fail from time to time with messages like:

ERROR (CommandError): No flavor with a name or ID of 'uberplay1053' exists.

It isn't every nova boot past the 1000th flavor:

ERROR (CommandError): No flavor with a name or ID of 'uberplay1053' exists.
ERROR (CommandError): No flavor with a name or ID of 'uberplay1055' exists.
ERROR (CommandError): No flavor with a name or ID of 'uberplay1062' exists.
ERROR (CommandError): No flavor with a name or ID of 'uberplay1064' exists.
...
ERROR (CommandError): No flavor with a name or ID of 'uberplay1312' exists.
ERROR (CommandError): No flavor with a name or ID of 'uberplay1316' exists.
ERROR (CommandError): No flavor with a name or ID of 'uberplay1319' exists.
ERROR (CommandError): No flavor with a name or ID of 'uberplay1321' exists.

The "holes" in that sequence are good - those instances booted and were accessible just fine. The flavors themselves are there - they can be seen via nova flavor-show <UUID>.

This is with Liberty. There were 6 flavors already defined before the script was started. "uberplay1053" was the first failure, counting in that sequence started at 0.

I will (attempt to) attach the script shortly.

~/rjones2$ nova --version
2.30.3

Neutron networking with OVS. Instances using local storage not Cinder. Glance using Swift for image storage.

[rick jones (perfgeek)]

The nova version appears to be 12.0.3.

[Matt Riedemann (mriedem)]

Hmm, are there any errors in the nova-api logs?

[Matt Riedemann (mriedem)]

To be clear, when you're getting the flavor not found errors, are those boot requests passing or failing? When calling nova boot with the --flavor the CLI looks up the flavor by name and if it can't find it it should fail the nova boot command.

[Matt Riedemann (mriedem)]

I think what might be happening is CONF.osapi_max_limit defaults to 1000 and that's the default limit when listing flavors. When you just pass the flavor name, novaclient is trying to lookup the flavor, which in this case is doing an index request to the nova API. What I'm not sure about is if it's hitting a limit and not actually using the filter, i.e. if there are already 1000 flavors in the database and you're searching for a flavor by name after those 1000, is it returning the first 1000 and not actually applying the name filter?

When you run this, can you add the --debug option to the nova boot command since that should show the flavor request/response details and that would be helpful when it fails.

[Matt Riedemann (mriedem)]

BTW, I think saving the id returned from nova flavor-create and then using that would probably get around this since then we wouldn't have to do the lookup by name, you might see if that resolves the issue. It's not a fix for the bug, but it'd be interesting to know if it fixes it.

[Matt Riedemann (mriedem)]

Yup, the problem is the name isn't passed to the server for filtering, you can recreate this with just a simple devstack setup and the default flavors. In this case I'm just doing:

nova --debug flavor-show m1.tiny

Here are the details:

1. Make a request to the server to get the flavor by the name, this fails with a 404 since the API expects the flavor id.

DEBUG (session:248) REQ: curl -g -i -X GET http://9.5.124.163:8774/v2.1/a1c6a5d891d1420abec0227f651b74c9/flavors/m1.tiny -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.25" -H "X-Auth-Token: {SHA1}e2866846f66ec299de60b3ebf17d931e7b9b9579"
DEBUG (connectionpool:387) "GET /v2.1/a1c6a5d891d1420abec0227f651b74c9/flavors/m1.tiny HTTP/1.1" 404 80
DEBUG (session:277) RESP: [404] Content-Length: 80 X-Compute-Request-Id: req-537e7042-8626-485d-afb3-deb172b370d0 Vary: X-OpenStack-Nova-API-Version Connection: keep-alive X-Openstack-Nova-Api-Version: 2.25 Date: Thu, 07 Apr 2016 01:10:24 GMT Content-Type: application/json; charset=UTF-8
RESP BODY: {"itemNotFound": {"message": "Flavor m1.tiny could not be found.", "code": 404}}

2. Now do a listing of the flavors, this is just passing is_public=None, it's not passing the name as a search filter. This returns all of the flavors.

DEBUG (session:248) REQ: curl -g -i -X GET http://9.5.124.163:8774/v2.1/a1c6a5d891d1420abec0227f651b74c9/flavors?is_public=None -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.25" -H "X-Auth-Token: {SHA1}e2866846f66ec299de60b3ebf17d931e7b9b9579"
DEBUG (connectionpool:387) "GET /v2.1/a1c6a5d891d1420abec0227f651b74c9/flavors?is_public=None HTTP/1.1" 200 1720
DEBUG (session:277) RESP: [200] Content-Length: 1720 X-Compute-Request-Id: req-5ea197c2-1441-4e42-8c9f-3d2b70fa1abe Vary: X-OpenStack-Nova-API-Version Connection: keep-alive X-Openstack-Nova-Api-Version: 2.25 Date: Thu, 07 Apr 2016 01:10:25 GMT Content-Type: application/json
RESP BODY: {"flavors": [{"id": "1", "links": [{"href": "http://9.5.124.163:8774/v2.1/a1c6a5d891d1420abec0227f651b74c9/flavors/1", "rel": "self"}, {"href": "http://9.5.124.163:8774/a1c6a5d891d1420abec0227f651b74c9/flavors/1", "rel": "bookmark"}], "name": "m1.tiny"}, {"id": "2", "links": [{"href": "http://9.5.124.163:8774/v2.1/a1c6a5d891d1420abec0227f651b74c9/flavors/2", "rel": "self"}, {"href": "http://9.5.124.163:8774/a1c6a5d891d1420abec0227f651b74c9/flavors/2", "rel": "bookmark"}], "name": "m1.small"}, {"id": "3", "links": [{"href": "http://9.5.124.163:8774/v2.1/a1c6a5d891d1420abec0227f651b74c9/flavors/3", "rel": "self"}, {"href": "http://9.5.124.163:8774/a1c6a5d891d1420abec0227f651b74c9/flavors/3", "rel": "bookmark"}], "name": "m1.medium"}, {"id": "4", "links": [{"href": "http://9.5.124.163:8774/v2.1/a1c6a5d891d1420abec0227f651b74c9/flavors/4", "rel": "self"}, {"href": "http://9.5.124.163:8774/a1c6a5d891d1420abec0227f651b74c9/flavors/4", "rel": "bookmark"}], "name": "m1.large"}, {"id": "42", "links": [{"href": "http://9.5.124.163:8774/v2.1/a1c6a5d891d1420abec0227f651b74c9/flavors/42", "rel": "self"}, {"href": "http://9.5.124.163:8774/a1c6a5d891d1420abec0227f651b74c9/flavors/42", "rel": "bookmark"}]...

[Matt Riedemann (mriedem)]

This is the code in novaclient that is doing the filtering logic when listing resources:

https://github.com/openstack/python-novaclient/blob/0634a38cdcb756199160aa1e476c8212c7145250/novaclient/base.py#L395

[Matt Riedemann (mriedem)]

Dropping to medium severity since you can use the flavor id to get around the listing/filtering issue.

[Matt Riedemann (mriedem)]

It actually looks like this is a limitation in the flavors API in nova, you can't filter on name:

https://github.com/openstack/nova/blob/347ebb20e3217c2f8abf261622ba420c9fd077bd/nova/api/openstack/compute/flavors.py#L78

We'd have to update the API to do this, and at least for this case it's probably not going to happen, since most deployments probably don't have 1000+ flavors to pick from.

[Matt Riedemann (mriedem)]

I'm going to say we're not going to fix this in the flavors API to filter by name. I don't honestly see a reason to support filtering 1000+ flavors by name which requires the API change (with a microversion) to do this properly, when you can just use the flavor id that was provided when creating the flavor to pass on the boot CLI.

[rick jones (perfgeek)]

The boot requests themselves fail.

Admittedly, 1000 or more flavors seems unlikely, but then customers can want the damnedest things. At this point I've not heard of one asking for it, the test I was running was as much seeking to stay ahead of the game as anything else.

Relying on keeping track of the flavor's UUID is going to be brittle - particularly if as it seems one cannot successfully do a nova flavor-list for more than 1000 flavors. Once one has lost track of that UUID, how can one regain it? And there is something of an inconsistency within nova here because nova image-list deals with > 1000 automagically (*), and for nova list one has the --limit option these days to deal with > 1000 instances (well, at least as admin)

What remains perplexing to me is that not all nova boot commands issued after the 1000th flavor was created were unsuccessful in looking up the flavor by name.

*) And it would seem that nova boot was fine with looking up images > 1000 by name - unless the flavor lookup errors were masking what would have been a later image lookup failure.

[Andrea Rosa (andrea-rosa-m)]

As the use case is unlikely I am with Matt in thinking that it is probably not worth to get a new API version for this, what we could do is to add a new nova-manage command for getting the flavor details filtered by flavor-name?