nova image-list can run slowly.

[Tom Hancock]

$nova image-list can often run very slowly (or any image listing done from nova) due to the fact that
when it calls into glanceclient it typically doesn't specify a page_size and so glanceclient uses its default of 20.
For larger lists of images (eg. if there were lots of snapshots, or if there were a lot of public images) this
results in a succession of queries from nova to the glance api server (and on to the database via registry)
for a small amount of images at a time, when a single request could often return many more or all images.
For 1000 images the hit is 50X, and so forth.

[Aaron Rosen (arosen)]

In my opinion I think the default of 20 in the python-glanceclient should be bumped from 20 to something more reasonable instead .

[Wei Wang (damon-devops)]

nice catch, I agree with @Aaron, we need a reasonalbe number.

[Andrew Laski (alaski)]

The default page size is set in glanceclient so I'm moving this issue over to that project.

[Travis Tripp (travis-tripp)]

Looks like simple fix in images.py under glanceclient/v1 and glanceclient/v2 Just change the DEFAULT_PAGE_SIZE variable. Need to load a lot of data and see if this affects CLI, though.

The bigger question is what should the default page size be?

[Yossi Ovadia (jabadia)]

So, my plan is to fill up the db with many (fake) images using -

for i in {1..1000} ;do i=$(uuidgen); mysql -uroot -ppassword glance -e "insert into images (id,name,size,status,is_public, created_at,deleted,min_disk,min_ram,protected,virtual_size) values ( \"$i\", 'yiossvol1', 1, 'active',1 ,now(),0,0 ,0,0,0);" ; done

Then, check 'nova image-list' time, before and after the suggested resolution above.
I'll keep you posted..

[Yossi Ovadia (jabadia)]

Update:

So, here are the results for 'glance image-list' : ( first is for 20, second is for 1000 )

± |bugfix/1279832_nova_image_list_slow ✗| → cat default_page_20.txt
real 0m4.442s
real 0m4.540s
real 0m4.580s
real 0m4.507s
real 0m4.618s
real 0m4.642s
real 0m4.722s
real 0m4.685s
real 0m4.625s
real 0m4.668s

After changing DEFAULT_IMAGE_SIZE to 1000 :

 ubuntu in /opt/stack/python-glanceclient/glanceclient/v2
± |bugfix/1279832_nova_image_list_slow ✗| → cat default_page_1000.txt
real 0m1.531s
real 0m1.533s
real 0m1.677s
real 0m1.551s
real 0m1.767s
real 0m1.564s
real 0m1.544s
real 0m1.483s
real 0m1.646s
real 0m1.572s

Indeed an impressive improvement.

But!

This does not effect nova image-list which , I assume , does not goes through glanceclient.

I'll keep looking at it and update..

[Yossi Ovadia (jabadia)]

Ok, i had to restart nova-api to load the new DEFAULT_IMAGE_SIZE , and indeed, nova image list works faster now.

real 0m2.587s
real 0m2.667s
real 0m2.524s
real 0m2.643s
real 0m2.672s
real 0m2.554s
real 0m2.882s
real 0m2.693s
real 0m2.669s
real 0m2.619s

But , adding Flavio who has a good comment .. which makes me wonder if we want to fix it.

[Yossi Ovadia (jabadia)]

PS, without it takes ~7 seconds ( vs 2.5 ) but as mentioned above, Flavio will comment.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/158982

[Nikhil Komawar (nikhil-komawar)]

About DEFAULT_IMAGE_SIZE:

11:10:00 [ nikhil_k] So, it was decided to not do that
11:10:08 [ nikhil_k] we've gone back and forth on that with Mark
11:10:36 [ nikhil_k] it would be nice to have a param passed to change the page size and logic added to the client (Nova in this case)

---

The main reason being adding that huge load on the Glance database. The queries are not simple and with large number of metadata on the image, they will almost always be slow queries.

Seems like the intent of the bug is speed of the call, it might make sense to add such logic on the client side so that it's customizable per client (in this case Nova, in other cases Cinder, Ironic etc).

Hope it helps.

[Flavio Percoco (flaper87)]

I remember participating on this discussion elsewhere but I can't remember where :(

It'd be great to have a chance to read what my thoughts back them were :P

As it stands right now, I don't agree with the proposed solution and I believe that's not a sane default. If there's a desire to change the page size to something as high as 1000, it should happen in a case-by-case basis. For example, Nova could have its own page size for glance.

[Ian Cordasco (icordasc)]

Most real world APIs have the following de facto limits:

- max: 100 items per page

- min: 10 items per page

- default page size: 25 or 30 items per page

Setting our default page limit to 1000 is quite unrealistic. In fact, I'm not convinced that it isn't backwards incompatible for us to be changing that default. Anything using the client as Python library that depends on the behaviour of 20 per page, will be broken after updating to glanceclient {{version this might be released in}}.

I think the above suggestions for fixing nova's use of glanceclient are correct.

[Shashwat Srivastava (shashwat-srivastava)]

Hi,

As to what I understand when the client makes a request for image then its Default size shall be over written with that of glance also to do so there will a addition in the command specifying the default page size.
So if my above assumption/understanding is correct, can we point down to a decent no. which should be the DEFAULT_PAGE_SIZE which will be defined while making the query.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Glance Bot (<email address hidden>) on branch: master
Review: https://review.openstack.org/158982

[Shashwat Srivastava (shashwat-srivastava)]

Hi,

This bug still stands still in my bug list.
Can we get down to a appropriate size or will the logic be added for each case depending upon the number of images present.
Like,
if number of images is <50, DEFAULT_PAGE_SIZE = 20;
if number of image >50 and <100, DEFAULT_PAGE_SIZE = 40;
if number of image >100 and <200, DEFAULT_PAGE_SIZE = 80;
.
.
.
.
so on till image size >1000