downloading large image from glance locks out threads

[Olaf Seibert]

Reproduction scenario:

- create a large base image (50 GB or so). This could be done for example by creating a VM, filling its ephemeral storage (which is 50 GB in our case) with lots of junk data, shutting down the VM, and creating an image from that.
- create a VM based on this image.

While the image is downloaded by nova-compute from glance, it seems that other threads are locked out (too long). Network connection failures get logged an if it lasts long enough, creating the VM often fails.

We first saw this on Queens, but I reproduced the same issue on Ussuri.
As hypervisor we use Libvirt + KVM.
For storage we use Quobyte (shared storage), but these tests were done on compute nodes with local storage (using LVM).
For networking we use Midonet (on Queens) and OVS on Ussuri.

My solution was to put "greenthreads.sleep(0)" in the inner loop, like so:

From: Olaf Seibert <email address hidden>
Date: Thu, 10 Jun 2021 11:38:16 +0000
Subject: Allow other threads to run.

While downloading a base image from Glance, other threads don't get
enough of a chance to run, and network connections start to time out.
See os-9400.
---
 nova/image/glance.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/nova/image/glance.py b/nova/image/glance.py
index 13bfd90..8231351 100644
--- a/nova/image/glance.py
+++ b/nova/image/glance.py
@@ -30,6 +30,7 @@ import time
 import cryptography
 from cursive import exception as cursive_exception
 from cursive import signature_utils
+from eventlet import greenthread
 import glanceclient
 import glanceclient.exc
 from glanceclient.v2 import schemas
@@ -384,6 +385,7 @@ class GlanceImageServiceV2(object):
                 try:
                     for chunk in image_chunks:
                         verifier.update(chunk)
+ greenthread.sleep(0)
                     verifier.verify()

                     LOG.info('Image signature verification succeeded '
@@ -400,6 +402,7 @@ class GlanceImageServiceV2(object):
                     if verifier:
                         verifier.update(chunk)
                     data.write(chunk)
+ greenthread.sleep(0)
                 if verifier:
                     verifier.verify()
                     LOG.info('Image signature verification succeeded '

However, the download happens in chunks of only 64 KB, so the sleep(0) is called extremely frequently. Maybe there is a better solution, but it should not be too complicated for such a tight loop.

I have attached some log files, since this bug tracker thinks the bug description is too long.

[melanie witt (melwitt)]

I don't have in-depth knowledge in this area but this sounds similar to a past bug involving image uploads to glance [1] which was fixed by [2]. We might need a matching fix for image download.

[1] https://bugs.launchpad.net/nova/+bug/1874032
[2] https://review.opendev.org/c/openstack/nova/+/734776

[melanie witt (melwitt)]

Setting this as Triaged because this is a legit issue and if it is indeed similar to bug 1874032, we know what can be done to address it.