disk usage of the nova image cache is not counted as used disk space
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Low
|
Balazs Gibizer | ||
Stein |
Fix Released
|
Low
|
Lee Yarwood | ||
Train |
Fix Released
|
Low
|
Balazs Gibizer | ||
Ussuri |
Fix Released
|
Low
|
Balazs Gibizer |
Bug Description
Description
===========
The nova-compute service keeps a local image cache for glance images used for nova servers to avoid multiple download of the same image from glance. The disk usage of such cache is not calculated as local disk usage in nova and not reported to placement as used DISK_GB. This leads to disk over-allocation.
Also the size of that cache cannot be limited by nova configuration so the deployer cannot reserve disk space for that cache with reserved_
Steps to reproduce
==================
* Set up a single node devstack
* Create and upload an image with a not too small physical size. Like an image with 1G physical size.
* Check the current disk usage of the Host OS and configure reserved_
* Boot two servers from that image with a flavor, like d1 (disk=5G)
* Nova will download the glance image once to the local cache which result in a 1GB disk usage
* Nova will create two root file systems, one for each VM. Those disks initially has minimal physical disk size, but has 5G virtual size.
* At this point Nova allocated 5G + 5G of DISK_GB in placement, but due to the image in the cache the total disk usage of the two VMs + cache can be 5G + 5G + 1G, if both VMs overwrite and fills the content of its own disk.
Expected result
===============
Option A)
Nova maintains a DISK_GB allocation in placement for the images in its cache. This way the expected DISK_GB allocation in placement is 5G + 5G + 1G at the end
Option B)
Nova provides a config option to limit the maximum size of the image cache and therefore the deployer can include the maximum image cache size into the reserved_
Actual result
=============
Only 5G + 5G was allocation from placement. So disk space is over-allocated by the image cache.
Environment
===========
Devstack from recent master
stack@aio:
4b62c90063 Merge "Remove stale nested backport from InstancePCIRequ
libvirt driver with file based image backend
Logs & Configs
==============
http://
tags: | added: placement resource-tracker |
description: | updated |
Changed in nova: | |
status: | In Progress → Confirmed |
Changed in nova: | |
importance: | Undecided → Low |
We discussed potential solution ideas in IRC [1].
Option B) does not work as there is no way to purge cached images from the cache when it hits the limit as the images are used as a backing store for the VM root fs images.
Option A) is complex. The scheduler does not know today if an image is cached on a compute or not. So it cannot include the extra disk resource request for the cache to the allocation_ candidates query.
We might be able to extend cache_images() compute RPC call to return information if an image is already cached or not. So we could filter the alternate hosts returned by the scheduler based on the fact that the host has the image cached already or it has extra free space to do so.
Option C)
Another alternative is to provide a way to disable the caching by using use_cow_images = False to make the VM root image independent from the cached image and then add a way to immediately purge the unused image from the cache[2].
Possible Workarounds
Workaround 1: least behavior of that filter to avoid overallocation.
Keep using the DiskFilter if you are on < Stein and rely on the disk_available_
Workaround 2: subdirectory_ name config can be used to define the name of the cache dir (defaulted to "_base") under the patch defined by instances_path config. So one way to separate the instance_path from the cache dir is to mount a different partition under $instances_ path/$image_ cache_subdirect ory_name directory.
Move the nova instance cache to a separate partition. The image_cache_
Workaround 3:
In case of non-edge deployments, where we don't need to fill the last piece of disk with VMs, one way to avoid this problem is to spread the VMs by free space available. The DiskWeigher in the scheduler does this by default and it is enabled by default.
[1] http:// eavesdrop. openstack. org/irclogs/ %23openstack- nova/%23opensta ck-nova. 2020-05- 11.log. html#t2020- 05-11T15: 12:29 /github. com/openstack/ nova/blob/ d6450879c7f7dd1 9366b6f002301fb bf87918026/ nova/virt/ libvirt/ imagebackend. py#L585
[2] https:/