disk usage of the nova image cache is not counted as used disk space

[Balazs Gibizer]

Description
===========
The nova-compute service keeps a local image cache for glance images used for nova servers to avoid multiple download of the same image from glance. The disk usage of such cache is not calculated as local disk usage in nova and not reported to placement as used DISK_GB. This leads to disk over-allocation.

Also the size of that cache cannot be limited by nova configuration so the deployer cannot reserve disk space for that cache with reserved_host_disk_mb config.

Steps to reproduce
==================
* Set up a single node devstack
* Create and upload an image with a not too small physical size. Like an image with 1G physical size.
* Check the current disk usage of the Host OS and configure reserved_host_disk_mb in nova-cpu.conf accordingly.
* Boot two servers from that image with a flavor, like d1 (disk=5G)
* Nova will download the glance image once to the local cache which result in a 1GB disk usage
* Nova will create two root file systems, one for each VM. Those disks initially has minimal physical disk size, but has 5G virtual size.
* At this point Nova allocated 5G + 5G of DISK_GB in placement, but due to the image in the cache the total disk usage of the two VMs + cache can be 5G + 5G + 1G, if both VMs overwrite and fills the content of its own disk.

Expected result
===============
Option A)
Nova maintains a DISK_GB allocation in placement for the images in its cache. This way the expected DISK_GB allocation in placement is 5G + 5G + 1G at the end

Option B)
Nova provides a config option to limit the maximum size of the image cache and therefore the deployer can include the maximum image cache size into the reserved_host_disk_mb during dimensioning of the disk space of the compute.

Actual result
=============
Only 5G + 5G was allocation from placement. So disk space is over-allocated by the image cache.

Environment
===========

Devstack from recent master

stack@aio:/opt/stack/nova$ git log --oneline | head -n 1
4b62c90063 Merge "Remove stale nested backport from InstancePCIRequests"

libvirt driver with file based image backend

Logs & Configs
==============
http://paste.openstack.org/show/793388/

[Balazs Gibizer (balazs-gibizer)]

We discussed potential solution ideas in IRC [1].

Option B) does not work as there is no way to purge cached images from the cache when it hits the limit as the images are used as a backing store for the VM root fs images.

Option A) is complex. The scheduler does not know today if an image is cached on a compute or not. So it cannot include the extra disk resource request for the cache to the allocation_candidates query.

We might be able to extend cache_images() compute RPC call to return information if an image is already cached or not. So we could filter the alternate hosts returned by the scheduler based on the fact that the host has the image cached already or it has extra free space to do so.

Option C)
Another alternative is to provide a way to disable the caching by using use_cow_images = False to make the VM root image independent from the cached image and then add a way to immediately purge the unused image from the cache[2].

Possible Workarounds

Workaround 1:
Keep using the DiskFilter if you are on < Stein and rely on the disk_available_least behavior of that filter to avoid overallocation.

Workaround 2:
Move the nova instance cache to a separate partition. The image_cache_subdirectory_name config can be used to define the name of the cache dir (defaulted to "_base") under the patch defined by instances_path config. So one way to separate the instance_path from the cache dir is to mount a different partition under $instances_path/$image_cache_subdirectory_name directory.

Workaround 3:
In case of non-edge deployments, where we don't need to fill the last piece of disk with VMs, one way to avoid this problem is to spread the VMs by free space available. The DiskWeigher in the scheduler does this by default and it is enabled by default.

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-05-11.log.html#t2020-05-11T15:12:29
[2] https://github.com/openstack/nova/blob/d6450879c7f7dd19366b6f002301fbbf87918026/nova/virt/libvirt/imagebackend.py#L585

[Balazs Gibizer (balazs-gibizer)]

OVH (aarents) confirmed that they saw this kind of problem. So I marked the bug Confirmed.

[Dan Smith (danms)]

Just to be clear, cache_images() already returns information about the current state of the requested image o the compute node. The change we could make would be a flag to cause it to *just* return that info and not also trigger caching of the image, which it currently will. Something like cache_images(check_only=True).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/727261

[Balazs Gibizer (balazs-gibizer)]

Option C) - disabling the cache is not viable with the current imagebackend codebase. The basic idea was to download the glance image directly into the instance directory instead of downloading it to the cache (_base). However even if the image is downloaded directly to the instance directory nova still needs to convert between image types. In an extreme case: qcow2 image in glance + force_raw_images=True config, + use_cow_images = True config means that the nova image fetch code downloads a qcow2 image from glance, then convert it to raw[1] due to force_raw_images, then the qcow2 imagebackend needs to convert such raw image back to qcow2. Such conversion uses double disk space temporarily so it fails in an almost fully allocated environment.

[1] https://github.com/openstack/nova/blob/ef3b5707328f460d0d03759061c01b1a20f28743/nova/virt/images.py#L137

[Balazs Gibizer (balazs-gibizer)]

Dan refined the idea of Option A) [1]:
1. Each time we cache an image, or run the periodic, we generate a disk allocation for the compute node uuid which consumes inventory according to how much space the cache is using

2. Same as above, but adjust the reserved amount.

Both cases need to consider the case where the _base is not on the same filesystem as the instances, but there's less synchronization involved, and we're not spraying tons of new allocations into place we could also make it a workaround that you opt into in the short term to see how it goes, because cleanup from it would be much easier (just nuke one $cnuuid allocation) and we could make compute node startup nuke that allocation if present and the workaround is disabled (or the self-correcting reserved amount)

Glance API provides the physical size of the image so nova-compute can try to allocate / reserve that amount of DISK_GB resource in placement _before_ download the image to the cache.

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-05-13.log.html#t2020-05-13T15:52:08

[Balazs Gibizer (balazs-gibizer)]

The complete solution for Option A) would consist of two major pieces:
1) Nova-compute maintains an DISK_GB allocation / reservation in placement for the current cache usage. See the comment #6 above for details

2) A pre-filter in the scheduler that modifies the allocation_candidate request to request flavor.disk* + image.size amount of DISK_GB resource, but then only allocate flavor.disk resources in placement for the instance.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Balazs Gibizer (<email address hidden>) on branch: master
Review: https://review.opendev.org/727261
Reason: It is a mighty dead end.

[Balazs Gibizer (balazs-gibizer)]

> need to consider the case where the _base is not on the same filesystem as the instances

>>> import os
>>> os.stat(CONF.instances_path).st_dev == os.stat(os.path.join(CONF.instances_path, CONF.image_cache.subdirectory_name)).st_dev

Note that os.stat() does not work for Windows as the st_dev always 0.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/728439

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/728439
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=89fe504abfbd41a9c37f9b544c0ce98b23b45799
Submitter: Zuul
Branch: master

commit 89fe504abfbd41a9c37f9b544c0ce98b23b45799
Author: Balazs Gibizer <email address hidden>
Date: Fri May 15 15:00:20 2020 +0200

    Reserve DISK_GB resource for the image cache

    If the nova compute image cache is on the same disk as the instance
    directory then the images in the cache will consume the same disk
    resource as the nova instances which can lead to disk overallocation.

    This patch adds a new config option
    [workarounds]/reserve_disk_resource_for_image_cache . If it is set the
    libvirt driver will reserve DISK_GB resources in placement for the
    actual size of the image cache.

    This is a low complexity solution with known limitations:
    * Reservation is updated long after the image is downloaded
    * This code allows the reservation to push the provider into negative
    available resource if the reservation + allocations exceed the total
    inventory.

    Change-Id: If874f018ea996587e178219569c2903c2ee923cf
    Closes-Bug: #1878024

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/729523

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/729350
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ab3fab03222a5e82ca96ed2a4e959ac7faa3f4fe
Submitter: Zuul
Branch: master

commit ab3fab03222a5e82ca96ed2a4e959ac7faa3f4fe
Author: Dan Smith <email address hidden>
Date: Tue May 19 11:27:03 2020 -0700

    Add admin doc information about image cache resource accounting

    This adds some details to the image cache page in the admin docs about
    how image cache disk usage is (not) considered in the scheduler disk
    space calculation. Workarounds and mitigation strategies are provided.

    Change-Id: I7f40f167cea073a73cf249a9adfd73e1187c031b
    Related-Bug: #1878024

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/729853

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/729523
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=968981b5853724a8225cfc16b04ea83b4f485a9a
Submitter: Zuul
Branch: stable/ussuri

commit 968981b5853724a8225cfc16b04ea83b4f485a9a
Author: Balazs Gibizer <email address hidden>
Date: Fri May 15 15:00:20 2020 +0200

    Reserve DISK_GB resource for the image cache

    If the nova compute image cache is on the same disk as the instance
    directory then the images in the cache will consume the same disk
    resource as the nova instances which can lead to disk overallocation.

    This patch adds a new config option
    [workarounds]/reserve_disk_resource_for_image_cache . If it is set the
    libvirt driver will reserve DISK_GB resources in placement for the
    actual size of the image cache.

    This is a low complexity solution with known limitations:
    * Reservation is updated long after the image is downloaded
    * This code allows the reservation to push the provider into negative
    available resource if the reservation + allocations exceed the total
    inventory.

    Change-Id: If874f018ea996587e178219569c2903c2ee923cf
    Closes-Bug: #1878024
    (cherry picked from commit 89fe504abfbd41a9c37f9b544c0ce98b23b45799)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/729853
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=57702ad4849a4ffc0c02df97269c8d7f44b57bae
Submitter: Zuul
Branch: stable/ussuri

commit 57702ad4849a4ffc0c02df97269c8d7f44b57bae
Author: Dan Smith <email address hidden>
Date: Tue May 19 11:27:03 2020 -0700

    Add admin doc information about image cache resource accounting

    This adds some details to the image cache page in the admin docs about
    how image cache disk usage is (not) considered in the scheduler disk
    space calculation. Workarounds and mitigation strategies are provided.

    Change-Id: I7f40f167cea073a73cf249a9adfd73e1187c031b
    Related-Bug: #1878024
    (cherry picked from commit ab3fab03222a5e82ca96ed2a4e959ac7faa3f4fe)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/730324

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/730326

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/730324
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=90c70f04d777e444eb8e2f0bccb8aa616e69dc66
Submitter: Zuul
Branch: stable/train

commit 90c70f04d777e444eb8e2f0bccb8aa616e69dc66
Author: Balazs Gibizer <email address hidden>
Date: Fri May 15 15:00:20 2020 +0200

    Reserve DISK_GB resource for the image cache

    If the nova compute image cache is on the same disk as the instance
    directory then the images in the cache will consume the same disk
    resource as the nova instances which can lead to disk overallocation.

    This patch adds a new config option
    [workarounds]/reserve_disk_resource_for_image_cache . If it is set the
    libvirt driver will reserve DISK_GB resources in placement for the
    actual size of the image cache.

    This is a low complexity solution with known limitations:
    * Reservation is updated long after the image is downloaded
    * This code allows the reservation to push the provider into negative
    available resource if the reservation + allocations exceed the total
    inventory.

    Conflicts:
      nova/conf/workarounds.py due to
      7c7a25aa1eda9b1815f12cce25dda0a840d562f1 is not in stable/train

      nova/virt/libvirt/driver.py due to
      I52fbbcac9dc386f24ee81b3321dd0d8355e01976 is not in stable/train

    Changes in imagechache.py, workarounds.py and test_imagecache.py are
    due to I3c49825ac0d70152b6c8ee4c8ca01546265f4b80 not in stable/train

    Change-Id: If874f018ea996587e178219569c2903c2ee923cf
    Closes-Bug: #1878024
    (cherry picked from commit 89fe504abfbd41a9c37f9b544c0ce98b23b45799)
    (cherry picked from commit 968981b5853724a8225cfc16b04ea83b4f485a9a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/730326
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=147f1c753a225a09485ddd2d575d489ad67428ae
Submitter: Zuul
Branch: stable/train

commit 147f1c753a225a09485ddd2d575d489ad67428ae
Author: Dan Smith <email address hidden>
Date: Tue May 19 11:27:03 2020 -0700

    Add admin doc information about image cache resource accounting

    This adds some details to the image cache page in the admin docs about
    how image cache disk usage is (not) considered in the scheduler disk
    space calculation. Workarounds and mitigation strategies are provided.

    Conflicts:
      doc/source/admin/image-caching.rst due to
      829ccbe2bbc4e80e92baf7553401d7925d883732 and
      828e8047e5c8651ea757bda7922670889d5e8818 is not in stable/train.

    Basically this patch is now a partial backport of
    829ccbe2bbc4e80e92baf7553401d7925d883732 squashed into
    57702ad4849a4ffc0c02df97269c8d7f44b57bae .

    Change-Id: I7f40f167cea073a73cf249a9adfd73e1187c031b
    Related-Bug: #1878024
    (cherry picked from commit ab3fab03222a5e82ca96ed2a4e959ac7faa3f4fe)
    (cherry picked from commit 57702ad4849a4ffc0c02df97269c8d7f44b57bae)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/732529

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/732542

[MarkMielke (mark-mielke)]

On Train, I tried enabling this feature and I saw these errors in the log:

2020-06-19 03:06:38.011 12118 ERROR nova.compute.manager FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/nova/instances/_base'

Was the case missed where base images do not always exist on every hypervisor?

In my case, these are brand new hypervisors, never used with ephemeral storage. Over time, I expect the "_base" would appear on each, and the error would disappear. But, I think there should be coverage for the case that hypervisors won't start with a "_base", and depending upon how machines are assigned, they may never get a "_base".

[Balazs Gibizer (balazs-gibizer)]

@Mark: Thanks for the feedback. I will try to reproduce the issue today. But if you have a full stack trace from the log then please paste it.

[Balazs Gibizer (balazs-gibizer)]

I managed to reproduce the problem locally and opened a separate bug[1] on it for tracking purposes.

[1]https://bugs.launchpad.net/nova/+bug/1884214

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/736964

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/736964
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a85753778f710f03616a682d294f8f638dea6baf
Submitter: Zuul
Branch: master

commit a85753778f710f03616a682d294f8f638dea6baf
Author: Balazs Gibizer <email address hidden>
Date: Fri Jun 19 11:18:04 2020 +0200

    Guard against missing image cache directory

    The original fix of bug 1878024 missed an edge case where on a fresh
    hypervisor the image cache directory hasn't been created yet. That
    directory is only created when the first image is downloaded.

    This patch makes sure that if the cache dir hasn't been created yet then
    0 disk is reserved for the cache usage instead of raising and logging an
    exception.

    Change-Id: Id1bbc955a9099de1abc11b9063fe177896646d03
    Related-Bug: #1878024
    Closes-Bug: #1884214

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/737990

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/737990
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a6a48e876c9c4f9a218de882270ef098d46bf767
Submitter: Zuul
Branch: stable/ussuri

commit a6a48e876c9c4f9a218de882270ef098d46bf767
Author: Balazs Gibizer <email address hidden>
Date: Fri Jun 19 11:18:04 2020 +0200

    Guard against missing image cache directory

    The original fix of bug 1878024 missed an edge case where on a fresh
    hypervisor the image cache directory hasn't been created yet. That
    directory is only created when the first image is downloaded.

    This patch makes sure that if the cache dir hasn't been created yet then
    0 disk is reserved for the cache usage instead of raising and logging an
    exception.

    Change-Id: Id1bbc955a9099de1abc11b9063fe177896646d03
    Related-Bug: #1878024
    Closes-Bug: #1884214
    (cherry picked from commit a85753778f710f03616a682d294f8f638dea6baf)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/738455

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/732529
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=86be251451716e3fa64f030c1f44954251bbee5d
Submitter: Zuul
Branch: stable/stein

commit 86be251451716e3fa64f030c1f44954251bbee5d
Author: Balazs Gibizer <email address hidden>
Date: Fri May 15 15:00:20 2020 +0200

    Reserve DISK_GB resource for the image cache

    If the nova compute image cache is on the same disk as the instance
    directory then the images in the cache will consume the same disk
    resource as the nova instances which can lead to disk overallocation.

    This patch adds a new config option
    [workarounds]/reserve_disk_resource_for_image_cache . If it is set the
    libvirt driver will reserve DISK_GB resources in placement for the
    actual size of the image cache.

    This is a low complexity solution with known limitations:
    * Reservation is updated long after the image is downloaded
    * This code allows the reservation to push the provider into negative
    available resource if the reservation + allocations exceed the total
    inventory.

    Conflicts:
      nova/conf/workarounds.py due to
      Ie38aa625dff543b5980fd437ad2febeba3b50079 ("Add support for translating CPU
      policy extra specs, image meta") is not in stable/stein

      nova/tests/unit/virt/libvirt/test_driver.py due to
      I25d70aa09080b22d1bfa0aa097f0a114de8bf15a ("Add reshaper for PCPU")
      is not in stable/stein

    Change-Id: If874f018ea996587e178219569c2903c2ee923cf
    Closes-Bug: #1878024
    (cherry picked from commit 89fe504abfbd41a9c37f9b544c0ce98b23b45799)
    (cherry picked from commit 968981b5853724a8225cfc16b04ea83b4f485a9a)
    (cherry picked from commit 90c70f04d777e444eb8e2f0bccb8aa616e69dc66)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/732542
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5429fd3d6e71dd2dd0e87c2055e55d69fc5023da
Submitter: Zuul
Branch: stable/stein

commit 5429fd3d6e71dd2dd0e87c2055e55d69fc5023da
Author: Dan Smith <email address hidden>
Date: Tue May 19 11:27:03 2020 -0700

    Add admin doc information about image cache resource accounting

    This adds some details to the image cache page in the admin docs about
    how image cache disk usage is (not) considered in the scheduler disk
    space calculation. Workarounds and mitigation strategies are provided.

    Basically this patch is now a partial backport of
    829ccbe2bbc4e80e92baf7553401d7925d883732 squashed into
    57702ad4849a4ffc0c02df97269c8d7f44b57bae .

    Change-Id: I7f40f167cea073a73cf249a9adfd73e1187c031b
    Related-Bug: #1878024
    (cherry picked from commit ab3fab03222a5e82ca96ed2a4e959ac7faa3f4fe)
    (cherry picked from commit 57702ad4849a4ffc0c02df97269c8d7f44b57bae)
    (cherry picked from commit 147f1c753a225a09485ddd2d575d489ad67428ae)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738455
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f26823c7805c96d155de6837a08ec34271117515
Submitter: Zuul
Branch: stable/train

commit f26823c7805c96d155de6837a08ec34271117515
Author: Balazs Gibizer <email address hidden>
Date: Fri Jun 19 11:18:04 2020 +0200

    Guard against missing image cache directory

    The original fix of bug 1878024 missed an edge case where on a fresh
    hypervisor the image cache directory hasn't been created yet. That
    directory is only created when the first image is downloaded.

    This patch makes sure that if the cache dir hasn't been created yet then
    0 disk is reserved for the cache usage instead of raising and logging an
    exception.

    Conflicts:
      nova/virt/libvirt/imagecache.py due to
      I3c49825ac0d70152b6c8ee4c8ca01546265f4b80 not in stable/train

    FileNotFoundError -> OSError changes in the test file is due to py2.7
    does not have FileNotFoundError.

    Change-Id: Id1bbc955a9099de1abc11b9063fe177896646d03
    Related-Bug: #1878024
    Closes-Bug: #1884214
    (cherry picked from commit a85753778f710f03616a682d294f8f638dea6baf)
    (cherry picked from commit a6a48e876c9c4f9a218de882270ef098d46bf767)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/740409

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740409
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ce1287aa7fdf56560fdc8a37c75cb5db3a10a07d
Submitter: Zuul
Branch: stable/stein

commit ce1287aa7fdf56560fdc8a37c75cb5db3a10a07d
Author: Balazs Gibizer <email address hidden>
Date: Fri Jun 19 11:18:04 2020 +0200

    Guard against missing image cache directory

    The original fix of bug 1878024 missed an edge case where on a fresh
    hypervisor the image cache directory hasn't been created yet. That
    directory is only created when the first image is downloaded.

    This patch makes sure that if the cache dir hasn't been created yet then
    0 disk is reserved for the cache usage instead of raising and logging an
    exception.

    Change-Id: Id1bbc955a9099de1abc11b9063fe177896646d03
    Related-Bug: #1878024
    Closes-Bug: #1884214
    (cherry picked from commit a85753778f710f03616a682d294f8f638dea6baf)
    (cherry picked from commit a6a48e876c9c4f9a218de882270ef098d46bf767)
    (cherry picked from commit f26823c7805c96d155de6837a08ec34271117515)