nova placement problems with low compute node storage available

[Drew Freiberger]

Nova-scheduler cannot identify placement host when raw image size is greater than available local storage, even when using raw image, show_image_direct_url=1, and ceph ephemeral storage, such that glance images would never land on local node storage.

Have following configuration for glance:
  juju config glance api-config-flags='show_image_direct_url=true'
  juju config glance registry-config-flags='show_image_direct_url=true'

Worked around issue by allowing storage over-commit using:
  juju config nova-cloud-controller config-flags="disk_allocation_ratio=5.0"

This is workaround is not ideal when working with qcow images which will require local nova-compute node storage for raw conversion.

nova-cloud-controller and nova-compute logs attached

[Drew Freiberger (afreiberger)]

For clarity, this issue was found on Mitaka

[James Page (james-page)]

Pretty sure this is not a charm specific issue, so raising bug task against nova.

[Andrey Volkov (avolkov)]

Could you please provide logs and steps for reproducing?

[Drew Freiberger (afreiberger)]

Sorry for lag in response. Logs not handy, but to reproduce:

Set libvirt-image-backend for nova to rbd (ceph)

Create raw bootable image in glance of size 100GB (ensuring glance is ceph rbd backed)

Fill all available hypervisors such that /var/lib/nova/instances/_base has less than 100GB free

set disk_allocation_ratio = 1.0 or less

recycle nova-scheduler/nova-conductor services and attempt to boot image as new server and you'll find that it can't find a suitable hypervisor to deploy.

If you kick disk_allocation_ratio to something that would allow the 100GB image to fit within the free-space with overcommit, then the same image will boot and glance rbd and nova rbd backend work together to never move data to /var/lib/nova/instances/_base for conversion (as it just creates an rbd snapshot+COW for the new nova rbd object.

If glance and nova libvirt backend both == rbd and show_image_direct_url == true and image-format == raw, disk_allocation_ratio check should not be made

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

[melanie witt (melwitt)]

This is unfortunately a known issue in nova for several years now, as nova historically has not had proper support for shared storage (there are many gaps).

We had the same issue for boot-from-volume servers until recently [1] and there's a bug open related to it [2].

For shared storage support, we have been working on it but making slow progress. You can find some information about it in this bug: https://bugs.launchpad.net/nova/+bug/1784020 and on our Stein PTG etherpad at L219: https://etherpad.openstack.org/p/nova-ptg-stein

Because of higher priority work happening this cycle and the fact that this issue is latent, I'm going to set this bug to Medium importance.

[1] https://launchpad.net/bugs/1469179
[2] https://launchpad.net/bugs/1796737