With libvirt/images_type = rbd, ephemeral instances silently ignore hw_qemu_guest_agent=yes

[Florian Haas]

Description
===========

If nova-compute is configured with libvirt/images_type = rbd, then instances booted off images with hw_qemu_guest_agent=yes do not invoke the qemu-ga guest-fsfreeze-freeze and guest-fsfreeze-thaw commands, and thus do not guarantee consistent snapshots. They also appear to silently ignore os_require_quiesce=yes.

Steps to reproduce
===========
The steps to verify whether or not the FIFREEZE and FITHAW ioctls are received by a guest are described in:

http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008648.html
https://xahteiwi.eu/resources/hints-and-kinks/ftrace-qemu-ga/

Expected result
===============

When you perform these described actions on an instance running on a compute node that does *not* set libvirt/images_type = rbd, then the FIFREEZE and FITHAW events are received as expected when the snapshot is created. This occurs irrespective of whether the instance is using boot-from-image, or boot-from-volume.

Actual result
=============

When you perform these described actions on an instance running on a compute node that *does* set libvirt/images_type = rbd, *and* the instance is set to boot from an image, then no qemu-ga events are received during snapshots at all.

The reason appears to be this direct_snapshot() call:

https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac92bfe75dc59/nova/virt/libvirt/driver.py#L2058

This is defined in
https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac92bfe75dc59/nova/virt/libvirt/imagebackend.py#L1055
and it uses RBD functionality only. Importantly, it never interacts with
qemu-ga, so it appears to not worry at all about freezing the filesystem.

This problem was apparently introduced in https://opendev.org/openstack/nova/commit/824c3706a3ea691781f4fcc4453881517a9e1c55.

However, the qemu-guest-agent calls *are* received correctly if the instance is configured to boot from volume.

Environment
===========

1. OpenStack release: Rocky (but this issue is present in current master).
2. Hypervisor: libvirt/KVM
3. Storage type: Ceph RBD
4. Networking: Neutron/ML2/OVS

Additional information
======================

A detailed discussion of the issue is available at:
https://<email address hidden>/thread/3YQCRO4JP56EDJN5KX5DWW5N2CSBHRHZ/?sort=date

[Matt Riedemann (mriedem)]

Boot from volume isn't affected since it doesn't go through the libvirt Rbd image backend class, the compute API makes a direct RPC call to the compute to quiesce the instance (if it's ACTIVE and configured to support that):

https://opendev.org/openstack/nova/src/commit/7bf75976016aae5d458eca9f6ddac92bfe75dc59/nova/compute/api.py#L3075

The direct snapshot functionality introduced in that referenced patch is optional - it was added to make snapshots with rbd-backed images significantly faster. You can disable the fast snapshot support by disabling direct URLs in glance as a workaround.

Obviously it would be nice to have the best of both worlds though, but I'm not sure what the fix for that looks like at this time.

[Florian Haas (fghaas)]

Thanks for looking into this!

Disabling direct URLs in Glance for this reason would be a bit like throwing the baby out with the bathwater, as you'd also lose the ability to use RBD clones for ephemeral storage and compute nodes would have to go back to streaming images down from Glance, if I'm not mistaken. (There would also be no rbd clone followed by rbd flatten on openstack volume create --image=<image> anymore, so that process would also take a whole lot longer, particularly for large images).

Jason and I came up with a different workaround involving yanking write permissions on the Glance pool from the CephX identity that Nova uses, see this message from the ceph-users ML:
https://<email address hidden>/message/KYFAF6BPRSI3XWANHKPIS656QX545LRD/

.. but that's basically just forcing the snapshot code into an error path while trying to create an RBD clone, so that's not very pretty either.

What bugs me is really primarily that os_require_quiesce is being silently ignored. So maybe the proper fix would be to have direct_snapshot() raise exception.ImageUnacceptable if the image meta includes os_require_quiesce?

[Jason Dillaman (jdillaman)]

> Obviously it would be nice to have the best of both worlds though, but I'm not sure what the fix for that looks like at this time.

Wouldn't it just be as simple as wrapping the 'direct_snapshot' call [1] in similar quiesce/unquiesce calls like '_live_snapshot' already does [2][3]?

[1] https://opendev.org/openstack/nova/src/branch/stable/stein/nova/virt/libvirt/driver.py#L2031
[2] https://opendev.org/openstack/nova/src/branch/stable/stein/nova/virt/libvirt/driver.py#L2256
[3] https://opendev.org/openstack/nova/src/branch/stable/stein/nova/virt/libvirt/driver.py#L2286

[Matt Riedemann (mriedem)]

> Wouldn't it just be as simple as wrapping the 'direct_snapshot' call [1] in similar quiesce/unquiesce calls like '_live_snapshot' already does [2][3]?

Yeah maybe, I think I'd want to see that code moved into a context manager or something though so we don't duplicate it since the snapshot and _live_snapshot methods in the libvirt driver are already very big and complicated.

[Florian Haas (fghaas)]

Possibly related: if os_require_quiesce is set, wouldn't it make sense to also freeze/thaw on suspend/resume, as opposed to just on snapshots?

[Florian Haas (fghaas)]

Sorry, I meant to put "would it make sense". Came out with a rather different connotation than intended; apologies.

[Balazs Gibizer (balazs-gibizer)]

Based on the above discussion I mark the bug Confirmed.