[18.04 FEAT] libvirt: libvirt support for QEMU live patching

[bugproxy]

== Comment: #0 - Frank P. Novak <email address hidden> - 2016-06-29 18:12:57 ==

libvirt support for QEMU live patching..

 QEMU hot-patching
Should be just {sic} a matter of updating the RPM package and, then, live migrating the VMs to another QEMU instance on the same host (which would point to the just installed new QEMU executable).

It is typcial for a software deployed in production to undergo multiple updates during its lifecycle. Updates typically are done to fix bugs, apply security fixes and add new features. Many a times, these updates involve some level of downtime for business as the software deployment needs to be brought down, newer versions need to be installed and it needs to be brought up
again live. If this downtime is not tolerable, then the software deployment has to be updated live without bringing it down at all. This method of patching the running software instance to mostly fix bugs is usually referred to as hot patching.
QEMU hot-patching

QEMU is an important component in KVM virtualization that provides virtual machine environment for the virtual machines (VM) or guests. QEMU does much of the heavy lifting in KVM virtualization and is used by the user (typically by libvirt or direct user) with plethora of command line options to start a virtual machine of a given flavour and required configuration.

Updating a running instance of QEMU to a newer version with the virtual machine experiencing near zero downtime is now possible with the recent developments in QEMU as well as Linux Kernel. This is extremely useful in virtualization deployments where the software solution deployed on virtual machines will not be aware of the updates that are done to the underlying hypervisor infrastructure (QEMU).

QEMU hot patching can essentially be achieved by live migrating a VM running on a buggy instance of QEMU to an instance that has fixes applied. However this has two problems:

- Migration involves copying over the entire RAM of the guest from source to destination instance and this can take quite a while to complete.
- If guest RAM is getting dirtied at a fast rate, there can be situations where the migration just doesn't complete as it keeps copying over the newly dirtied RAM pages from source to the destination.

These two issues have been addressed by a new migration technique in QEMU called the postcopy migration.
Userfaultfd and postcopy migration

Page faults have always been handled entirely by the Kernel, but a new system call named userfaultfd added to Linux Kernel changes it. userfaultfd allows for memory externalization: a user space program can create a file descriptor (FD) using this call and register regions of process' virtual address space for which page faults can be handled by user space by talking to the Kernel using the userfaultfd protocol.

One of the main motivations for this system call was to allow QEMU to achieve what is called postcopy kind of VM migration. Live migrating a VM from source host to destination host involves among other things, copying the contents of guest RAM from source to destination host over the network. Since the guest RAM can keep changing, copying becomes an iterative process which continues until a convergence is reached at which point the VM becomes live at the destination and is brought down at the source. If the guest memory is being dirtied at a rate which is more than the rate at which guest memory pages are transferred from source of migration, the live migration would never terminate and this is where postcopy migration helps.

In postcopy migration, VM becomes available at the destination as soon as some minimal required data is transferred from source to destination. Bulk of the guest memory is transferred on demand from source to destination whenever guest requires those pages. QEMU registers the RAM areas of the guest with userfaultfd and whenever there is page fault in the destination host for any page from guest's RAM space, the required page is brought over from source host. Thus postcopy migration in QEMU uses userfaultfd system call to achive a quick and time bound live migration of guest.
QEMU hot patching using postcopy migration

The quick and time bound way to migrate a VM between two hosts using postcopy migration opens up an interesting use case when the source and the destination hosts are the same. If postcopy migration is used to migrate a guest from one instance to another within the same host, we could effectively achieve QEMU hot-patching. If VM is being run using QEMU which has a known bug, then we could start another instance with updated QEMU (incoming instance) and migrate the VM from the source to the destination instance on the same host using
postcopy migration. Given that postcopy migration doesn't copy all the guest RAM upfront, we can migrate the VM from one instance to another with practically zero downtime. So we can use postcopy migration within the same host to dynamically update the QEMU thus achieving QEMU hot-patching.
Pre requisites

- userfaultfd system call is part of Linux Kernel and will be available from kernel version 4.3 onwards.

- Postcopy migration support is under development and will hopefully be available from QEMU-2.5 onwards.
Steps involved

- Assume a VM is running on a buggy instance of qemu.
  # qemu-system-ppc64 ....
- Start a new incoming instance on the same host.
  # qemu-system-ppc64 .... -incoming tcp:localhost:4444
- Post-copy migrate the exising VM
  (qemu) migrate_set_capability x-postcopy-ram on
  (qemu) migrate -d tcp:localhost:4444
  (qemu) migrate_start_postcopy
- After this the VM will be up and running with the fixed version of QEMU.
Limitations

- The destination instance of QEMU that we use can't vary widely from the source instance and should conform to all the requirements that should be met for VM migration.
- Support for postcopy migration isn't present in libvirt yet. Even when it is supported, libvirt doesn't support the notion of VM migration within the same host. Some development effort in addition to just postcopy migration is needed in libvirt to support hot-patching.

[Christian Ehrhardt  (paelzer)]

Hi,
first of all thanks for sharing a nice extra use case.
A few comments:

"Given that postcopy migration doesn't copy all the guest RAM upfront, we can migrate the VM from one instance to another with practically zero downtime. So we can use postcopy migration within the same host to dynamically update the QEMU thus achieving QEMU hot-patching.
Pre requisites"

Zero "full-stop" downtime that is true, but please if you e.g. announce/advertise this to anybody make it clear that the guest will have through a phase of long-latency page-faults.
That is not that long on a local system, but still perceivable in the guest - so it is great, but not a 100% free hot-patch.

"Support for postcopy migration isn't present in libvirt yet. Even when it is supported, libvirt doesn't support the notion of VM migration within the same host. Some development effort in addition to just postcopy migration is needed in libvirt to support hot-patching."

Yes, I agree given an official support in Libvirt this might be a nice and fast local update.
Once that is done upstream in libvirt I'd absolutely love to test/exploit this.

Until then let me share a trick how you can achieve this today via a smart setup - you might use that for your tests while working on this.
You can drive KVM from a LXD container basics are at [1] and a full test/dev example can be [2].
With that set up you can on a single host migrate from old/new qemu, but even more than that you can exchange effectively everything except the kernel which is shared by the containers.

Eventually an implementation in libvirt is much better, but I thought until then this might help you.

[1]: https://insights.ubuntu.com/2016/03/22/lxd-2-0-your-first-lxd-container/
[2]: http://paste.ubuntu.com/24622647/

[Christian Ehrhardt  (paelzer)]

This is more than a feature request (which would be "wishlist" in general), it is a not-yet-developed feature request (which would be "opinion"). But since I really like the case and think it would help Ubuntu users I'll start with "wishlist".

[Manoj Iyer (manjo)]

Since this feature is not yet under development, marking as invalid, and please reopen a bug report once patches are available to support this feature request.