stateful snapshot not working

[Ioanna Alifieraki]

lxc snapshot --stateful fails with
"Error: Migration is disabled when VirtFS export path '/var/lib/lxd/devices/f-vm/config.mount' is mounted in the guest using mount_tag 'config'"

Reproducer:
$ lxc profile show default
config:
  limits.memory: 4096MiB
  migration.stateful: "true"
description: Default LXD profile
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    size.state: 4096MiB
    type: disk
name: default
used_by: []

$ lxc launch ubuntu:focal f-vm --vm
$ lxc snapshot --stateful f-vm
Error: Migration is disabled when VirtFS export path '/var/lib/lxd/devices/f-vm/config.mount' is mounted in the guest using mount_tag 'config'

Versions affected: 4.24, 5.0, latest upstream (at commit 837d8228b45010bcbeecf581720e2ce7f17f04dc)

[Ioanna Alifieraki (joalif)]

Some initial analysis I've done.

The error "Error: Migration is disabled when VirtFS export path '/var/lib/lxd/devices/f-vm/config.mount' is mounted in the guest using mount_tag 'config'" comes from qemu.
IIUC qemu returns this error because 9p driver is in use
(see https://gitlab.com/qemu-project/qemu/-/commit/e9a0152ba182c5ce9929ee60f83a37b61a1d5195 ).

In lxd logs I can see :
"WARNING[2022-05-25T17:14:46Z] Unable to use virtio-fs for config drive, using 9p as a fallback err="Stateful migration unsupported" instance=f-vm instanceType=virtual-machine project=default".

This warning comes from [1] because DiskVMVirtiofsdStart returns error [2].
IIUC DiskVMVirtiofsdStart returns error because stateful.migration is set to true and according to [3],
"To allow for this, the VM must have migration.stateful set to true and size.state set on its root disk device to allow
for additional space to store the runtime state. In this mode, PCI passthrough will be disabled as will be virtiofs."
However, when falling back to 9p also does not support stateful migration.

[1] https://github.com/lxc/lxd/blob/master/lxd/instance/drivers/driver_qemu.go#L1198
[2] https://github.com/lxc/lxd/blob/837d8228b45010bcbeecf581720e2ce7f17f04dc/lxd/device/device_utils_disk.go#L447
[3] https://discuss.linuxcontainers.org/t/lxd-4-12-has-been-released/10424#stateful-stop-and-stateful-snapshots-for-virtual-machines-4

[Stéphane Graber (stgraber)]

Hello,

We usually prefer for support questions to be asked at https://discuss.linuxcontainers.org and issues be filed at https://github.com/lxc/lxd/issues.

We only really look at Launchpad for bugs in the LXD deb which these days is limited to Ubuntu 18.04 as everything afterwards is using the snap.

In this particular instance, you're most likely dealing with an outdated LXD agent setup which is keeping the 9p mount active and preventing the stateful snapshot.

You may be able to update the setup by running:
 - cd /run/lxd_agent
 - ./install.sh
 - reboot

This should refresh the systemd units used to start the agent with a current version.

[Ioanna Alifieraki (joalif)]

@Stephane, thank you for reply, I'll do what you suggest and I'll open an issue in github (if I confirm this is a bug)