Comment 14 for bug 2059272

Steps with test packages on Focal (shutdown-on-runtime)
---

Environment:
---

On top of LXD VM in comments #12/#13.

Enable PPA & debug symbols

 sudo add-apt-repository -yn ppa:mfo/lp2059272
 sudo sed '/^deb / s,$, main/debug,' -i /etc/apt/sources.list.d/mfo-ubuntu-lp2059272-focal.list
 sudo apt update

Install packages

 sudo apt install --yes libvirt{0,-daemon{,-driver-qemu}}{,-dbgsym} libvirt-clients gdb qemu-system-x86

 $ dpkg -s libvirt-daemon | grep ^Version:
 Version: 6.0.0-0ubuntu8.18~ppa1

Libvirtd debug logging

 cat <<EOF | sudo tee -a /etc/libvirt/libvirtd.conf
 log_filters="1:qemu 1:libvirt"
 log_outputs="3:syslog:libvirtd 1:file:/var/log/libvirt/libvirtd-debug.log"
 EOF

Follow `Steps to reproduce on Focal (shutdown-on-runtime)` in comment #12
---

Up to ...

Check the backtrace of the domain status XML save function, coming from device deleted event

 Thread 20 "libvirtd" hit Breakpoint 3, virDomainObjSave (obj=0x7fb3202733b0, xmlopt=0x7fb3200400d0, statusDir=0x7fb32000a620 "/run/libvirt/qemu") at ../../../src/conf/domain_conf.c:29157
 29157 ../../../src/conf/domain_conf.c: No such file or directory.
 (gdb) bt
 #0 virDomainObjSave (obj=0x7fb3202733b0, xmlopt=0x7fb3200400d0, statusDir=0x7fb32000a620 "/run/libvirt/qemu") at ../../../src/conf/domain_conf.c:29157
 #1 0x00007fb32c20d627 in processDeviceDeletedEvent (devAlias=0x564ae9f032e0 "input0", vm=0x7fb3202733b0, driver=0x7fb320039ed0) at ../../../src/qemu/qemu_driver.c:4367
 #2 qemuProcessEventHandler (data=0x564ae9fc88f0, opaque=0x7fb320039ed0) at ../../../src/qemu/qemu_driver.c:4943
 #3 0x00007fb333ed51af in virThreadPoolWorker (opaque=opaque@entry=0x564ae9fd3080) at ../../../src/util/virthreadpool.c:163
 #4 0x00007fb333ed451c in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:196
 #5 0x00007fb333b95609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
 #6 0x00007fb333aba353 in clone () from /lib/x86_64-linux-gnu/libc.so.6

 $ sudo kill $(pidof libvirtd)

 Thread 1 "libvirtd" hit Breakpoint 1, qemuStateCleanup () at ../../../src/qemu/qemu_driver.c:1180

 t 20

 (gdb) p xmlopt.privateData.format
 $1 = (virDomainXMLPrivateDataFormatFunc) 0x7fb32c167890 <qemuDomainObjPrivateXMLFormat>

 t 1
 c &

Check the formatter/options again; it is *STILL* referenced, not 0x0 anymore:

 t 20

 (gdb) p xmlopt.privateData.format
 $2 = (virDomainXMLPrivateDataFormatFunc) 0x7fb32c167890 <qemuDomainObjPrivateXMLFormat>

Check the shutdown/cleanup thread is waiting for it,
in the path to free the worker thread pool:

 (gdb) i th 1
   Id Target Id Frame
   1 Thread 0x7fb33007bb40 (LWP 6585) "libvirtd" (running)

 t 1
 interrupt

 (gdb) bt
 #0 0x00007fb333b9c376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
 #1 0x00007fb333ed479b in virCondWait (c=<optimized out>, m=<optimized out>) at ../../../src/util/virthread.c:144
 #2 0x00007fb333ed5438 in virThreadPoolFree (pool=<optimized out>) at ../../../src/util/virthreadpool.c:286
 #3 0x00007fb32c1d89e3 in qemuStateCleanup () at ../../../src/qemu/qemu_driver.c:1186
 #4 0x00007fb33407246f in virStateCleanup () at ../../../src/libvirt.c:669
 #5 0x0000564ae98babc8 in main (argc=<optimized out>, argv=<optimized out>) at ../../../src/remote/remote_daemon.c:1447

Let the save function continue, and libvirt finishes shutting down:

 (gdb) c &
 Continuing.
 (gdb) t 20
 (gdb) c
 [Inferior 1 (process 6585) exited normally]
 (gdb) q

Check the VM status XML *after*:

 $ sudo grep -e '<domstatus' -e '<domain' -e 'monitor path' /run/libvirt/qemu/test-vm.xml
 <domstatus state='running' reason='booted' pid='6634'>
   <monitor path='/var/lib/libvirt/qemu/domain-1-test-vm/monitor.sock' type='unix'/>
   <domain type='qemu' id='1'>

It *still* has the 'monitor path' tag/field.

Now, the next time libvirtd starts, it correctly parses that XML:

 $ sudo systemctl start libvirtd.service

 $ journalctl -b -u libvirtd.service | grep -A1 error
 Mar 30 22:27:20 lp2059272-focal libvirtd[6670]: 6686: error : dnsmasqCapsRefreshInternal:714 : Cannot check dnsmasq binary /usr/sbin/dnsmasq: No such file or directory

And libvirt is aware of the domain, and can manage it:

 $ virsh list
  Id Name State
 -------------------------
  1 test-vm running

 $ virsh destroy test-vm
 Domain test-vm destroyed