Steps with test packages on Focal (shutdown-on-runtime)
---

Environment:
---

On top of LXD VM in comments #12/#13.

Enable PPA & debug symbols

	sudo add-apt-repository -yn ppa:mfo/lp2059272
	sudo sed '/^deb / s,$, main/debug,' -i /etc/apt/sources.list.d/mfo-ubuntu-lp2059272-focal.list
	sudo apt update

Install packages

	sudo apt install --yes libvirt{0,-daemon{,-driver-qemu}}{,-dbgsym} libvirt-clients gdb qemu-system-x86

	$ dpkg -s libvirt-daemon | grep ^Version:
	Version: 6.0.0-0ubuntu8.18~ppa1

Libvirtd debug logging

	cat <<EOF | sudo tee -a /etc/libvirt/libvirtd.conf
	log_filters="1:qemu 1:libvirt"
	log_outputs="3:syslog:libvirtd 1:file:/var/log/libvirt/libvirtd-debug.log"
	EOF

Follow `Steps to reproduce on Focal (shutdown-on-runtime)` in comment #12
---

Up to ...

Check the backtrace of the domain status XML save function, coming from device deleted event

	Thread 20 "libvirtd" hit Breakpoint 3, virDomainObjSave (obj=0x7fb3202733b0, xmlopt=0x7fb3200400d0, statusDir=0x7fb32000a620 "/run/libvirt/qemu") at ../../../src/conf/domain_conf.c:29157
	29157   ../../../src/conf/domain_conf.c: No such file or directory.
	(gdb) bt
	#0  virDomainObjSave (obj=0x7fb3202733b0, xmlopt=0x7fb3200400d0, statusDir=0x7fb32000a620 "/run/libvirt/qemu") at ../../../src/conf/domain_conf.c:29157
	#1  0x00007fb32c20d627 in processDeviceDeletedEvent (devAlias=0x564ae9f032e0 "input0", vm=0x7fb3202733b0, driver=0x7fb320039ed0) at ../../../src/qemu/qemu_driver.c:4367
	#2  qemuProcessEventHandler (data=0x564ae9fc88f0, opaque=0x7fb320039ed0) at ../../../src/qemu/qemu_driver.c:4943
	#3  0x00007fb333ed51af in virThreadPoolWorker (opaque=opaque@entry=0x564ae9fd3080) at ../../../src/util/virthreadpool.c:163
	#4  0x00007fb333ed451c in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:196
	#5  0x00007fb333b95609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
	#6  0x00007fb333aba353 in clone () from /lib/x86_64-linux-gnu/libc.so.6

	$ sudo kill $(pidof libvirtd)

	Thread 1 "libvirtd" hit Breakpoint 1, qemuStateCleanup () at ../../../src/qemu/qemu_driver.c:1180

	t 20

	(gdb) p xmlopt.privateData.format
	$1 = (virDomainXMLPrivateDataFormatFunc) 0x7fb32c167890 <qemuDomainObjPrivateXMLFormat>

	t 1
	c &

Check the formatter/options again; it is *STILL* referenced, not 0x0 anymore:

	t 20

	(gdb) p xmlopt.privateData.format
	$2 = (virDomainXMLPrivateDataFormatFunc) 0x7fb32c167890 <qemuDomainObjPrivateXMLFormat>

Check the shutdown/cleanup thread is waiting for it,
in the path to free the worker thread pool:

	(gdb) i th 1
	  Id   Target Id                                   Frame
	  1    Thread 0x7fb33007bb40 (LWP 6585) "libvirtd" (running)

	t 1
	interrupt

	(gdb) bt
	#0  0x00007fb333b9c376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
	#1  0x00007fb333ed479b in virCondWait (c=<optimized out>, m=<optimized out>) at ../../../src/util/virthread.c:144
	#2  0x00007fb333ed5438 in virThreadPoolFree (pool=<optimized out>) at ../../../src/util/virthreadpool.c:286
	#3  0x00007fb32c1d89e3 in qemuStateCleanup () at ../../../src/qemu/qemu_driver.c:1186
	#4  0x00007fb33407246f in virStateCleanup () at ../../../src/libvirt.c:669
	#5  0x0000564ae98babc8 in main (argc=<optimized out>, argv=<optimized out>) at ../../../src/remote/remote_daemon.c:1447

Let the save function continue, and libvirt finishes shutting down:

	(gdb) c &
	Continuing.
	(gdb) t 20
	(gdb) c
	[Inferior 1 (process 6585) exited normally]
	(gdb) q

Check the VM status XML *after*:

	$ sudo grep -e '<domstatus' -e '<domain' -e 'monitor path' /run/libvirt/qemu/test-vm.xml
	<domstatus state='running' reason='booted' pid='6634'>
	  <monitor path='/var/lib/libvirt/qemu/domain-1-test-vm/monitor.sock' type='unix'/>
	  <domain type='qemu' id='1'>

It *still* has the 'monitor path' tag/field.

Now, the next time libvirtd starts, it correctly parses that XML:

	$ sudo systemctl start libvirtd.service

	$ journalctl -b -u libvirtd.service | grep -A1 error
	Mar 30 22:27:20 lp2059272-focal libvirtd[6670]: 6686: error : dnsmasqCapsRefreshInternal:714 : Cannot check dnsmasq binary /usr/sbin/dnsmasq: No such file or directory

And libvirt is aware of the domain, and can manage it:

	$ virsh list
	 Id   Name      State
	-------------------------
	 1    test-vm   running

	$ virsh destroy test-vm
	Domain test-vm destroyed