I'm partially ok with this, it explains: 1. Why it isn't restored in the first place 2. Why our experiments in comment 5 behaved so different as the hot add/remove was a qcow that was not readonly, due to that it behaved differently than cdrom insert (which automatically is readonly). To be sure I combined all former experiments in one and re-run them checking if it all is consistent in regard to the explanation we got. The last element of confusion out of our experiments was about the nbd (comment 7) being changed back to root:root is also clear now. This reminded me of bug 691590 where we fixed this for "normal" files. /dev/nbd is stored on devfs which has no xattr support that is needed to "remember" the owner. I'll bring that up upstream as well for awareness. => https://gitlab.com/libvirt/libvirt/-/issues/517 Notes on full repro of all cases: # Prep different kind of potential attachments ``` # 1. Create a libvirt guest your preferred way # 2. get an iso of your choice $ wget https://releases.ubuntu.com/22.04.2/ubuntu-22.04.2-live-server-amd64.iso $ sudo mv ubuntu-22.04.2-live-server-amd64.iso /var/lib/libvirt/images/ $ sudo ls -laF /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso -rw-rw-r-- 1 ubuntu ubuntu 1975971840 Feb 17 21:57 /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso # 3. Create test image $ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test.qcow2 100M $ ll /var/lib/libvirt/images/test.qcow2 -rw-r--r-- 1 root root 196616 Aug 7 08:58 /var/lib/libvirt/images/test.qcow2 # 4. Create test to be available as disk $sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test-nbd.qcow2 100M $ sudo qemu-nbd --connect=/dev/nbd0 /var/lib/libvirt/images/test-nbd.qcow2 $ sudo sgdisk -n 0:0:0 /dev/nbd0 Creating new GPT entries in memory. The operation has completed successfully. $ ll /dev/nbd0 brw-rw---- 1 root disk 43, 0 Aug 7 09:00 /dev/nbd0 ``` So after this we have: - an .iso that is user owned - a .qcow2 that is root:root - a block device that is root:disk In one shot that looks like: ``` $ sudo ls -laF /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso /var/lib/libvirt/images/test.qcow2 /dev/nbd0 brw-rw---- 1 root disk 43, 0 Aug 7 09:00 /dev/nbd0 -rw-r--r-- 1 root root 196616 Aug 7 08:58 /var/lib/libvirt/images/test.qcow2 -rw-rw-r-- 1 ubuntu ubuntu 1975971840 Feb 17 21:57 /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso ``` # Variant A - start and stop statically with the guest Guest XML section for those devices: ``` ``` Pre ``` $ sudo ls -laF /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso /var/lib/libvirt/images/test.qcow2 /dev/nbd0 brw-rw---- 1 root disk 43, 0 Aug 7 09:00 /dev/nbd0 -rw-r--r-- 1 root root 196616 Aug 7 08:58 /var/lib/libvirt/images/test.qcow2 -rw-rw-r-- 1 ubuntu ubuntu 1975971840 Feb 17 21:57 /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso ``` Running ``` $ sudo ls -laF /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso /var/lib/libvirt/images/test.qcow2 /dev/nbd0 brw-rw---- 1 root disk 43, 0 Aug 7 09:00 /dev/nbd0 -rw-r--r-- 1 libvirt-qemu kvm 196616 Aug 7 08:58 /var/lib/libvirt/images/test.qcow2 -rw-rw-r-- 1 libvirt-qemu kvm 1975971840 Feb 17 21:57 /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso ``` Post ``` sudo ls -laF /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso /var/lib/libvirt/images/test.qcow2 /dev/nbd0 brw-rw---- 1 root root 43, 0 Aug 7 09:26 /dev/nbd0 -rw-r--r-- 1 root root 196616 Aug 7 08:58 /var/lib/libvirt/images/test.qcow2 -rw-rw-r-- 1 libvirt-qemu kvm 1975971840 Feb 17 21:57 /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso ``` As expected that changed the qcow file, left disk ndb0 untouched and does not restore the iso file. So far all behaves as explained. # Variant B - hot add/remove Keep this in the guest config (cdrom without disk inserted). Remove the qcow and nbd devices. ```
``` Use the XML shown above in the guest for test-qcow2.xml and test-nbd.xml files used in hot add/remove. ``` $ for f in test-*; do echo "File $f:"; cat $f; done File test-nbd.xml: File test-qcow2.xml: ``` Now hot-add all of those in their way ``` $ virsh change-media j sda /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso --insert Successfully inserted media. $ virsh attach-device j test-qcow2.xml Device attached successfully $ virsh attach-device j test-nbd.xml Device attached successfully ``` Pre ``` ubuntu@node-horsea:~$ sudo ls -laF /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso /var/lib/libvirt/images/test.qcow2 /dev/nbd0 brw-rw---- 1 root disk 43, 0 Aug 7 09:26 /dev/nbd0 -rw-r--r-- 1 root root 196616 Aug 7 08:58 /var/lib/libvirt/images/test.qcow2 -rw-rw-r-- 1 ubuntu ubuntu 1975971840 Feb 17 21:57 /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso ``` Running ``` $ sudo ls -laF /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso /var/lib/libvirt/images/test.qcow2 /dev/nbd0 brw-rw---- 1 root disk 43, 0 Aug 7 09:26 /dev/nbd0 -rw-r--r-- 1 libvirt-qemu kvm 196616 Aug 7 08:58 /var/lib/libvirt/images/test.qcow2 -rw-rw-r-- 1 libvirt-qemu kvm 1975971840 Feb 17 21:57 /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso ``` Post ``` $ sudo ls -laF /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso /var/lib/libvirt/images/test.qcow2 /dev/nbd0 brw-rw---- 1 root disk 43, 0 Aug 7 09:26 /dev/nbd0 -rw-r--r-- 1 root root 196616 Aug 7 08:58 /var/lib/libvirt/images/test.qcow2 -rw-rw-r-- 1 libvirt-qemu kvm 1975971840 Feb 17 21:57 /var/lib/libvirt/images/ubuntu-22.04.2-live-server-amd64.iso ```