Comment 28 for bug 1874647

------- Comment From <email address hidden> 2020-09-16 08:38 EDT-------
Edited /etc/apt/sources.list by duplicating all lines containing groovy-update and replacing it with groovy-proposed.
Running "apt update" and "apt upgrade"
Rebooting host

Seems like libvirt remained unchanged and qemu and kernel updated
# apt list --installed | grep -e libvirt -e qemu -e linux-image

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

gir1.2-libvirt-glib-1.0/groovy,now 3.0.0-1 s390x [installed,automatic]
libvirt-clients/groovy,now 6.6.0-1ubuntu2 s390x [installed]
libvirt-daemon-driver-qemu/groovy,now 6.6.0-1ubuntu2 s390x [installed,automatic]
libvirt-daemon-system-systemd/groovy,now 6.6.0-1ubuntu2 s390x [installed,automatic]
libvirt-daemon-system/groovy,now 6.6.0-1ubuntu2 s390x [installed,automatic]
libvirt-daemon/groovy,now 6.6.0-1ubuntu2 s390x [installed]
libvirt-dev/groovy,now 6.6.0-1ubuntu2 s390x [installed]
libvirt-glib-1.0-0/groovy,now 3.0.0-1 s390x [installed,automatic]
libvirt0/groovy,now 6.6.0-1ubuntu2 s390x [installed,automatic]
linux-image-5.4.0-21-generic/now 5.4.0-21.25 s390x [installed,local]
linux-image-5.4.0-47-generic/now 5.4.0-47.51 s390x [installed,local]
linux-image-5.8.0-18-generic/groovy,now 5.8.0-18.19 s390x [installed,automatic]
linux-image-5.8.0-19-generic/groovy-proposed,groovy-proposed,now 5.8.0-19.20 s390x [installed,automatic]
linux-image-generic/groovy-proposed,groovy-proposed,now 5.8.0.19.23 s390x [installed,automatic]
linux-image-unsigned-5.4.0-9019-generic/now 5.4.0-9019.23 s390x [installed,local]
python3-libvirt/groovy,now 6.1.0-1 s390x [installed,automatic]
qemu-block-extra/groovy-proposed,groovy-proposed,now 1:5.0-5ubuntu8 s390x [installed,automatic]
qemu-kvm/groovy-proposed,groovy-proposed,now 1:5.0-5ubuntu8 s390x [installed]
qemu-system-common/groovy-proposed,groovy-proposed,now 1:5.0-5ubuntu8 s390x [installed,automatic]
qemu-system-data/groovy-proposed,groovy-proposed,now 1:5.0-5ubuntu8 all [installed,automatic]
qemu-system-s390x/groovy-proposed,groovy-proposed,now 1:5.0-5ubuntu8 s390x [installed]
qemu-utils/groovy-proposed,groovy-proposed,now 1:5.0-5ubuntu8 s390x [installed,automatic]
qemu/groovy-proposed,groovy-proposed,now 1:5.0-5ubuntu8 s390x [installed]

Protected virtualization was enabled on the system before and after upgrade with kernel parameter "prot_virt=1".
Libvirts cached capabilities file was updated due to new kernel booting and virt-host-validate returned "pass" for "QEMU: Checking for secure guest support".
Unexpectedly starting an SE guest with console (virsh start guest01 --console) resulted in a crash of the guest.

Here is what I caught in the log:
2020-09-16 10:48:14.805+0000: starting up libvirt version: 6.6.0, package: 1ubuntu2 (Christian Ehrhardt <email address hidden> Tue, 25 Aug 2020 14:53:26 +0200), qemu version: 5.0.0Debian 1:5.0-5ubuntu8, kernel: 5.8.0-19-generic, hostname: linux02.
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
HOME=/var/lib/libvirt/qemu/domain-1-focal \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-focal/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-focal/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-focal/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-s390x \
-name guest=focal,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-focal/master-key.aes \
-machine s390-ccw-virtio-5.0,accel=kvm,usb=off,dump-guest-core=off,loadparm=5 \
-cpu gen15b-base,aen=on,cmmnt=on,vxpdeh=on,aefsi=on,csske=on,mepoch=on,msa9=on,msa8=on,msa7=on,msa6=on,msa5=on,msa4=on,msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,deflate=on,edat2=on,unpack=on,etoken=on,vx=on,ipter=on,mepochptff=on,ap=on,vxeh=on,vxpd=on,esop=on,vxeh2=on,esort=on,apqi=on,apft=on,iep=on,apqci=on,cte=on,ais=on,bpb=on,gs=on,ppa15=on,zpci=on,sea_esop2=on,te=on,cmm=on \
-m 2000 \
-overcommit mem-lock=off \
-smp 4,sockets=4,cores=1,threads=1 \
-object iothread,id=iothread1 \
-object iothread,id=iothread2 \
-uuid fa435f71-7724-4ff1-8515-77ce1d0f22d1 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=28,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/focal.qcow2","aio":"native","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device virtio-blk-ccw,iothread=iothread1,iommu_platform=on,devno=fe.0.0000,drive=libvirt-1-format,id=virtio-disk0,bootindex=1,write-cache=on \
-netdev tap,fd=30,id=hostnet0 \
-device virtio-net-ccw,netdev=hostnet0,id=net0,mac=52:54:00:36:d3:88,devno=fe.0.0001,iommu_platform=on \
-chardev pty,id=charconsole0 \
-device sclpconsole,chardev=charconsole0,id=console0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/1 (label charconsole0)
2020-09-16 10:48:15.496+0000: panic s390: core='1' psw-mask='0x0002000000000000' psw-addr='0x0000000000000000' reason='disabled-wait'
2020-09-16T10:52:42.641203Z qemu-system-s390x: terminating on signal 15 from pid 3382 (/usr/sbin/libvirtd)
2020-09-16 10:52:43.241+0000: shutting down, reason=destroyed

Retrying without console worked without crash.
Again retrying with console worked without crash and trying multiple times to recreate the crash failed.
Note: libvirts capabilities cache file remained unchanged during all this!
Rebooted and tried to cause the crash again failed as well. I am not sure what caused this glitch!

Continued testing by replacing kernel parameter prot_virt=1 with prot-virt=0 and rebooting.
Libvirts cached capabilities file has been updated, virt-host-validate returned "WARN (IBM Secure Execution appears to be disabled in kernel. Add prot_virt=1 to kernel cmdline arguments)" for "QEMU: Checking for secure guest support" and as expected SE guest was NOT able to start. Guest log:
2020-09-16 11:17:06.859+0000: panic s390: core='0' psw-mask='0x0002000080000000' psw-addr='0x0000000000004607' reason='disabled-wait'

Replaced kernel parameter "prot-virt=0" with "prot-virt=1" and rebooted.
Libvirts cached capabilities file has been updated, virt-host-validate returned "PASS" for "QEMU: Checking for secure guest support" and as expected SE guest was able to start.

Removed kernel parameter "prot-virt=1" and rebooted host.
Libvirts cached capabilities file has been updated, virt-host-validate returned "WARN (IBM Secure Execution appears to be disabled in kernel. Add prot_virt=1 to kernel cmdline arguments)" for "QEMU: Checking for secure guest support" and as expected SE guest was NOT able to start. Guest log:
2020-09-16 12:06:12.182+0000: panic s390: core='0' psw-mask='0x0002000080000000' psw-addr='0x0000000000004607' reason='disabled-wait'

Added kernel parameter "prot-virt=1 prot_virt=0" and rebooted host.
Libvirts cached capabilities file has been update, virt-host-validate returned "pass" for "QEMU: Checking for secure guest support" and SE guest was able to start successfully.

Played around with kernel parameter values (prot_virt or prot-virt setting it to 0 or 1) a couple of times.
All seems to work as expected.
I was never able to recreate the guest crash experienced first and since I doubt that libvirt is involved in this behavior I regard this test successful.