PowerPC L2(nested virt) kvm guest fails to boot with ic-mode=dual,kernel-irqchip=on - `KVM is too old to support ic-mode=dual,kernel-irqchip=on`

[Satheesh Rajendran]

Env:
HW: Power 9 DD2.3
Host L0: 5.8.0-rc5-g8ba4ffcd8
Qemu: 5.0.50 (v5.0.0-533-gdebe78ce14)
Libvirt: 6.4.0
L1: 5.8.0-rc5-ge9919e11e
qemu_version': '5.0.50 (v5.1.0-rc2-dirty)
libvirt_version': '6.4.0'
L2: 5.8.0-rc7-g6ba1b005f

1. boot a L2 KVM guest with `ic-mode=dual,kernel-irqchip=on`

/usr/bin/virt-install --connect=qemu:///system --hvm --accelerate --name 'vm1' --machine pseries --memory=8192 --vcpu=8,maxvcpus=8,sockets=1,cores=2,t
hreads=4 --import --nographics --serial pty --memballoon model=virtio --disk path=/home/tests/data/avocado-vt/images/f31-ppc64le.qcow2,bus=virtio,size=10,format=qcow2 --network
=bridge=virbr0,model=virtio,mac=52:54:00:e6:fe:f6 --mac=52:54:00:e6:fe:f6 --boot emulator=/usr/share/avocado-plugins-vt/bin/qemu,kernel=/tmp/linux/vmlinux,kernel_args="root=/de
v/vda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0" --noautoconsole --qemu-commandline=" -M pseries,ic-mode=dual,kernel-irqchip=on"

ERROR internal error: process exited while connecting to monitor: 2020-08-04T11:12:53.304482Z qemu: KVM is too old to support ic-mode=dual,kernel-irqchip=on

Qemu Log:
```
/usr/share/avocado-plugins-vt/bin/qemu \
-name guest=vm1,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-vm1/master-key.aes \
-machine pseries-5.1,accel=kvm,usb=off,dump-guest-core=off \
-cpu POWER9 \
-m 8192 \
-overcommit mem-lock=off \
-smp 8,sockets=1,dies=1,cores=2,threads=4 \
-uuid 20a3351b-2776-4e75-9059-c070fe3dd44b \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=34,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-kernel /tmp/linux/vmlinux \
-append 'root=/dev/vda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0' \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.0,addr=0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 \
-blockdev '{"driver":"file","filename":"/home/tests/data/avocado-vt/images/f31-ppc64le.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-disk0,bootindex=1 \
-netdev tap,fd=37,id=hostnet0,vhost=on,vhostfd=38 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:e6:fe:f6,bus=pci.0,addr=0x1 \
-chardev pty,id=charserial0 \
-device spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 \
-chardev socket,id=charchannel0,fd=39,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \
-M pseries,ic-mode=dual,kernel-irqchip=on \
-msg timestamp=on
2020-08-04 11:12:53.169+0000: Domain id=5 is tainted: custom-argv
2020-08-04 11:12:53.179+0000: 11120: info : libvirt version: 6.4.0, package: 1.fc31 (Unknown, 2020-06-02-05:09:40, ltc-wspoon4.aus.stglabs.ibm.com)
2020-08-04 11:12:53.179+0000: 11120: info : hostname: atest-guest
2020-08-04 11:12:53.179+0000: 11120: info : virObjectUnref:347 : OBJECT_UNREF: obj=0x7fff0c117c40
char device redirected to /dev/pts/0 (label charserial0)
2020-08-04T11:12:53.304482Z qemu: KVM is too old to support ic-mode=dual,kernel-irqchip=on
2020-08-04 11:12:53.694+0000: shutting down, reason=failed
```

[Greg Kurz (gkurz)]

This is currently expected because the L2 KVM guest uses the historical KVM XICS device (not the XICS-on-XIVE one) and this can be only created once during the VM lifetime for the moment.

This is a limitation in KVM, that can be addressed in several ways:
1) change the historical KVM XICS device to implement the release() method instead of destroy(), so that userspace can close() and re-create the device multiple times during the VM lifetime, as we have already done in KVM XIVE and KVM XICS-on-XIVE for powernv
2) have the KVM XICS-on-XIVE device to work under pseries

I already have a tentative patch for 1) and I guess 2) would be part of a more global work to supporting nested KVM XIVE.

But it is definitely not an issue in QEMU.

[Satheesh Rajendran (sathnaga)]

As per this the table https://www.qemu.org/docs/master/specs/ppc-spapr-xive.html#kvm-negotiation

reported qemu error msg "KVM is too old to support ic-mode=dual,kernel-irqchip=on" indicates the
guest os is legacy, but that's not the case here, whereas kernel levels are near upstream which has support for xive.

My understanding of the env I used as below

Level | XIVE KVM support | XIVE support(in kernel or emulation)
--------------------------------------------------
 L0 | Yes | Yes
 L1 | No | Yes(booted with irqchip: in-kernel)
 L2 | No | Yes

So, ideally when a L2 guest is started with ic-mode=dual,kernel-irqchip=on, we should have seen below error
(2) QEMU fails with ``kernel_irqchip requested but unavailable:
    IRQ_XIVE capability must be present for KVM``

but we actually saw the reported one, which is misleading.

[Satheesh Rajendran (sathnaga)]

this section of table in particular, https://www.qemu.org/docs/master/specs/ppc-spapr-xive.html#no-xive-support-in-kvm

[Greg Kurz (gkurz)]

Hmm... the documentation might need an update. I'll have a look.

[Greg Kurz (gkurz)]

Posted a patch to the list.

http://patchwork.ozlabs.org/project<email address hidden>/

Satheesh,

Can you please review and test ?

[Satheesh Rajendran (sathnaga)]

@Greg,

Thanks for the patch, I see it already got applied into 5.2, tested and works fine,

# git log -2 --oneline
1972794 (HEAD -> master) spapr: Clarify error and documentation for broken KVM XICS
e1d322c (grafted, tag: v5.1.0-rc3, origin/master, origin/HEAD) Update version for v5.1.0-rc3 release

# /usr/bin/virt-install --connect=qemu:///system --hvm --accelerate --name 'vm1' --machine pseries --memory=8192 --vcpu=8,maxvcpus=8,sockets=1,cores=8,threads=1 --import --nographics --serial pty --memballoon model=virtio --controller type=scsi,model=virtio-scsi --disk path=/home/tests/data/avocado-vt/images/f31-ppc64le.qcow2,bus=scsi,size=10,format=qcow2 --network=bridge=virbr0,model=virtio,mac=52:54:00:5c:f1:fe --mac=52:54:00:5c:f1:fe --boot emulator=/home/qemu/ppc64-softmmu/qemu-system-ppc64,kernel=/boot/vmlinuz-5.8.0-rc5-ge9919e11e,kernel_args="root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0" --noautoconsole --qemu-commandline=" -M pseries,ic-mode=dual,kernel-irqchip=on";virsh console vm1

Starting install...
ERROR internal error: process exited while connecting to monitor: 2020-08-07T07:05:38.633057Z qemu-system-ppc64: KVM is incompatible with ic-mode=dual,kernel-irqchip=on
This can happen with an old KVM or in a KVM nested guest.
Try without kernel-irqchip or with kernel-irqchip=off.

Regards,
-Satheesh

[Thomas Huth (th-huth)]

Released with QEMU v5.2.0.