kvm_pr on power9 not loadable

[Christian Ehrhardt ]

Hi,
this might be the worst bug report ever, but there just isn't a lot more info available when the issue occurs.

I tried to set up 2nd level KVM on a P9 machine with kvm_pr but I run into (this is the series on commands I used on P8 btw):
  $ sudo modprobe kvm_pr
  modprobe: ERROR: could not insert 'kvm_pr': Input/output error

Unfortunately there is no dmesg message (not even with level set to debug) or anything like it, so I'm stuck asking if P9 does not work with kvm_pr at all or if there are other constraints one should know?

Note: while no more a problem for kvm_hv on P9 I disabled SMT but the issue persists.

Modinfo LGTM and matches the kernel:
ubuntu@bionic-kvm:~$ modinfo kvm_pr
filename: /lib/modules/4.15.0-23-generic/kernel/arch/powerpc/kvm/kvm-pr.ko
alias: devname:kvm
alias: char-major-10-232
license: GPL
srcversion: CE18B50FD03CB9D9C432700
depends: kvm
intree: Y
name: kvm_pr
vermagic: 4.15.0-23-generic SMP mod_unload mprofile-kernel
signat: PKCS#7
signer:
sig_key:
sig_hashalgo: md4
ubuntu@bionic-kvm:~$ uname -a
Linux bionic-kvm 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 17:59:00 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

It feels more that I'm missing something than a bug, but in that case I have to ask what it is.

[Christian Ehrhardt  (paelzer)]

Added the project, please reverse mirror to get explained what (should) happen in this case.

[Breno Leitão (breno-leitao)]

hi Christian, this is interesting. I tried to reproduce here on my P9 (with cosmic and kernel 4.15.0-22) , but I was not able to:

➜ ~ sudo modprobe kvm_pr

➜ ~ lsmod | grep kvm_pr
kvm_pr 107493 0
kvm 225431 1 kvm_pr

➜ ~ uname -a
Linux cosmic 4.15.0-22-generic #24-Ubuntu SMP Wed May 16 12:12:37 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

➜ ~ cat /proc/cpuinfo | grep model
model : IBM pSeries (emulated by qemu)

[Christian Ehrhardt  (paelzer)]

Hmm, in that case just teach me what data you'd want to look on when this happens again.
We will keep the bug as incomplete and when I hit it again I can check what you would need to analyze it better.

[Christian Ehrhardt  (paelzer)]

I have run into this again.
I've read that it works for you, but here is all I have and as I said before - what would you want me to report to look at?

---

On power you can never load the usual kvm_hv module in a guest.
 $ sudo modprobe kvm_hv
 modprobe: ERROR: could not insert 'kvm_hv': No such device

That is normal and expected.
But on a Power9 system I can neither load kvm_pr which was usually the way to go for nested KVM.

$ sudo modprobe kvm_pr
modprobe: ERROR: could not insert 'kvm_pr': Input/output error

There is no error whatsoever in dmesg when triggering this.
From libvirt the guest is defined with all defaults - see http://paste.ubuntu.com/p/2r34SG6pq6/

That gives the guest CPU this guest view:
cat /proc/cpuinfo
processor : 0
cpu : POWER9 (architected), altivec supported
clock : 2134.000000MHz
revision : 2.2 (pvr 004e 1202)

timebase : 512000000
platform : pSeries
model : IBM pSeries (emulated by qemu)
machine : CHRP IBM pSeries (emulated by qemu)
MMU : Radix

Nothing obvious here, also the module exists and seems normal to me:
$ modinfo kvm_pr
filename: /lib/modules/4.17.0-6-generic/kernel/arch/powerpc/kvm/kvm-pr.ko
alias: devname:kvm
alias: char-major-10-232
license: GPL
srcversion: 2472883ADEEBC51508B6BF4
depends: kvm
intree: Y
name: kvm_pr
vermagic: 4.17.0-6-generic SMP mod_unload mprofile-kernel relocatable
sig_id: PKCS#7
signer:
sig_key:
sig_hashalgo: md4
signature: 30:82...

Current kernel is From Cosmic (as that is what I test for atm): 4.17.0-6-generic

If I start the guest with the "least virtual" cpu via
  <cpu mode='host-passthrough' check='none'/>

I get no difference in /proc/cpuinfo from the guest nor does the problem change.

FYI Host CPU reports as (this is already with smt off)
[...]
processor : 156
cpu : POWER9, altivec supported
clock : 3800.000000MHz
revision : 2.2 (pvr 004e 1202)

timebase : 512000000
platform : PowerNV
model : 9006-12C
machine : PowerNV 9006-12C
firmware : OPAL
MMU : Radix

[Christian Ehrhardt  (paelzer)]

FYI - until resolved some of my usual pre-upload tests have to be skipped on P9.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-12-27 14:20 EDT-------
Hello,

I could reproduce this on Power9, in the following kernels:
4.15.0-20-generic
4.18.0-10-generic

I compiled kvm_pr for 4.18, and debbuged it until I found this:

arch/powerpc/kvm/book3s_pr.c:2038
/*
* PR KVM can work on POWER9 inside a guest partition
* running in HPT mode. It can't work if we are using
* radix translation (because radix provides no way for
* a process to have unique translations in quadrant 3).
*/
if (cpu_has_feature(CPU_FTR_ARCH_300) && radix_enabled())
return -EIO;
return 0;

Here, cpu_has_feature(CPU_FTR_ARCH_300) returns:
false for Power8,
true for Power9.

And radix_enabled() returns:
false if bootargs contains "disable_radix",
true otherwise.

##

Meaning that, if one wants to load the kvm_pr module on a guest with Power9, the guest kernel must be booted with "disable_radix", as it looks like radix is the default choice.

paelzer, please verify if your tests run with this parameter.

##

BTW, it only works on vanilla kernel v4.18+. Before that, kvm_pr was just disabled for Power9. I am not sure about Ubuntu kernel including this patch on earlier kernels.

[Christian Ehrhardt  (paelzer)]

Interesting, thanks Leonardo!

Tested on Bionic (4.15) and Disco (4.18).
Default:
sudo modprobe kvm_pr
modprobe: ERROR: could not insert 'kvm_pr': Input/output error

Added "disable_radix" to the guest kernel commandline and retried loading the module again.
Bionic: Malformed early option 'disable_radix'
Disco: accepts the option

With that option set kvm_pr can be loaded.
Thanks now all fits together, and that is "enough" for me.

If there are kernel fixes that you'd want in Bionic to get disable_radix to work we can add a kernel task (I don't need it).

TL;DR: for Disco forward I should be able to enable P9 based nested tests with that tweak in place.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-02-06 06:59 EDT-------
Just an additional information: starting with linux 4.20 and QEMU 3.1 one can created nested guests on RPT guests as well using kvm_hv (kvm_pr will not work with RPT).

[Christian Ehrhardt  (paelzer)]

Since this works (using disable_radix thanks leonardo) in Cosmic and also through the HWE kernel on Bionic I think there is no more work needed. The trigger was P9 HW which is fine for this specific (and uncommon) function to define the HWE kernel as requirement.