Avoid creation of /dev/kvm in Xenial KVM virtual machines on s390x

[Frank Heimes]

Since nested virtualization with KVM is not supported on s390x, please disable this in Xenial - in case it's doable with reasonable effort.
(I think it was already disabled for later Ubuntu releases ... was also discussed on IRC)

[Stefan Bader (smb)]

Note that the KVM kernel side in Xenial does not support nested at all. The confusion is caused by the fact that something seems to create /dev/kvm even when running in a KVM VM. Nested KVM support in the kernel was added (but disabled by default) around Zesty (4.10 kernels).

[Frank Heimes (fheimes)]

So can it be avoided (with reasonable effort) that /dev/kvm is created in the xenial kernel, too?

(Adjusted title and description accordingly)

[Joseph Salisbury (jsalisbury)]

This may be something that needs to be done outside of the kernel? There could be a udev rule that needs updating?

[Frank Heimes (fheimes)]

jsalisbury, this is a kernel module option which could have been set via modprobe.d files, but kvm mobule on s390x is compiled in, hence always loaded. hence in later kernels compiled in default is changed to not create nested kvm devices by default. hence this needs to be resolved as a cherrypick of upstream kernel change. -- says xnox from my laptop

[Joseph Salisbury (jsalisbury)]

Frank, Is there a way to test a test kernel I built? I built a Xenial test kernel with a back port of commit a411edf1320e ("KVM: s390: vsie: add module parameter "nested"").

I'm not sure off hand if just this commit is needed or if there are others as well.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1735420/

[Frank Heimes (fheimes)]

So I just did a brief test - stock kernel first and then the modified one:

ubuntu@ubu-vm-01:~$ uname -a
Linux ubu-vm-01 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:21:31 UTC 2018 s390x s390x s390x GNU/Linux
ubuntu@ubu-vm-01:~$ ls -la /dev/kvm
crw------- 1 root root 10, 232 Mar 13 22:42 /dev/kvm

ubuntu@ubu-vm-01:~$ uname -a
Linux ubu-vm-01 4.4.0-116-generic #140~lp1735420 SMP Tue Mar 13 21:48:51 UTC 2018 s390x s390x s390x GNU/Linux
ubuntu@ubu-vm-01:~$ ls -la /dev/kvm
crw------- 1 root root 10, 232 Mar 14 03:47 /dev/kvm
ubuntu@ubu-vm-01:~$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
ubuntu@ubu-vm-01:~$

So I couldn't identify a difference between both kernels so far ...

[Joseph Salisbury (jsalisbury)]

Since we don't know the commit that would solve this off hand, we can perform a "Reverse" bisect to identify that commit.

Can you see if the following kernel has the bug:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8/

[Frank Heimes (fheimes)]

Mainline Kernel 4.8 does not seem to have the patch included:

ubuntu@ubu-vm-01:~$ uname -a
Linux ubu-vm-01 4.8.0-040800-generic #201610022031 SMP Mon Oct 3 02:27:31 UTC 2016 s390x s390x s390x GNU/Linux
ubuntu@ubu-vm-01:~$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
ubuntu@ubu-vm-01:~$ ls -la /dev/kvm
crw------- 1 root root 10, 232 Mar 20 05:16 /dev/kvm

Knowing that it is in 4.13, I tried mainline 4.10 as well and - yes - it must have the patch:

ubuntu@ubu-vm-01:~$ uname -a
Linux ubu-vm-01 4.10.0-041000-generic #201702191831 SMP Mon Feb 20 00:15:43 UTC 2017 s390x s390x s390x GNU/Linux
ubuntu@ubu-vm-01:~$ kvm-ok
INFO: /dev/kvm does not exist
HINT: sudo modprobe kvm
INFO: For more detailed results, you should run this as root
HINT: sudo /usr/sbin/kvm-ok
ubuntu@ubu-vm-01:~$ ls -la /dev/kvm
ls: cannot access '/dev/kvm': No such file or directory
ubuntu@ubu-vm-01:~$

So looks like the patch came in with either 4.9 or 4.10 (probably 4.10).

[Joseph Salisbury (jsalisbury)]

Can you see if the patch is in v4.10-rc1:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10-rc1/

[Frank Heimes (fheimes)]

Doesn't seem to be in - kvm device still there:

ubuntu@ubu-vm-01:~$ uname -a
Linux ubu-vm-01 4.10.0-041000rc1-generic #201612252031 SMP Mon Dec 26 02:37:39 UTC 2016 s390x s390x s390x GNU/Linux
ubuntu@ubu-vm-01:~$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
ubuntu@ubu-vm-01:~$ ls -la /dev/kvm
crw------- 1 root root 10, 232 Mar 20 07:34 /dev/kvm

[Frank Heimes (fheimes)]

I tried some more:

4.10 rc3 does have the patch:
ubuntu@ubu-vm-01:~$ uname -a
Linux ubu-vm-01 4.10.0-041000rc3-generic #201701081831 SMP Mon Jan 9 00:34:16 UTC 2017 s390x s390x s390x GNU/Linux
ubuntu@ubu-vm-01:~$ kvm-ok
INFO: /dev/kvm does not exist
HINT: sudo modprobe kvm
INFO: For more detailed results, you should run this as root
HINT: sudo /usr/sbin/kvm-ok
ubuntu@ubu-vm-01:~$ ls -la /dev/kvm
ls: cannot access '/dev/kvm': No such file or directory

4.10-rc2 seems to be the last one without:

ubuntu@ubu-vm-01:~$ uname -a
Linux ubu-vm-01 4.10.0-041000rc2-generic #201701011831 SMP Mon Jan 2 00:41:33 UTC 2017 s390x s390x s390x GNU/Linux
ubuntu@ubu-vm-01:~$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
ubuntu@ubu-vm-01:~$ ls -la /dev/kvm
crw------- 1 root root 10, 232 Mar 20 07:41 /dev/kvm

[Joseph Salisbury (jsalisbury)]

There was only one commit that sticks out between 4.10-rc2 and v4.10-rc3, it is kvm and s390 related:

cabab3f ("s390/kbuild: enable modversions for symbols exported from asm")

I built a Xenial test kernel with a pick of this commit. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1735420/

Can you test this kernel and see if it resolves the bug? If it does not, we'll have to reverse bisect to identify the commit.

[Frank Heimes (fheimes)]

Well, does not seem to be in:

ubuntu@ubu-vm-01:~$ uname -a
Linux ubu-vm-01 4.4.0-116-generic #140~lp1735420Commitcabab3f SMP Tue Mar 20 20:07:24 UTC 2018 s390x s390x s390x GNU/Linux
ubuntu@ubu-vm-01:~$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
ubuntu@ubu-vm-01:~$ ls -la /dev/kvm
crw-rw---- 1 root kvm 10, 232 Mar 20 12:17 /dev/kvm

Interestingly I couldn't find any indication in the 4.10.0-041000rc3 changes file that point to this functionality ...

[Joseph Salisbury (jsalisbury)]

Yeah, I couldn't find anything specific in v4.10-rc3 either. I'll start a reverse bisect to see which commit that points to. It will take testing of about 7 or so test kernels.

[Joseph Salisbury (jsalisbury)]

Hmm, it could also be a config change that is needed. The 4.10-rc2 kernel uses the 4.9.0-2.3 configs and the 4.10-rc3 kernel uses the 4.10.0-0.2 config. I'll compare those two as well.

[Joseph Salisbury (jsalisbury)]

I noted a difference between the two configs. The v4.10-rc3 config has the KVM module built in. However, the v4.10-rc2 config has KVM built in for the config file, but was still set to module in the annotations file.

I fixed the annotations file and built a Xenial test kernel.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1735420/

Can you test this kernel and see if it resolves the bug?

[Frank Heimes (fheimes)]

Sorry for the delay - got a bit distracted by other topics.

Just did a retest with
"linux-image-4.4.0-116-generic_4.4.0-116.140~lp1735420ModifiedAnnotations_s390x.deb"
but still can see the kvm device:

ubuntu@ubu-vm-01:~$ uname -a
Linux ubu-vm-01 4.4.0-116-generic #140~lp1735420ModifiedAnnotations SMP Wed Mar 21 17:57:02 UTC 20 s390x s390x s390x GNU/Linux
ubuntu@ubu-vm-01:~$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
ubuntu@ubu-vm-01:~$ ls -la /dev/kvm
crw-rw---- 1 root kvm 10, 232 Mar 27 01:45 /dev/kvm
ubuntu@ubu-vm-01:~$

[Frank Heimes (fheimes)]

Lowering importance to Low due to currently different focus.

[Frank Heimes (fheimes)]

Since we already reached the stabilization phase of xenial/16.04 with .5 (and .6), with the focus on stability and security, and because this is not a really bug, rather than 'nice to have' to _not_ show /dev/kvm in case already running virtualized on KVM (because nested KVM is not supported on s390x), I'm now going to change this Bug to Won't Fix.
In anybody objects please leave a comment ...