"kvm_init_vcpu failed: Invalid argument" error on ARM big.LITTLE boards

[RussianNeuroMancer]

Hello!

Upon reproducing bug 1832394 I found another issue - it's seems like sometimes qemu start on core with wrong microarchitecture . Which is, I guess, doesn't happen on ARM servers, but do happen on SBC due to big.LITTLE architecture.

Tested hardware: NanoPC-T4 and ROCKPro64.
Both of Ubuntu 18.04 and 19.04 was tested, so it's known that this issue is reproducible with at least libvirt 5.0.0, qemu 3.1 and virt-manager 2.0.0.

I also tested not only latest stable Linux 4.4 release but also Linux 5.1 on every board.

Logs is below:

~$ virt-install --name=ubuntu-server --ram=2048 --arch=aarch64 --vcpus=1 --virt-type=kvm --hvm --os-type=linux --os-variant=ubuntu18.04 --boot loader=/usr/share/AAVMF/AAVMF_CODE.fd,loader_ro=yes,loader_type=pflash,nvram=/var/lib/libvirt/qemu/nvram/ubuntu-server_VARS.fd --location=/var/lib/libvirt/images/ubuntu-19.04-server-arm64.iso --disk=/var/lib/libvirt/images/ubuntu-server.img --network=bridge=br0,model=virtio

Starting install...
Retrieving file .treeinfo... | 0 B 00:00:00
Retrieving file content... | 0 B 00:00:00
Retrieving file info... | 62 B 00:00:00
Retrieving file vmlinuz... | 8.6 MB 00:00:00
Retrieving file initrd.gz... | 10 MB 00:00:00
Allocating 'virtinst-vmlinuz.FoXAzF' | 8.6 MB 00:00:00
Transferring virtinst-vmlinuz.FoXAzF | 8.6 MB 00:00:00
Allocating 'virtinst-initrd.gz.CoM13X' | 10 MB 00:00:00
Transferring virtinst-initrd.gz.CoM13X | 10 MB 00:00:00
ERROR internal error: process exited while connecting to monitor: kvm_init_vcpu failed: Invalid argument
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start ubuntu-server
otherwise, please restart your installation.

Another attempt with "--cpu=host-passthrough":

~$ virt-install --name=ubuntu-server --ram=2048 --arch=aarch64 --vcpus=1 --virt-type=kvm --hvm --cpu=host-passthrough --os-type=linux --os-variant=ubuntu18.04 --boot loader=/usr/share/AAVMF/AAVMF_CODE.fd,loader_ro=yes,loader_type=pflash,nvram=/var/lib/libvirt/qemu/nvram/ubuntu-server_VARS.fd --location=/var/lib/libvirt/images/ubuntu-19.04-server-arm64.iso --disk=/var/lib/libvirt/images/ubuntu-server.img --network=bridge=br0,model=virtio

Starting install...
Retrieving file .treeinfo... | 0 B 00:00:00
Retrieving file content... | 0 B 00:00:00
Retrieving file info... | 62 B 00:00:00
Retrieving file vmlinuz... | 8.6 MB 00:00:00
Retrieving file initrd.gz... | 10 MB 00:00:00
Allocating 'virtinst-vmlinuz.6qngzC' | 8.6 MB 00:00:00
Transferring virtinst-vmlinuz.6qngzC | 8.6 MB 00:00:00
Allocating 'virtinst-initrd.gz.X5ZnUW' | 10 MB 00:00:00
Transferring virtinst-initrd.gz.X5ZnUW | 10 MB 00:00:00
ERROR internal error: process exited while connecting to monitor: 2019-06-11 18:35:40.283+0000: Domain id=5 is tainted: host-cpu
kvm_init_vcpu failed: Invalid argument
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start ubuntu-server
otherwise, please restart your installation.

[RussianNeuroMancer (russianneuromancer)]

Below is content of /var/log/libvirt/qemu/instance-00000007.log
Source of instance-00000007 is second link in this comment: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1706630/comments/9

(network device was removed, img file name was changed, everything else is untouched)

2019-06-11 19:00:10.630+0000: starting up libvirt version: 4.0.0, package: 1ubuntu8.10 (Marc Deslauriers <email address hidden> Tue, 14 May 2019 15:11:45 -0400), qemu version: 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.14), hostname: nanopct4
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -name guest=instance-00000007,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-27-instance-00000007/master-key.aes -machine virt-2.10,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 -cpu host -drive file=/usr/share/AAVMF/AAVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/instance-00000007_VARS.fd,if=pflash,format=raw,unit=1 -m 256 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 674a2e53-cfbb-4043-a6f0-09536acc1781 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-27-instance-00000007/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-shutdown -boot strict=on -device ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 -device ioh3420,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 -device ioh3420,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 -device ioh3420,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 -device ioh3420,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 -drive file=/var/lib/libvirt/images/ubuntu-server.img,format=raw,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.2,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -add-fd set=0,fd=27 -chardev pty,id=charserial0,logfile=/dev/fdset/0,logappend=on -serial chardev:charserial0 -device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 -msg timestamp=on
2019-06-11 19:00:10.630+0000: Domain id=27 is tainted: host-cpu
2019-06-11T19:00:10.696070Z qemu-system-aarch64: -chardev pty,id=charserial0,logfile=/dev/fdset/0,logappend=on: char device redirected to /dev/pts/1 (label charserial0)
kvm_init_vcpu failed: Invalid argument
2019-06-11 19:00:10.729+0000: shutting down, reason=failed

[Kevin Zhao (kevin-zhao)]

I can also reproduce this. I'm using Khadas VIM3 with amologic CPUs.
Ubuntu 20.04, kernel 5.7 and libvirt 6.0.0, qemu 4.2

[RussianNeuroMancer (russianneuromancer)]

Workaround: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/177#issuecomment-717318013