[arm64] kvm run failed Function not implemented

[Junien F]

Hi,

On an arm64 Nova compute node, instances will sometimes end up in the Paused state with the following libvirt log :

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-aarch64 -name instance-001285c8 -S -machine virt,accel=kvm,usb=off -cpu host -bios /usr/share/qemu-efi/QEMU_EFI.fd -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 353d3da4-da46-48c0-9ed2-4acf940bc61e -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-001285c8.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-shutdown -boot strict=on -usb -drive file=/var/lib/nova/instances/353d3da4-da46-48c0-9ed2-4acf940bc61e/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=unsafe -device virtio-blk-device,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/nova/instances/353d3da4-da46-48c0-9ed2-4acf940bc61e/disk.swap,if=none,id=drive-virtio-disk1,format=qcow2,cache=unsafe -device virtio-blk-device,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-device,netdev=hostnet0,id=net0,mac=fa:16:3e:55:bb:d8 -serial file:/var/lib/nova/instances/353d3da4-da46-48c0-9ed2-4acf940bc61e/console.log -serial pty -device virtio-balloon-device,id=balloon0 -msg timestamp=on
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-curl.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-rbd.so: failed to map segment from shared object: Permission denied
char device redirected to /dev/pts/6 (label serial1)
2016-12-02T09:20:48.382852Z qemu-system-aarch64: binding does not support guest notifiers
2016-12-02T09:20:48.383041Z qemu-system-aarch64: unable to start vhost net: 38: falling back on userspace virtio
error: kvm run failed Function not implemented
PC=000000023faf924c SP=0000000240000020
X00=0000000000000000 X01=000000023fafb064 X02=0000000040000305 X03=0000000000000000
X04=00000000dbadc0de X05=1de7ec7edbadc0de X06=0000000000000004 X07=000000023fa60430
X08=fffffffffffff001 X09=0000000000000400 X10=000000023fb87de8 X11=000000023bdd9a18
X12=00000000a007e03a X13=000000023fdadffc X14=000000023fb87cb8 X15=0000000000000001
X16=0000000000000003 X17=000000000020d0d0 X18=000000001f526ead X19=000000023be36060
X20=000000023be36018 X21=0000000000000010 X22=000000023fb9da38 X23=0000000000000000
X24=0000000000000064 X25=000000004007fb40 X26=0000000000000000 X27=000000004007c278
X28=000000004007c030 X29=0000000000000000 X30=000000023faf9178 PSTATE=60000305 (flags -ZC-)

The 2 "qemu-system-aarch64:" lines are also present on successful instances (i.e. instances which don't end up in the Paused state)

This is on trusty, with :
qemu-system-arm 1:2.3+dfsg-5ubuntu9~ubuntu14.04.1
nova-compute-kvm 1:2015.1.1-0ubuntu2+ppa1
libvirt0 1.2.12-0ubuntu14.4+ppa1
Linux kernel 4.8.0-040800rc8-generic

[Richard (richard-redhat-bugs)]

Description of problem:

This has similarities to the following bug, but is much rarer:
https://bugzilla.redhat.com/show_bug.cgi?id=1194366

I have a script which boots a guest 1000 times on the Fedora/aarch64
host. Guest and host kernels are identical (4.2.0-1.fc24.aarch64).

About 5 in every 1000 boots fail. The host kernel prints:

kvm [3683]: load/store instruction decoding not implemented

The corresponding qemu process hangs after printing:

error: kvm run failed Function not implemented
PC=000000006bbfd238 SP=00000000700000b0
X00=aa1903e1aa0303e2 X01=0000000068e67b40 X02=000000006baa7cec X03=0000000000e68200
X04=000000006bac1398 X05=00000000009ffaf8 X06=0000000000000000 X07=000000006f04c85c
X08=000000006f04cb78 X09=0000000000000000 X10=0000000000000004 X11=0000000000000000
X12=00000000700fe0fa X13=0000000000000000 X14=0000000000000000 X15=0000000000000000
X16=000000006f04cdf0 X17=0000000000000000 X18=0000000000000000 X19=000000006bff0018
X20=0000000000000000 X21=0000000000000000 X22=0000000000000000 X23=0000000000000000
X24=0000000000000000 X25=0000000000000000 X26=0000000000000000 X27=0000000000000000
X28=0000000000000000 X29=0000000000000000 X30=0000000000000000 PSTATE=60000305 (flags -ZC-)

The PC address does not correspond to any kernel address.

Version-Release number of selected component (if applicable):

kernel 4.2.0-1.fc24.aarch64

How reproducible:

Rare, approximately 1 in 200 boots.

Steps to Reproduce:
1. In the libguestfs test suite, run:
   ./tests/qemu/qemu-boot -n 1000

Additional info:

The error message usually indicates that the guest has jumped into random code.

I'm still investigating this bug, will update this bug with further details
as I collect it.

[Richard (richard-redhat-bugs)]

Created attachment 1081033
serial port log from guest

Even though this is Fedora, I'm using AAVMF
(AAVMF-20141113-5.git77d5dac.aa7a.aarch64). It appears to
be crashing in AAVMF.

Attached is the guest serial port log up to the point where
the guest stops.

[Richard (richard-redhat-bugs)]

Created attachment 1081034
libvirt/qemu log from guest

This is the libvirt/qemu log from the same guest as in comment 1.

[Richard (richard-redhat-bugs)]

As this works with Gerd's edk2.git/aarch64 package, I'm going
to say this was because I was using a too old version of AAVMF,
so it's likely not a bug.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christian Ehrhardt  (paelzer)]

Hi Junien,
I've seen issues like that but they were fixed in qemu 2.3 which you have (https://bugzilla.redhat.com/show_bug.cgi?id=1213478).

In general I wonder where you actual qemu comes from, Trusty is on 2.0 and the closest would be UVA-Liberty with 1:2.3+dfsg-5ubuntu9.4~cloud2, so what kind of Trusty setup is that?

Next as this might be arm platform specific it would be great to know what exact arm-board the host is in this case.

Subscribing Dannf for arm/qemu experience ...

[Christian Ehrhardt  (paelzer)]

The issue would also appear in second level virtualization which this is not right?

[Junien F (axino)]

I'm upgrading qemu to mitaka-staging version by the way (1:2.5+dfsg-5ubuntu10.7~cloud0)

[Junien F (axino)]

Hi,

This just happened again with qemu 1:2.5+dfsg-5ubuntu10.7~cloud0

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-aarch64 -name instance-00143dc7 -S -machine virt,accel=kvm,usb=off -cpu host -bios /usr/share/qemu-efi/QEMU_EFI.fd -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 38301f7e-d86e-4506-94b7-3a1f8390c9ff -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00143dc7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-shutdown -boot strict=on -usb -drive file=/var/lib/nova/instances/38301f7e-d86e-4506-94b7-3a1f8390c9ff/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=unsafe -device virtio-blk-device,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/nova/instances/38301f7e-d86e-4506-94b7-3a1f8390c9ff/disk.swap,if=none,id=drive-virtio-disk1,format=qcow2,cache=unsafe -device virtio-blk-device,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-device,netdev=hostnet0,id=net0,mac=fa:16:3e:04:8d:9a -serial file:/var/lib/nova/instances/38301f7e-d86e-4506-94b7-3a1f8390c9ff/console.log -serial pty -device virtio-balloon-device,id=balloon0 -msg timestamp=on
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-iscsi.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-curl.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-rbd.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-dmg.so: failed to map segment from shared object: Permission denied
char device redirected to /dev/pts/6 (label serial1)
error: kvm run failed Function not implemented
PC=000000023faf924c SP=0000000240000020
X00=000000023be36018 X01=0000000000000000 X02=000000023faf7a9c X03=000000000000004c
X04=0000000000000000 X05=0000000000000000 X06=0000000000000000 X07=000000023fa60430
X08=fffffffffffff001 X09=0000000000000400 X10=000000023fb87de8 X11=000000023bdd9a18
X12=00000000a007e03a X13=000000023fdadffc X14=000000023fb87cb8 X15=0000000000000001
X16=0000000000000003 X17=000000000020d0d0 X18=000000001f526ead X19=000000023be36060
X20=000000023be36018 X21=0000000000000010 X22=000000023fb9da38 X23=0000000000000000
X24=0000000000000064 X25=000000004007fb40 X26=0000000000000000 X27=000000004007c278
X28=000000004007c030 X29=0000000000000000 X30=000000023fb92d64
PSTATE=60000305 -ZC- EL1h

and

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-aarch64 -name instance-00143e83 -S -machine virt,accel=kvm,usb=off -cpu host -bios /usr/share/qemu-efi/QEMU_EFI.fd -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid bcd8353e-0521-492b-a103-e577bc90bb64 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00143e83.monitor,server,nowait -mon...

[dann frazier (dannf)]

It looks like the kernel is returning -ENOSYS. Are there any messages in dmesg coinciding with this event?

[Junien F (axino)]

Ah, indeed there is one :

kvm [13833]: load/store instruction decoding not implemented

[dann frazier (dannf)]

Thanks Junien. Is that correlated with the pause-on-start, or does that appear in dmesg on normal startups as well?

[Junien F (axino)]

This is highly correlated with pause-on-start. I can't say 100% because I haven't look at kern.log for all the Paused instances (and I'm not sure I can correlate this easily now), but every time I looked it was there.

And it's definitely not there during normal startups.

[Junien F (axino)]

Hi,

It happened again, with the "load/store instruction decoding not implemented" in the log.

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-aarch64 -name instance-0015baf9 -S -machine virt,accel=kvm,usb=off -cpu host -bios /usr/share/qemu-efi/QEMU_EFI.fd -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 5bfa9507-f5d6-4449-8fce-582d20e8f5a5 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0015baf9.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-shutdown -boot strict=on -usb -drive file=/var/lib/nova/instances/5bfa9507-f5d6-4449-8fce-582d20e8f5a5/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=unsafe -device virtio-blk-device,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/nova/instances/5bfa9507-f5d6-4449-8fce-582d20e8f5a5/disk.swap,if=none,id=drive-virtio-disk1,format=qcow2,cache=unsafe -device virtio-blk-device,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-device,netdev=hostnet0,id=net0,mac=fa:16:3e:b4:bc:49 -serial file:/var/lib/nova/instances/5bfa9507-f5d6-4449-8fce-582d20e8f5a5/console.log -serial pty -device virtio-balloon-device,id=balloon0 -msg timestamp=on
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-iscsi.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-curl.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-rbd.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-dmg.so: failed to map segment from shared object: Permission denied
char device redirected to /dev/pts/5 (label serial1)
error: kvm run failed Function not implemented
PC=000000023faf924c SP=0000000240000020
X00=0000000000000000 X01=000000023fb1648c X02=0000000040000305 X03=0000000000000000
X04=00000000dbadc0de X05=1de7ec7edbadc0de X06=0000000000000004 X07=0000000004000000
X08=000000023711d570 X09=0000000000000400 X10=000000023fb87b38 X11=000000023bdd9a18
X12=00000000a007e03a X13=000000023fdadffc X14=000000023fb87a08 X15=0000000000000001
X16=0000000000000003 X17=000000000020d0d0 X18=000000001f526ead X19=0000000000000000
X20=000000023fb47c18 X21=0000000000800080 X22=0000000000330030 X23=0000000000040000
X24=0000000004000000 X25=0000000000000000 X26=000000000000057c X27=000000023fb87a08
X28=00000000ffffffff X29=0000000000000000 X30=000000023faf9178
PSTATE=60000305 -ZC- EL1h

[Junien F (axino)]

And again. The "load/store instruction decoding not implemented" is still here.

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-aarch64 -name instance-0015c4f9 -S -machine virt,accel=kvm,usb=off -cpu host -bios /usr/share/qemu-efi/QEMU_EFI.fd -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 4d7af74f-49d3-4c98-881c-5ed2f1d77ab4 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0015c4f9.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-shutdown -boot strict=on -usb -drive file=/var/lib/nova/instances/4d7af74f-49d3-4c98-881c-5ed2f1d77ab4/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=unsafe -device virtio-blk-device,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/nova/instances/4d7af74f-49d3-4c98-881c-5ed2f1d77ab4/disk.swap,if=none,id=drive-virtio-disk1,format=qcow2,cache=unsafe -device virtio-blk-device,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-device,netdev=hostnet0,id=net0,mac=fa:16:3e:e6:d4:f8 -serial file:/var/lib/nova/instances/4d7af74f-49d3-4c98-881c-5ed2f1d77ab4/console.log -serial pty -device virtio-balloon-device,id=balloon0 -msg timestamp=on
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-iscsi.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-curl.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-rbd.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-dmg.so: failed to map segment from shared object: Permission denied
char device redirected to /dev/pts/5 (label serial1)
error: kvm run failed Function not implemented
PC=000000023faf924c SP=0000000240000010
X00=0000000000000000 X01=000000023fb1648c X02=0000000080000305 X03=0000000000000000
X04=00000000dbadc0de X05=1de7ec7edbadc0de X06=0000000000000004 X07=0000000000000848
X08=0000000000000008 X09=0000000000000400 X10=000000023fb87cc8 X11=000000023bdd9a18
X12=00000000a007e03a X13=000000023fdadffc X14=000000023fb87b98 X15=0000000000000001
X16=0000000000000003 X17=000000000020d0d0 X18=000000001f526ead X19=0000000000000000
X20=0000000004000000 X21=0000000000000844 X22=0000000000000001 X23=0000000000040000
X24=0000000004000000 X25=0000000000000000 X26=00000000000007dc X27=000000023fb87b88
X28=00000000ffffffff X29=0000000000000000 X30=000000023faf9178
PSTATE=60000305 -ZC- EL1h

[Junien F (axino)]

One more. The "load/store instruction decoding not implemented" is still here.

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-aarch64 -name instance-001613c8 -S -machine virt,accel=kvm,usb=off -cpu host -bios /usr/share/qemu-efi/QEMU_EFI.fd -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid e1cf8136-2321-49d1-ab86-045731bab45b -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-001613c8.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-shutdown -boot strict=on -usb -drive file=/var/lib/nova/instances/e1cf8136-2321-49d1-ab86-045731bab45b/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=unsafe -device virtio-blk-device,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/nova/instances/e1cf8136-2321-49d1-ab86-045731bab45b/disk.swap,if=none,id=drive-virtio-disk1,format=qcow2,cache=unsafe -device virtio-blk-device,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-device,netdev=hostnet0,id=net0,mac=fa:16:3e:93:10:78 -serial file:/var/lib/nova/instances/e1cf8136-2321-49d1-ab86-045731bab45b/console.log -serial pty -device virtio-balloon-device,id=balloon0 -msg timestamp=on
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-iscsi.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-curl.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-rbd.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-dmg.so: failed to map segment from shared object: Permission denied
char device redirected to /dev/pts/0 (label serial1)
error: kvm run failed Function not implemented
PC=000000023faf924c SP=0000000240000020
X00=0000000000000000 X01=000000023fb1648c X02=0000000040000305 X03=0000000000000000
X04=00000000dbadc0de X05=1de7ec7edbadc0de X06=0000000000000004 X07=0000000000000000
X08=ffffffffffffc001 X09=ffffffffffffffff X10=0000000000000002 X11=0000000000000000
X12=00000000a007e03a X13=000000023fdadffc X14=000000000020d0d0 X15=00000000000000ff
X16=000000023fb87e30 X17=000000000020d0d0 X18=000000001f526ead X19=0000000000000000
X20=000000023fb47c18 X21=0000000000800080 X22=0000000000ff55aa X23=0000000000040000
X24=0000000004000000 X25=0000000000000000 X26=0000000000000118 X27=000000023fb87ca8
X28=00000000ffffffff X29=0000000000000000 X30=000000023faf9178
PSTATE=60000305 -ZC- EL1h

[Junien F (axino)]

One more, still with the "load/store instruction decoding not implemented" kernel message.

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -name instance-00166b4b -S -machine virt,accel=kvm,usb=off -cpu host -bios /usr/share/qemu-efi/QEMU_EFI.fd -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid fcdb656b-a09e-498f-9101-46768c3a0cde -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00166b4b.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-shutdown -boot strict=on -usb -drive file=/var/lib/nova/instances/fcdb656b-a09e-498f-9101-46768c3a0cde/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=unsafe -device virtio-blk-device,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/nova/instances/fcdb656b-a09e-498f-9101-46768c3a0cde/disk.swap,if=none,id=drive-virtio-disk1,format=qcow2,cache=unsafe -device virtio-blk-device,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-device,netdev=hostnet0,id=net0,mac=fa:16:3e:2f:db:9b -serial file:/var/lib/nova/instances/fcdb656b-a09e-498f-9101-46768c3a0cde/console.log -serial pty -device virtio-balloon-device,id=balloon0 -msg timestamp=on
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-iscsi.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-curl.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-rbd.so: failed to map segment from shared object: Permission denied
Failed to open module: /usr/lib/aarch64-linux-gnu/qemu/block-dmg.so: failed to map segment from shared object: Permission denied
BDB0126 mmap: Permission denied
char device redirected to /dev/pts/7 (label serial1)
error: kvm run failed Function not implemented
PC=000000023faf9248 SP=0000000240000030
X00=0000000000000000 X01=000000023fa747b4 X02=0000000060000305 X03=0000000000000000
X04=00000000dbadc0de X05=1de7ec7edbadc0de X06=0000000000000004 X07=000000023fa60430
X08=0000000000000000 X09=0000000237003741 X10=0000000000000000 X11=0000000000000000
X12=0000000237068908 X13=0000000237069150 X14=0000000000000001 X15=00000000fffffffe
X16=000000023fb87d00 X17=000000000020d0d0 X18=000000001f526ead X19=000000023fb87b48
X20=0000000000000001 X21=0000000000000000 X22=0000000237005720 X23=00000000000009d8
X24=0000000000000002 X25=0000000000000000 X26=0000000000000000 X27=0000000237069e0c
X28=00000002370b4990 X29=000000023fb87b20 X30=000000023faf9178
PSTATE=60000305 -ZC- EL1h

[dann frazier (dannf)]

I've got a system setup w/ a config as close to yours as I could, and I'm running VM start/stop loops to try and reproduce this problem. No luck so far, but I'm at just over 100 attempts at this point. One question - what version of qemu-efi are you using? It is possible that the bug lies there (see https://bugzilla.redhat.com/show_bug.cgi?id=1269779), so it'd be good to be testing w/ the same version you are using.

[dann frazier (dannf)]

libvirt xml from one of the failing guests would also be useful.

[dann frazier (dannf)]

fyi, I'm at over 15,000 VM launches w/o a reproduction.

Kernel: 4.8.0-040800rc8-generic
QEMU: 1:2.5+dfsg-5ubuntu10.5~cloud0
libvirt: 1.3.1-1ubuntu10.6~cloud0
aavmf: 0~20160104.c2a892d7-1+ppa1~ubuntu14.04.1

[Junien F (axino)]

Using qemu-efi 0~20150106.5c2d456b-2~14.04~ppa1

[dann frazier (dannf)]

OK, I was able to reproduce with that version after 792 attempts. Using a newer version, I was unable to reproduce in over 33,000 attempts. Since this does not impact a version of qemu-efi in any supported Ubuntu release, I'll mark invalid.

[dann frazier (dannf)]

Correction, marking Invalid for qemu - but Fix Released for qemu-efi, since it likely did impact 0~20150106.5c2d456b-2 back when that was a current version.