PCI broken in qemu ppc e500 in v2.12.0 and other versions

[ecsdn]

The same qemu -M mpc... command that works on qemu-system-ppc version 2.8.0 freezes guest on bootup and shows error for qemu-system-ppc version 4.2.0release and 4.19dirtygit:

qemu-system-ppc: virtio-blk failed to set guest notifier (-24), ensure -accel kvm is set.
qemu-system-ppc: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).

ends/freezes at:
nbd: registered device at major 43
 vda:

I'm using -drive file=/home/me/rawimage.dd,if=virtio and works fine in version 2.8.0 installed with apt-get install (Ubuntu 17.04) and also with 2.8.0 official release from git/github that I compiled/built myself. But both of the newer releases fail on the same exact machine same config.

I also noticed that qemu-2.8.0 was fine with mtd but the newer ones I tried weren't, ie gave
qemu-system-ppc: -drive if=mtd: machine type does not support if=mtd,bus=0,unit=0
(but I removed -drive if=mtd since wasn't using it anyway)

I also tried on windows but I think virtio doesn't work on windows hosts at all? On windows host it fails the same way, even version 2.12 as well as 4.1.10...

used:
./configure --prefix=/opt/... --enable-fdt --enable-kvm --enable-debug

(basically all steps the same on same exact system same config, yet 2.8.0 works fine whether apt-get installed or built from source while the others I built, 4.19/4.2.0 or 2.12/4.1.10(win) don't.)

In case newer qemu versions act weird on various kernels, I did try with both vmlinuz-4.10.0-19-generic and vmlinuz-4.13.12-041312-generic (I didn't compile them but I can provide config-..files. This is on Ubuntu 17.04 x86_64 host emulating e500v2 cpm guest, ie -M mpc... GUEST kernel 2.6.32.44 which is why I can't use -M ppce500 instead..)
tx
     ecs

[ecsdn (ecsdn)]

Also tested on another system (Debian GNU/Linux 9 \n \l with kernel SMP Debian 3.16.56-1+deb8u1 (2018-05-08) x86_64) besides the previous Ubuntu 17.04 and confirmed even Qemu 2.8.1 is working but Qemu 3.1.10 and higher not working, virtio fails/freezes guest at vda as on the other system.

[Laurent Vivier (laurent-vivier)]

Could you provide the full qemu command line?

[ecsdn (ecsdn)]

Did you try with just a basic virtio disk and it works for you?

Because even a basic virtio drive addition fails for me, even this fails on higher than 2.8.1

For example:
qemu-system-ppc -M mpc8544ds -nographic -kernel /home/me/boot/uImage-2.6.32 -append "root=/dev/vda rw" -drive file=/home/me/mmcblk0p2.dd,if=virtio

The only thing I can think of, is if somehow vda fails due to me running the higher version qemu binaries from their build locations/paths without actually "installing" them in usual paths etc.

But I ran the 2.8 version from build location I compiled it from and it worked from there, but perhaps the 2.8 version was also the distro installed default one so maybe it found dependencies it needed?

Anyway I just now reconfigured 4.2.0 with --prefix /opt/qemu4.2.0 and ran it from installed dir:

root@myserver:/opt/qemu4.2.0/bin# ./qemu-system-ppc -M mpc8544ds -nographic -kernel /home/me/boot/uImage-2.6.32 -append "root=/dev/vda rw" -drive file=/home/me/mmcblk0p2.dd,if=virtio

But it still fails even after make install and running it from the /opt/qemu4.2.0/bin directory.
Is it somehow conflicting with the other qemu version 2.8.. installed by usual apt-get install?

Regardless of how I start them, version 3.1.0 and 4.2.0rc4 and some other 4.19git and 4.2.0final all fail/freeze at:
"
....
nbd: registered device at major 43
 vda:
"

[Laurent Vivier (laurent-vivier)]

Perhaps you can try to disable the "modern" mode of virtio (The endianness of the API has been changed):

replace

  -drive file=/home/me/mmcblk0p2.dd,if=virtio

by

  -device virtio-blk-pci,drive=drive0,disable-modern=true \
  -drive file=mmcblk0p2.dd,if=none,id=drive0,format=raw

[ecsdn (ecsdn)]

Thanks I tried with:

/root/QEMU/qemu-git-4.2.0rc4/qemu/build/ppc-softmmu/qemu-system-ppc -M mpc8544ds -nographic -kernel /home/me/boot/uImage-2.6.32 -append "root=/dev/vda rw" -device virtio-blk-pci,drive=drive0,disable-modern=true -drive file=/home/me/mmcblk0p2.dd,if=none,id=drive0,format=raw

And again it worked with qemu 2.8.1 but failed with the above 4.2.0rc4 on the same x86_64 host.

On another x86_64 host I confirmed that the below works with qemu 2.8.0

root@myserver:~# qemu-system-ppc -M mpc8544ds -nographic -kernel /home/me/boot/uImage-2.6.32 -append "root=/dev/vda rw" -device virtio-blk-pci,drive=drive0,disable-modern=true -drive file=/home/me/mmcblk0p2.dd,if=none,id=drive0,format=raw

But again even on this system 4.2.0 failes with that same command:
root@myserver:~# /root/QEMU/qemu-4.2.0/build/ppc-softmmu/qemu-system-ppc -M mpc8544ds -nographic -kernel /home/me/boot/uImage-2.6.32 -append "root=/dev/vda rw" -device virtio-blk-pci,drive=drive0,disable-modern=true -drive file=/home/me/mmcblk0p2.dd,if=none,id=drive0,format=raw

Fails/freezes at the same vda: location.

Running it from its installed location didn't help, the following still failed at vda: also.

root@myserver:/opt/qemu4.2.0/bin# ./qemu-system-ppc -M mpc8544ds -nographic -kernel /home/me/boot/uImage-2.6.32 -append "root=/dev/vda rw" -device virtio-blk-pci,drive=drive0,disable-modern=true -drive file=/home/me/mmcblk0p2.dd,if=none,id=drive0,format=raw

Although I didn't think its required for the softmmu qemu "emulation" only, ie not "kvm", I even enabled kvm as well as DMAR+IOMMU on the kernel and recompiled 4.2.0 but had same vda: failure.

[ecsdn (ecsdn)]

fyi from what I recall guest kernel was built using mpc85xx_defconfig with some additions like virtio etc. If virtio is working for you just fine using same command as mine, then perhaps its some peculiarity to do with my specific guest kernel or kernel version? (uImage is about 3.4M with equivalent vmlinux about 72M)

[ecsdn (ecsdn)]

Hope you enjoyed the Holidays, Happy 2020! I would really appreciate if you could confirm for me if virtio works fine for you with ppc -M mpc8544ds with older Linux guest kernels like 2.6.32
thanks!

[Laurent Vivier (laurent-vivier)]

Could you provide your binary uImage-2.6.32?

[ecsdn (ecsdn)]

With some precautionary measures I think I can provide it. Not sure what of our drivers may already be compiled in etc so I need to send it to you privately so only you have access for testing etc after which you would delete it once issue fixed or discovered etc. Is it possible to send you private message on here with such a link or better email? thanks

[ecsdn (ecsdn)]

Sorry for the delay, I have sent you a private message/email with the actual kernel image. thx!

[Laurent Vivier (laurent-vivier)]

This is broken by:

commit 67113c03423a23e60915574275aed7d60e9f85e1
Author: Michael Davidsaver <email address hidden>
Date: Sun Nov 26 15:59:05 2017 -0600

    e500: fix pci host bridge class/type

    Correct some confusion wrt. the PCI facing
    side of the PCI host bridge (not PCIe root complex).
    The ref. manual for the mpc8533 (as well as
    mpc8540 and mpc8540) give the class code as
    PCI_CLASS_PROCESSOR_POWERPC.
    While the PCI_HEADER_TYPE field is oddly omitted,
    the tables in the "PCI Configuration Header"
    section shows a type 0 layout using all 6 BAR
    registers (as 2x 32, and 2x 64 bit regions)

    So 997505065dc92e533debf5cb23012ba4e673d387
    seems to be in error. Although there was
    perhaps some confusion as the mpc8533
    has a separate PCIe root complex.
    With PCIe, a root complex has PCI_HEADER_TYPE=1.

    Neither the PCI host bridge, nor the PCIe
    root complex advertise class PCI_CLASS_BRIDGE_PCI.

    This was confusing Linux guests, which try
    to interpret the host bridge as a pci-pci
    bridge, but get confused and re-enumerate
    the bus when the primary/secondary/subordinate
    bus registers don't have valid values.

    Signed-off-by: Michael Davidsaver <email address hidden>
    Signed-off-by: David Gibson <email address hidden>

[Laurent Vivier (laurent-vivier)]

If I revert 67113c03423a on top of master, vda is correctly detected.

[ecsdn (ecsdn)]

Thanks for all the help Laurent! I'm new to git so not surre how to 'properly' revert a previous commit on top of master, so I'll google, but if you have some a good link please do send.

Also, I've heard of the term "bisect" for figuring out at which commit something breaks and if there were some good documentation to spell out the steps to do that for users that aren't, well advanced kernel gurus :D , I'm sure we'd be happy to save you smarter guys time with any mundane testing steps when possible :) thx!

[Thomas Huth (th-huth)]

The QEMU project is currently considering to move its bug tracking to
another system. For this we need to know which bugs are still valid
and which could be closed already. Thus we are setting older bugs to
"Incomplete" now.

If you still think this bug report here is valid, then please switch
the state back to "New" within the next 60 days, otherwise this report
will be marked as "Expired". Or please mark it as "Fix Released" if
the problem has been solved with a newer version of QEMU already.

Thank you and sorry for the inconvenience.

[Launchpad Janitor (janitor)]

[Expired for QEMU because there has been no activity for 60 days.]