Unable to pass-through PCIe devices from a ppc64le host to an x86_64 guest

[cyrozap]

Attempting to pass through a PCIe device from a ppc64le host to an x86_64 guest with QEMU v5.2.0-3031-g571d413b5d (built from git master) fails with the following error:

    include/exec/memory.h:43:IOMMU_MEMORY_REGION: Object 0x10438eb00 is not an instance of type qemu:iommu-memory-region

To reproduce this issue, simply run the following command on a POWER9 system:

    qemu-system-x86_64 -machine q35 -device vfio-pci,host=$DBSF

Where $DBSF is a domain:bus:slot.function PCIe device address.

This also fails with QEMU 3.1.0 (from Debian Buster), so I assume this has never worked. Helpfully, the error message it prints seems to indicate where the problem is:

    hw/vfio/spapr.c:147:vfio_spapr_create_window: Object 0x164473510 is not an instance of type qemu:iommu-memory-region

My kernel (Linux v5.8.0 plus some small unrelated patches) is built with the page size set to 4k, so this issue shouldn't be due to a page size mismatch. And as I stated earlier, my host arch is ppc64le, so it shouldn't be an endianness issue, either.

I assume this should be possible (in theory) since I've seen reports of others getting PCIe passthrough working with aarch64 guests on x86_64 hosts, but of course that (passthrough to weird guest arch on x86) is somewhat the opposite of what I'm trying to do (passthrough to x86 on weird host arch) so I don't know for sure. If it is possible, I'm willing to develop a fix myself, but I'm almost completely unfamiliar with QEMU's internals so if anyone has any advice on where to start I'd greatly appreciate it.

[cyrozap (cyrozap)]

I've done some more investigating, and have produced a backtrace of the error:

    #0 0x00003ffff6b63228 in __libc_signal_restore_set (set=0x3fffffffcec8) at ../sysdeps/unix/sysv/linux/internal-signals.h:84
    #1 0x00003ffff6b63228 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:48
    #2 0x00003ffff6b4358c in __GI_abort () at abort.c:79
    #3 0x000000010080d524 in object_dynamic_cast_assert
        (obj=0x1016db860, typename=0x100bf9980 "qemu:iommu-memory-region", file=0x100bf9940 "/usr/src/qemu/include/exec/memory.h", line=<optimized out>, func=0x100c08a70 <__func__.21845> "IOMMU_MEMORY_REGION") at ../qom/object.c:883
    #4 0x00000001006b6f84 in IOMMU_MEMORY_REGION (obj=<optimized out>) at /usr/src/qemu/include/exec/memory.h:42
    #5 0x00000001006b6f84 in vfio_spapr_create_window (container=0x102357380, section=0x3fffffffd410, pgsize=0x3fffffffd368)
        at ../hw/vfio/spapr.c:149
    #6 0x00000001007a09a0 in vfio_listener_region_add (listener=0x102357390, section=0x3fffffffd410) at ../hw/vfio/common.c:709
    #7 0x00000001006ea6f4 in listener_add_address_space (as=<optimized out>, listener=0x102357390) at ../softmmu/memory.c:2729
    #8 0x00000001006ea6f4 in memory_listener_register (listener=0x102357390, as=<optimized out>) at ../softmmu/memory.c:2796
    #9 0x00000001007a36f4 in vfio_connect_container (errp=0x3fffffffe818, as=<optimized out>, group=0x102357300)
        at ../hw/vfio/common.c:1886
    #10 0x00000001007a36f4 in vfio_get_group (groupid=<optimized out>, as=<optimized out>, errp=0x3fffffffe818)
        at ../hw/vfio/common.c:2003
    #11 0x000000010071a2a8 in vfio_realize (pdev=0x102350f80, errp=0x3fffffffe818) at ../hw/vfio/pci.c:2834
    #12 0x0000000100488e20 in pci_qdev_realize (qdev=0x102350f80, errp=0x3fffffffe940) at ../hw/pci/pci.c:2113
    #13 0x00000001008063e0 in device_set_realized (obj=0x102350f80, value=<optimized out>, errp=0x3fffffffea50) at ../hw/core/qdev.c:761
    #14 0x000000010080afbc in property_set_bool
        (obj=0x102350f80, v=<optimized out>, name=<optimized out>, opaque=0x1014f1930, errp=0x3fffffffea50) at ../qom/object.c:2257
    #15 0x000000010080ee2c in object_property_set (obj=0x102350f80, name=0x100c023a0 "realized", v=
        0x102351d30, errp=0x101450b30 <error_fatal>) at ../qom/object.c:1402
    #16 0x000000010080a55c in object_property_set_qobject
        (obj=0x102350f80, name=0x100c023a0 "realized", value=<optimized out>, errp=0x101450b30 <error_fatal>) at ../qom/qom-qobject.c:28
    #17 0x000000010080f1b0 in object_property_set_bool
        (obj=0x102350f80, name=0x100c023a0 "realized", value=<optimized out>, errp=0x101450b30 <error_fatal>) at ../qom/object.c:1472
    #18 0x00000001008042bc in qdev_realize (dev=0x102350f80, bus=<optimized out>, errp=0x101450b30 <error_fatal>)
        at ../hw/core/qdev.c:389
    #19 0x000000010036cfac in qdev_device_add (opts=0x1014e9960, errp=0x101450b30 <error_fatal>)
        at /usr/src/qemu/include/hw/qdev-core.h:17
    #20 0x00000001006d5e68 in device_init_func (opaque=<optimized out>, opts=<optimized out>, errp=<optimized out>)
        at ../softmmu/vl.c:1202
    #21 0x0000000100abe070 in qemu_opts_fore...

[Laurent Vivier (laurent-vivier)]

I think cross-arch VFIO has already been discussed in:

https://bugs.launchpad.net/qemu/+bug/1869006

Perhaps you will have some answers.

[cyrozap (cyrozap)]

> https://bugs.launchpad.net/qemu/+bug/1869006

Unfortunately, that's not the same issue I'm having, and the error I see happens regardless of how much or how little RAM I allocate to the VM.

[Thomas Huth (th-huth)]

The QEMU project is currently moving its bug tracking to another system.
For this we need to know which bugs are still valid and which could be
closed already. Thus we are setting the bug state to "Incomplete" now.

If the bug has already been fixed in the latest upstream version of QEMU,
then please close this ticket as "Fix released".

If it is not fixed yet and you think that this bug report here is still
valid, then you have two options:

1) If you already have an account on gitlab.com, please open a new ticket
for this problem in our new tracker here:

    https://gitlab.com/qemu-project/qemu/-/issues

and then close this ticket here on Launchpad (or let it expire auto-
matically after 60 days). Please mention the URL of this bug ticket on
Launchpad in the new ticket on GitLab.

2) If you don't have an account on gitlab.com and don't intend to get
one, but still would like to keep this ticket opened, then please switch
the state back to "New" or "Confirmed" within the next 60 days (other-
wise it will get closed as "Expired"). We will then eventually migrate
the ticket automatically to the new system (but you won't be the reporter
of the bug in the new system and thus you won't get notified on changes
anymore).

Thank you and sorry for the inconvenience.

[cyrozap (cyrozap)]

I've moved this bug over to GitLab here: https://gitlab.com/qemu-project/qemu/-/issues/391

[Thomas Huth (th-huth)]

Thanks for moving it over! ... so I'm closing this on Launchpad now.