qemu-system-x86_64: "VFIO_MAP_DMA : -28" error when we attache 6 VF's to guest machine

[IndrasenaReddy Gali]

We are trying to attach 6 VF's to the guest machine on 4.1.1 qemu emulator.
We are observing "VFIO_MAP_DMA : -28" error.

We are using w-bits=48 bits while lunching VM.

[Thomas Huth (th-huth)]

Please provide information how you started QEMU, and some information about your PCI device (e.g. the output of lspci).

[IndrasenaReddy Gali (gisena)]

qemu-system-x86_64 -name guest=fedora24 -machine q35,accel=kvm,kernel-irqchip=split \
        -enable-kvm \
        -m 4G \
        -smp 8,sockets=1,cores=8,threads=1 \
        -device intel-iommu,intremap=on,caching-mode=on,aw-bits=48 \
        -drive file=<OS_IMAGE_FILE>,format=raw \
        -device ioh3420,id=pcie.1,chassis=1 \
        -device virtio-net-pci,bus=pcie.1,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on,netdev=net0 \
        -netdev user,id=net0,hostfwd=tcp::1111-:22\
        -device vfio-pci,host=3f:02.1 \
  -device vfio-pci,host=3f:02.2 \
  -device vfio-pci,host=3f:02.3 \
  -device vfio-pci,host=3f:02.4 \
        -device vfio-pci,host=3d:02.4 \
  -device vfio-pci,host=3d:02.5 \
  -device vfio-pci,host=3d:02.6 \
        -nographic

[IndrasenaReddy Gali (gisena)]

Please find the above qemu command to lunch guest machine

[Alex Williamson (alex-l-williamson)]

Presumably w-bits (aw-bits?) implies using intel-iommu, there's a opportunity for the vfio iommu backend to return -ENOSPC (-28) if we exceed the default number of in-flight DMA mappings per container. The default limit is 65535. You can try increasing this by changing the dma_entry_limit module option on the vfio_iommu_type1 module. Note that in a typical vIOMMU config there's a container per device, so the number of VFs attached is possibly not a factor. It is however a lot of DMA mappings for a single device if this is the issue and you'd generally want to boot the guest with iommu=pt in order to have reasonable assigned device performance with a vIOMMU, which would also greatly reduce the number of mappings.

[IndrasenaReddy Gali (gisena)]

After increasing dma_entry_limit limit no issue observed.

But ideal senario device is getting hung and recovery happening only with host hard rebooting.

[Launchpad Janitor (janitor)]

[Expired for QEMU because there has been no activity for 60 days.]