Boot one VM with two GPU(in same numa)by pci passthrough cannot have GPUDirect P2P capability

[dazhaoyu]

Hi,
I have two GPU cards, all of them was connect with one same numa CPU socket as below link info:
https://paste.opendev.org/show/b7Qi8qCnbLVxO2W0JdQw/

I can boot one nova instance successfully with the two GPU cards by PCI Passthrough way.

but in the booted instances, use deviceQuery method would get the below message:
Peer access from NVIDIA RTX 6000(GPU0) -> NVIDIA RTX 6000(GPU1): NO
Peer access from NVIDIA RTX 6000(GPU1) -> NVIDIA RTX 6000(GPU0): NO

The expected return should be as below：
Peer access from NVIDIA RTX 6000(GPU0) -> NVIDIA RTX 6000(GPU1): YES
Peer access from NVIDIA RTX 6000(GPU1) -> NVIDIA RTX 6000(GPU0): YES

so that the memory can be shared between the two GPUs.

I'm running Openstack Xena release in Intel Xeon Gold 5220R CPU

[Sylvain Bauza (sylvain-bauza)]

This seems very unrelated to the Nova project as this seems to be coming from the Nvidia driver, right ?
If so, sorry but we can't help you about it, we just pass the PCI device directly to the guest by using qemu.

[Sylvain Bauza (sylvain-bauza)]

Please reopen the bug report by changing the status back to new if you think it's related to Nova.

[dazhaoyu (lufus)]

yes, I think it's related to Nova, reopen this bug, thanks