ccing bug for the record on testing On Wed, May 10, 2017 at 8:33 AM, Nate Watterson wrote: > Ok, what is the physical device that ./fio.virtualfs is being created > on? Is it and NVME or SATA (internal or PCI attached) disk? > > -----Original Message----- > From: Manoj Iyer [mailto:] > Sent: Wednesday, May 10, 2017 9:06 AM > To: Nate Watterson > Cc: Manoj Iyer ; Timur Tabi > > Subject: RE: how to test iommu.passthough=0/1 ?? > > On Wed, 10 May 2017, Nate Watterson wrote: > >> >> Manoj, >> >> What was the backing store for the file mounted as /dev/loop0 in >> the results >> you sent? > > dd if=/dev/zero of=./fio.virtualfs bs=1M count=5120 > mkfs.ext4 ./fio.virtualfs > sudo losetup /dev/loop0 ./fio.virtualfs > > >> >> >> >> For iperf, I typically only test the target device as the iperf >> server and >> directly connect a host PC to run the client threads. I test this >> way >> because it usually has the worst performance with SMMU enabled. >> >> >> >> Target CMD: iperf -s >> >> Client CMD: iperf -c 192.168.2.222 -fg -i10 -t60 -P8 >> >> >> >> -Nate >> >> >> >> From: Manoj Iyer [mailto:] >> Sent: Tuesday, May 9, 2017 10:29 PM >> To: Nate Watterson >> Cc: Timur Tabi >> Subject: RE: how to test iommu.passthough=0/1 ?? >> >> >> >> Nate, >> >> >> >> Thanks for that info. I will give that a try. Did you see my iperf >> results ? >> those were kind of odd too.. I thought with the mlx card I might be >> testing >> with PCIe device .. but may I was hitting a limitation with iperf.. >> >> >> >> Also, I tested the same kernel on Cavium system, and with >> iommu.passthough=1/0 was I seeing a significant difference in the >> iops with >> a file mounted to /dev/loop0. Its attached here.. >> >> >> >> >> On Tue, May 9, 2017 at 8:52 PM, Nate Watterson >> >> wrote: >> >> Hi Manoj, >> >> Sorry for the late reply. I finally found some time to test >> passthrough mode and verified using the SMMU performance >> counters that the ‘iommu.passthrough’ param is working as >> I >> would expect it to (1 == Passthrough / 0 == non-Passthrough). >> I >> am not sure why you’re seeing this strange behavior but >> based on >> the logs you’ve sent, I have a few comments: >> >> >> >> Your boot log is showing that only the PCIe SMMUs are enabled >> (this is good, it means your FW is recent). Because of this, >> you >> will need to do your testing with a PCIe attached device >> instead >> of the SD card or SSD (assuming it was connected to the >> onboard >> SATA). I would recommend using an NVME drive as the impact of >> enabling passthrough mode will only really be apparent with >> devices capable of sustaining high IO throughput. >> >> >> >> I would recommend changing your FIO profile to access the >> device >> directly instead of going through the filesystem. >> Additionally, >> I would recommend using a readonly profile without >> randomization >> to preserve the disk and get consistent results. The >> following >> is the FIO command that I use: >> >> >> >> fio --name=global --readonly --group_reporting \ >> >> --direct=1 --ioengine=libaio --rw=read --eta-newline=1s \ >> >> --size=1T --blocksize=512k --iodepth=32 --numjobs=1 >> --runtime=10s \ >> >> --name=nvme_0 --filename=/dev/nvme0n1 >> >> >> >> *** iommu.passthrough=1 *** >> >> READ: io=25301MB, aggrb=2528.3MB/s, minb=2528.3MB/s, >> maxb=2528.3MB/s, mint=10007msec, maxt=10007msec >> >> >> >> *** iommu.passthrough=0 *** >> >> READ: io=22657MB, aggrb=2265.5MB/s, minb=2265.5MB/s, >> maxb=2265.5MB/s, mint=10001msec, maxt=10001msec >> >> >> >> Hopefully this helps. Let me know if you have any more >> questions. >> >> -Nate >> >> >> >> From: Manoj Iyer [mailto:] >> Sent: Thursday, May 4, 2017 4:55 PM >> To: Nate Watterson >> Cc: Timur Tabi >> Subject: RE: how to test iommu.passthough=0/1 ?? >> >> >> >> Nate, >> >> >> >> I am attaching fio output for the CAF kernel that is available in >> the >> centriq PPA. The results are not that much different. This time I >> ran >> fio on the loop back disk and also on the mmcblk0 SD card. >> >> >> >> Please see the attached test report. >> >> >> >> Thanks >> >> Manoj Iyer >> >> On Thu, May 4, 2017 at 2:46 PM, Manoj Iyer >> >> wrote: >> >> I have attached the boot log from the system with >> iommu.passthrough=1. >> >> >> >> Also I ran iperf on the mlx card interface.. and again the >> results don't make sense. I was expecting see better numbers >> with iommu.passthrough=1. Please see the attached test data for >> UDP and TCP. >> >> >> >> Figuring out what firmware I have was always a challenge. I had >> brought this up with the fw folks that there is no way to say I >> am on version X Y or Z.. anyways.. >> >> >> >> FW on this SDP: >> >> XBL.DF.2.0.R1-00406 QDF2400_REL CRM >> >> >> >> Type: FF6D3C53-2748-4017-838F-07BACCC91341 >> >> Version: 0000016C >> >> Version Name: QDF2400.FW.2.0.R1-00364 bait-qsvbuild-24-bldr-lnx >> >> >> >> Which I think translates to FW: >> >> commit 891a97c41712112ce38ba70220c9ca85c9e52869 >> >> Author: QC Publisher >> >> Date: Mon Apr 17 13:02:52 2017 -0700 >> >> >> >> Commit label r00364.1 - N/A 0.0.364.1 >> >> >> >> TRACKING-ID:920c38a9-68f4-4218-870e-e9c593e321ad >> >> >> >> >> >> >> On Thu, May 4, 2017 at 2:21 PM, Nate Watterson >> wrote: >> >> Ok that makes a bit more sense. I still need the >> boot log though. Is the USB device connected to one >> of the onboard USB controllers or a PCI based USB >> controller? Do you happen to know what FW version >> you’re running? >> >> -Nate >> >> >> >> From: Manoj Iyer [mailto:] >> Sent: Thursday, May 4, 2017 2:56 PM >> To: Nate Watterson >> Cc: Timur Tabi >> Subject: RE: how to test iommu.passthough=0/1 ?? >> >> >> >> Opps my bad.. I am terribly sorry.. /dev/sdd is a >> USB device. >> >> >> >> *-usb:2 >> >> description: Mass storage device >> >> product: ADATA USB Flash Drive >> >> vendor: ADATA >> >> description: SCSI Disk >> >> physical id: 0.0.0 >> >> bus info: scsi@9:0.0.0 >> >> logical name: /dev/sdd >> >> >> >> >> On Thu, May 4, 2017 at 1:30 PM, Nate Watterson >> wrote: >> >> Please send the boot log. How is the SD card >> attached? I would have expected it to show up >> as an mmcblk device but it appears to be >> showing up at /dev/sdd. >> >> -Nate >> >> >> >> From: Nate Watterson >> Sent: Thursday, May 4, 2017 2:12 PM >> To: 'Manoj Iyer' >> Cc: Timur Tabi >> Subject: RE: how to test iommu.passthough=0/1 >> ?? >> >> >> >> Some of those results are shocking. I’ll try your >> FIO profile here a bit later and let you know what I >> get. >> >> -Nate >> >> >> >> From: Manoj Iyer [mailto:] >> Sent: Thursday, May 4, 2017 1:00 PM >> To: Nate Watterson >> Cc: Timur Tabi >> Subject: RE: how to test iommu.passthough=0/1 ?? >> >> >> >> I tried fio on a file on SSD mounted to loopback >> device, and on the SD card on the AW SDP. I dont >> have any additional disk on it so I am limited to >> those options. But the results are a bit puzzling.. >> I see lower iops for iommu.passthough=1. I have >> attached the test and output, may be I am looking at >> this wrong... >> >> >> >> Any thoughts? >> >> On Thu, May 4, 2017 at 10:25 AM, Nate Watterson >>