ccing bug for the record on testing.

On Wed, May 10, 2017 at 10:07 AM, Nate Watterson 
<email address hidden> wrote:
> Unless this is a PCIe attached SATA disk, you will not see any impact 
> from the iommu.passthrough setting since the SATA SMMUs are not even 
> enabled! Additionally, the FIO results you sent last week I think 
> showed rates beyond the physical capabilities of SATA so you should 
> probably make sure that file system caching is disabled or just 
> access the disk directly using a readonly profile.  Also, the SMMU is 
> sufficiently fast that you will typically not even see the SMMU 
> impact on a relatively low BW peripheral like SATA. I would recommend 
> getting an NVME drive.
> -Nate
> 
> From: Manoj Iyer [mailto:<email address hidden>]
> Sent: Wednesday, May 10, 2017 10:05 AM
> To: Nate Watterson <email address hidden>
> Cc: Timur Tabi <email address hidden>
> Subject: RE: how to test iommu.passthough=0/1 ??
> 
> 
> 
> On Wed, May 10, 2017 at 8:33 AM, Nate Watterson 
> <email address hidden> wrote:
> 
>> Ok, what is the physical device that ./fio.virtualfs is being 
>> created on? Is it and NVME or SATA (internal or PCI attached) disk?
> 
> Ah! it was on the SATA drive. Sorry misunderstood your initial 
> question.
> 
> 
>> -----Original Message----- From: Manoj Iyer 
>> [mailto:<email address hidden>] Sent: Wednesday, May 10, 2017 9:06 
>> AM To: Nate Watterson <email address hidden> Cc: Manoj Iyer 
>> <email address hidden>; Timur Tabi <email address hidden> 
>> Subject: RE: how to test iommu.passthough=0/1 ?? On Wed, 10 May 
>> 2017, Nate Watterson wrote:
>>> Manoj, What was the backing store for the file mounted as 
>>> /dev/loop0 in the results you sent?
>> dd if=/dev/zero of=./fio.virtualfs bs=1M count=5120 mkfs.ext4 
>> ./fio.virtualfs sudo losetup /dev/loop0 ./fio.virtualfs
>>>   For iperf, I typically only test the target device as the iperf 
>>> server and directly connect a host PC to run the client threads. I 
>>> test this way because it usually has the worst performance with 
>>> SMMU enabled.   Target CMD: iperf -s Client CMD: iperf -c 
>>> 192.168.2.222 -fg -i10 -t60 -P8   -Nate   From: Manoj Iyer 
>>> [mailto:<email address hidden>] Sent: Tuesday, May 9, 2017 10:29 
>>> PM To: Nate Watterson <email address hidden> Cc: Timur Tabi 
>>> <email address hidden> Subject: RE: how to test 
>>> iommu.passthough=0/1 ??   Nate,   Thanks for that info. I will give 
>>> that a try. Did you see my iperf results ? those were kind of odd 
>>> too.. I thought with the mlx card I might be testing with PCIe 
>>> device .. but may I was hitting a limitation with iperf..    Also, 
>>> I tested the same kernel on Cavium system, and with 
>>> iommu.passthough=1/0 was I seeing a significant difference in the 
>>> iops with a file mounted to /dev/loop0. Its attached here..   On 
>>> Tue, May 9, 2017 at 8:52 PM, Nate Watterson 
>>> <email address hidden> wrote: Hi Manoj, Sorry for the late 
>>> reply. I finally found some time to test passthrough mode and 
>>> verified using the SMMU performance counters that the 
>>> ‘iommu.passthrough’ param is working as I would expect it to (1 
>>> == Passthrough / 0 == non-Passthrough). I am not sure why you’re 
>>> seeing this strange behavior but based on the logs you’ve sent, I 
>>> have a few comments:   Your boot log is showing that only the PCIe 
>>> SMMUs are enabled (this is good, it means your FW is recent). 
>>> Because of this, you will need to do your testing with a PCIe 
>>> attached device instead of the SD card or SSD (assuming it was 
>>> connected to the onboard SATA). I would recommend using an NVME 
>>> drive as the impact of enabling passthrough mode will only really 
>>> be apparent with devices capable of sustaining high IO throughput.  
>>>  I would recommend changing your FIO profile to access the device 
>>> directly instead of going through the filesystem. Additionally, I 
>>> would recommend using a readonly profile without randomization to 
>>> preserve the disk and get consistent results.  The following is the 
>>> FIO command that I use:   fio --name=global --readonly 
>>> --group_reporting \ --direct=1 --ioengine=libaio --rw=read 
>>> --eta-newline=1s \ --size=1T --blocksize=512k --iodepth=32 
>>> --numjobs=1 --runtime=10s \ --name=nvme_0 --filename=/dev/nvme0n1   
>>> *** iommu.passthrough=1 *** READ: io=25301MB, aggrb=2528.3MB/s, 
>>> minb=2528.3MB/s, maxb=2528.3MB/s, mint=10007msec, maxt=10007msec   
>>> *** iommu.passthrough=0 *** READ: io=22657MB, aggrb=2265.5MB/s, 
>>> minb=2265.5MB/s, maxb=2265.5MB/s, mint=10001msec, maxt=10001msec   
>>> Hopefully this helps. Let me know if you have any more questions. 
>>> -Nate   From: Manoj Iyer [mailto:<email address hidden>] Sent: 
>>> Thursday, May 4, 2017 4:55 PM To: Nate Watterson 
>>> <email address hidden> Cc: Timur Tabi <email address hidden> 
>>> Subject: RE: how to test iommu.passthough=0/1 ??   Nate,   I am 
>>> attaching fio output for the CAF kernel that is available in the 
>>> centriq PPA. The results are not that much different. This time I 
>>> ran fio on the loop back disk and also on the mmcblk0 SD card.   
>>> Please see the attached test report.    Thanks Manoj Iyer On Thu, 
>>> May 4, 2017 at 2:46 PM, Manoj Iyer <email address hidden> 
>>> wrote: I have attached the boot log from the system with 
>>> iommu.passthrough=1.    Also I ran iperf on the mlx card 
>>> interface.. and again the results don't make sense. I was expecting 
>>> see better numbers with iommu.passthrough=1. Please see the 
>>> attached test data for UDP and TCP.   Figuring out what firmware I 
>>> have was always a challenge. I had brought this up with the fw 
>>> folks that there is no way to say I am on version X Y or Z.. 
>>> anyways..    FW on this SDP:  XBL.DF.2.0.R1-00406 QDF2400_REL CRM   
>>> Type: FF6D3C53-2748-4017-838F-07BACCC91341 Version: 0000016C  
>>> Version Name: QDF2400.FW.2.0.R1-00364 bait-qsvbuild-24-bldr-lnx   
>>> Which I think translates to FW: commit 
>>> 891a97c41712112ce38ba70220c9ca85c9e52869 Author: QC Publisher 
>>> <email address hidden> Date:   Mon Apr 17 13:02:52 2017 
>>> -0700       Commit label r00364.1 - N/A 0.0.364.1          
>>> TRACKING-ID:920c38a9-68f4-4218-870e-e9c593e321ad     On Thu, May 4, 
>>> 2017 at 2:21 PM, Nate Watterson <email address hidden> wrote: 
>>> Ok that makes a bit more sense. I still need the boot log though. 
>>> Is the USB device connected to one of the onboard USB controllers 
>>> or a PCI based USB controller? Do you happen to know what FW 
>>> version you’re running? -Nate   From: Manoj Iyer 
>>> [mailto:<email address hidden>] Sent: Thursday, May 4, 2017 2:56 
>>> PM To: Nate Watterson <email address hidden> Cc: Timur Tabi 
>>> <email address hidden> Subject: RE: how to test 
>>> iommu.passthough=0/1 ??   Opps my bad.. I am terribly sorry.. 
>>> /dev/sdd is a USB device.     *-usb:2              description: 
>>> Mass storage device              product: ADATA USB Flash Drive     
>>>          vendor: ADATA                 description: SCSI Disk       
>>>           physical id: 0.0.0                 bus info: scsi@9:0.0.0 
>>>                 logical name: /dev/sdd   On Thu, May 4, 2017 at 
>>> 1:30 PM, Nate Watterson <email address hidden> wrote: Please 
>>> send the boot log. How is the SD card attached? I would have 
>>> expected it to show up as an mmcblk device but it appears to be 
>>> showing up at /dev/sdd. -Nate   From: Nate Watterson Sent: 
>>> Thursday, May 4, 2017 2:12 PM To: 'Manoj Iyer' 
>>> <email address hidden> Cc: Timur Tabi <email address hidden> 
>>> Subject: RE: how to test iommu.passthough=0/1 ??   Some of those 
>>> results are shocking. I’ll try your FIO profile here a bit later 
>>> and let you know what I get. -Nate   From: Manoj Iyer 
>>> [mailto:<email address hidden>] Sent: Thursday, May 4, 2017 1:00 
>>> PM To: Nate Watterson <email address hidden> Cc: Timur Tabi 
>>> <email address hidden> Subject: RE: how to test 
>>> iommu.passthough=0/1 ??   I tried fio on a file on SSD mounted to 
>>> loopback device, and on the SD card on the AW SDP. I dont have any 
>>> additional disk on it so I am limited to those options. But the 
>>> results are a bit puzzling.. I see lower iops for 
>>> iommu.passthough=1. I have attached the test and output, may be I 
>>> am looking at this wrong...    Any thoughts?  On Thu, May 4, 2017 
>>> at 10:25 AM, Nate Watterson <email address hidden> wrote: 
>>> Manoj, I don’t think there will be anything obvious in dmesg you 
>>> can check. You’ll pretty much have to run an IO benchmark (eg: 
>>> iperf/fio) in the *host* kernel on a device with sufficient IO 
>>> throughput to show the effects of the change. You should see a 
>>> significant performance benefit to having iommu.passthrough==1. 
>>> Note that this will have absolutely no impact at all on IO 
>>> performance within a VM. Let me know if you have any other 
>>> questions.   -Nate   From: Manoj Iyer 
>>> [mailto:<email address hidden>] Sent: Thursday, May 4, 2017 1:26 
>>> AM To: Timur Tabi <email address hidden>; Nate Watterson 
>>> <email address hidden> Subject: how to test 
>>> iommu.passthough=0/1 ??   Nate/Timur,   I backported the SMMU pass 
>>> though using default domain patch series from linux-next. I have a 
>>> public bug filed 
>>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1688158 to SRU 
>>> this patch series. But I cant figure out a way of testing 
>>> iommu.passthough=0/1. Could you pls suggest a test and tell me what 
>>> to look for dmesg etc to compare both cases? I understand it is 
>>> supposed to improve performance is there a way of checking that ?   
>>>  I compared dmesg on both cases, and also launched vms and looked 
>>> to see if  I can spot anything obvious.. Really appreciate your 
>>> help with suggestions on testing this. I need to post some testing 
>>> data with my SRU request.    Thanks Manoj Iyer.
>> -- ============================ Manoj Iyer Ubuntu/Canonical ARM 
>> Servers - Cloud ============================