Comment 9 for bug 1688158

Thanks Jeff/Nate I will sru these patches shortly.

On Mon, May 15, 2017 at 2:55 PM, Jeff Hugo <email address hidden>
wrote:
> It’s working as expected. I see improvement with passthrough
> enabled (see below). Nate also confirmed the SMMU setup via JTAG.
>
> Passthrough=0 -
>
> root@null-8cfdf006971f:/home/ubuntu# fio --name=global --readonly
> --group_reporting --direct=1 --ioengine=libaio --rw=read
> --eta-newline=1s --size=1T --blocksize=512k --iodepth=32 --numjobs=1
> --runtime=10s --name=nvme_0 --filename=/dev/nvme0n1
> nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K,
> ioengine=libaio, iodepth=32
> fio-2.2.10
> Starting 1 process
> Jobs: 1 (f=1): [R(1)] [36.4% done] [2540MB/0KB/0KB /s] [5080/0/0
> iops] [eta 00m:07s]
> Jobs: 1 (f=1): [R(1)] [45.5% done] [2539MB/0KB/0KB /s] [5077/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [2540MB/0KB/0KB /s]
> [5079/0/0 iops] [eta 00m:05s]
> Jobs: 1 (f=1): [R(1)] [63.6% done] [2540MB/0KB/0KB /s] [5079/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [2539MB/0KB/0KB /s]
> [5078/0/0 iops] [eta 00m:03s]
> Jobs: 1 (f=1): [R(1)] [81.8% done] [2539MB/0KB/0KB /s] [5078/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [2540MB/0KB/0KB /s]
> [5080/0/0 iops] [eta 00m:01s]
> Jobs: 1 (f=1): [R(1)] [100.0% done] [2539MB/0KB/0KB /s] [5078/0/0
> iops] [eta 00m:00s]
> nvme_0: (groupid=0, jobs=1): err= 0: pid=36481: Mon May 15 17:10:26
> 2017
> read : io=25377MB, bw=2537.5MB/s, iops=5074, runt= 10001msec
> slat (usec): min=32, max=440, avg=191.74, stdev=157.14
> clat (usec): min=153, max=7641, avg=6106.54, stdev=107.40
> lat (usec): min=442, max=7904, avg=6298.45, stdev=174.49
> clat percentiles (usec):
> | 1.00th=[ 5920], 5.00th=[ 5984], 10.00th=[ 6048], 20.00th=[
> 6048],
> | 30.00th=[ 6112], 40.00th=[ 6112], 50.00th=[ 6112], 60.00th=[
> 6112],
> | 70.00th=[ 6112], 80.00th=[ 6112], 90.00th=[ 6176], 95.00th=[
> 6176],
> | 99.00th=[ 6304], 99.50th=[ 6368], 99.90th=[ 6432], 99.95th=[
> 6432],
> | 99.99th=[ 7008]
> bw (MB /s): min= 2498, max= 2540, per=99.98%, avg=2536.84,
> stdev= 9.44
> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%
> lat (msec) : 2=0.01%, 4=0.02%, 10=99.96%
> cpu : usr=3.84%, sys=96.12%, ctx=67, majf=0, minf=528
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued : total=r=50754/w=0/d=0, short=r=0/w=0/d=0,
> drop=r=0/w=0/d=0
> latency : target=0, window=0, percentile=100.00%, depth=32
>
> Run status group 0 (all jobs):
> READ: io=25377MB, aggrb=2537.5MB/s, minb=2537.5MB/s,
> maxb=2537.5MB/s, mint=10001msec, maxt=10001msec
>
> Disk stats (read/write):
> nvme0n1: ios=200550/0, merge=0/0, ticks=23848/0, in_queue=23828,
> util=91.79%
>
>
>
> passthrough=1 –
>
> root@null-8cfdf006971f:/home/ubuntu#
> ing --direct=1 --ioengine=libaio --rw=read --eta-newline=1s
> --size=1T --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s
> --name=nvme_0 --filename=/dev/nvme0n1
> nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K,
> ioengine=libaio, iodepth=32
> fio-2.2.10
> Starting 1 process
> Jobs: 1 (f=1): [R(1)] [36.4% done] [3238MB/0KB/0KB /s] [6476/0/0
> iops] [eta 00m:07s]
> Jobs: 1 (f=1): [R(1)] [45.5% done] [3238MB/0KB/0KB /s] [6476/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [3239MB/0KB/0KB /s]
> [6477/0/0 iops] [eta 00m:05s]
> Jobs: 1 (f=1): [R(1)] [63.6% done] [3238MB/0KB/0KB /s] [6476/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [3238MB/0KB/0KB /s]
> [6476/0/0 iops] [eta 00m:03s]
> Jobs: 1 (f=1): [R(1)] [81.8% done] [3238MB/0KB/0KB /s] [6476/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [3239MB/0KB/0KB /s]
> [6477/0/0 iops] [eta 00m:01s]
> Jobs: 1 (f=1): [R(1)] [100.0% done] [3238MB/0KB/0KB /s] [6476/0/0
> iops] [eta 00m:00s]
> nvme_0: (groupid=0, jobs=1): err= 0: pid=1600: Mon May 15 17:13:46
> 2017
> read : io=32384MB, bw=3236.8MB/s, iops=6473, runt= 10005msec
> slat (usec): min=13, max=210, avg=16.19, stdev= 6.40
> clat (usec): min=1505, max=9532, avg=4925.01, stdev=87.39
> lat (usec): min=1616, max=9566, avg=4941.38, stdev=86.47
> clat percentiles (usec):
> | 1.00th=[ 4896], 5.00th=[ 4896], 10.00th=[ 4896], 20.00th=[
> 4896],
> | 30.00th=[ 4896], 40.00th=[ 4960], 50.00th=[ 4960], 60.00th=[
> 4960],
> | 70.00th=[ 4960], 80.00th=[ 4960], 90.00th=[ 4960], 95.00th=[
> 4960],
> | 99.00th=[ 4960], 99.50th=[ 4960], 99.90th=[ 4960], 99.95th=[
> 4960],
> | 99.99th=[ 8640]
> bw (MB /s): min= 3209, max= 3237, per=99.96%, avg=3235.57,
> stdev= 6.18
> lat (msec) : 2=0.03%, 4=0.03%, 10=99.94%
> cpu : usr=1.80%, sys=16.99%, ctx=64755, majf=0, minf=528
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued : total=r=64767/w=0/d=0, short=r=0/w=0/d=0,
> drop=r=0/w=0/d=0
> latency : target=0, window=0, percentile=100.00%, depth=32
>
> Run status group 0 (all jobs):
> READ: io=32384MB, aggrb=3236.8MB/s, minb=3236.8MB/s,
> maxb=3236.8MB/s, mint=10005msec, maxt=10005msec
>
> Disk stats (read/write):
> nvme0n1: ios=255788/0, merge=0/0, ticks=1246388/0,
> in_queue=1246788, util=99.05%
>
> Jeffrey Hugo
> Senior Engineer
> Qualcomm Datacenter Technologies, Inc.
> 1-303-247-5002
>
> From: Manoj Iyer [mailto:<email address hidden>]
> Sent: Thursday, May 11, 2017 9:23 PM
> To: Jeff Hugo <email address hidden>; <email address hidden>
> Cc: Nate Watterson <email address hidden>; Timur Tabi
> <email address hidden>
> Subject: [Bug #1688158] Please help test kernel for
> iommu.passthrough=0/1
>
> Jeff,
>
> I have back-ported patches to address "Support SMMU passthrough using
> the default domain". This adds a new kernel command line option
> iommu.passthrough=0/1 to Configure DMA to bypass the IOMMU for ARM64.
> http://bugs.launchpad.net/bugs/1688158
>
> 0 - Use IOMMU translation for DMA.
> 1 - Bypass the IOMMU for DMA.
>
> The test case recommended by Nate using 'fio' is:
>
> $ sudo apt update ; sudo apt install fio
>
> $ sudo fio --name=global --readonly --group_reporting --direct=1
> --ioengine=libaio --rw=read --eta-newline=1s --size=1T
> --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s
> --name=nvme_0 --filename=/dev/nvme0n1
>
> You can find the kernel and the instructions to install the kernel in
> the Centriq test PPA:
> https://launchpad.net/~centriq-team/+archive/ubuntu/test/
>
> Nate and I have gone back and forth on testing these patches and
> finally came down to having this test run on an NVME device.
> Unfortunately I cant seem to find any nvme HW at this time, we would
> need to order these drive and could take time.
>
> Could you please test the kernel using the test case mentioned above
> and reply back to this email with test case out put for both cases.
> iommu.passthrough=0 and iommu.passthrough=1. We should see better
> performance (iops may be) with IOMMU bypassed for DMA (ie
> iommu.passthrough=1).
>
> The bug is cced on this email, so if you replay with a summary output
> in both cases it will be automatically recorded in the bug. Once I
> get your test results I will be able to make a case for SRUing these
> patches.
>
> Thanks a ton
> Manoj Iyer