Thanks Jeff/Nate I will sru these patches shortly.
On Mon, May 15, 2017 at 2:55 PM, Jeff Hugo <email address hidden> wrote: > It’s working as expected. I see improvement with passthrough > enabled (see below). Nate also confirmed the SMMU setup via JTAG. > > Passthrough=0 - > > root@null-8cfdf006971f:/home/ubuntu# fio --name=global --readonly > --group_reporting --direct=1 --ioengine=libaio --rw=read > --eta-newline=1s --size=1T --blocksize=512k --iodepth=32 --numjobs=1 > --runtime=10s --name=nvme_0 --filename=/dev/nvme0n1 > nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, > ioengine=libaio, iodepth=32 > fio-2.2.10 > Starting 1 process > Jobs: 1 (f=1): [R(1)] [36.4% done] [2540MB/0KB/0KB /s] [5080/0/0 > iops] [eta 00m:07s] > Jobs: 1 (f=1): [R(1)] [45.5% done] [2539MB/0KB/0KB /s] [5077/0/0 > iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [2540MB/0KB/0KB /s] > [5079/0/0 iops] [eta 00m:05s] > Jobs: 1 (f=1): [R(1)] [63.6% done] [2540MB/0KB/0KB /s] [5079/0/0 > iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [2539MB/0KB/0KB /s] > [5078/0/0 iops] [eta 00m:03s] > Jobs: 1 (f=1): [R(1)] [81.8% done] [2539MB/0KB/0KB /s] [5078/0/0 > iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [2540MB/0KB/0KB /s] > [5080/0/0 iops] [eta 00m:01s] > Jobs: 1 (f=1): [R(1)] [100.0% done] [2539MB/0KB/0KB /s] [5078/0/0 > iops] [eta 00m:00s] > nvme_0: (groupid=0, jobs=1): err= 0: pid=36481: Mon May 15 17:10:26 > 2017 > read : io=25377MB, bw=2537.5MB/s, iops=5074, runt= 10001msec > slat (usec): min=32, max=440, avg=191.74, stdev=157.14 > clat (usec): min=153, max=7641, avg=6106.54, stdev=107.40 > lat (usec): min=442, max=7904, avg=6298.45, stdev=174.49 > clat percentiles (usec): > | 1.00th=[ 5920], 5.00th=[ 5984], 10.00th=[ 6048], 20.00th=[ > 6048], > | 30.00th=[ 6112], 40.00th=[ 6112], 50.00th=[ 6112], 60.00th=[ > 6112], > | 70.00th=[ 6112], 80.00th=[ 6112], 90.00th=[ 6176], 95.00th=[ > 6176], > | 99.00th=[ 6304], 99.50th=[ 6368], 99.90th=[ 6432], 99.95th=[ > 6432], > | 99.99th=[ 7008] > bw (MB /s): min= 2498, max= 2540, per=99.98%, avg=2536.84, > stdev= 9.44 > lat (usec) : 250=0.01%, 500=0.01%, 750=0.01% > lat (msec) : 2=0.01%, 4=0.02%, 10=99.96% > cpu : usr=3.84%, sys=96.12%, ctx=67, majf=0, minf=528 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, > >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, > >=64=0.0% > issued : total=r=50754/w=0/d=0, short=r=0/w=0/d=0, > drop=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=32 > > Run status group 0 (all jobs): > READ: io=25377MB, aggrb=2537.5MB/s, minb=2537.5MB/s, > maxb=2537.5MB/s, mint=10001msec, maxt=10001msec > > Disk stats (read/write): > nvme0n1: ios=200550/0, merge=0/0, ticks=23848/0, in_queue=23828, > util=91.79% > > > > passthrough=1 – > > root@null-8cfdf006971f:/home/ubuntu# > ing --direct=1 --ioengine=libaio --rw=read --eta-newline=1s > --size=1T --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s > --name=nvme_0 --filename=/dev/nvme0n1 > nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, > ioengine=libaio, iodepth=32 > fio-2.2.10 > Starting 1 process > Jobs: 1 (f=1): [R(1)] [36.4% done] [3238MB/0KB/0KB /s] [6476/0/0 > iops] [eta 00m:07s] > Jobs: 1 (f=1): [R(1)] [45.5% done] [3238MB/0KB/0KB /s] [6476/0/0 > iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [3239MB/0KB/0KB /s] > [6477/0/0 iops] [eta 00m:05s] > Jobs: 1 (f=1): [R(1)] [63.6% done] [3238MB/0KB/0KB /s] [6476/0/0 > iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [3238MB/0KB/0KB /s] > [6476/0/0 iops] [eta 00m:03s] > Jobs: 1 (f=1): [R(1)] [81.8% done] [3238MB/0KB/0KB /s] [6476/0/0 > iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [3239MB/0KB/0KB /s] > [6477/0/0 iops] [eta 00m:01s] > Jobs: 1 (f=1): [R(1)] [100.0% done] [3238MB/0KB/0KB /s] [6476/0/0 > iops] [eta 00m:00s] > nvme_0: (groupid=0, jobs=1): err= 0: pid=1600: Mon May 15 17:13:46 > 2017 > read : io=32384MB, bw=3236.8MB/s, iops=6473, runt= 10005msec > slat (usec): min=13, max=210, avg=16.19, stdev= 6.40 > clat (usec): min=1505, max=9532, avg=4925.01, stdev=87.39 > lat (usec): min=1616, max=9566, avg=4941.38, stdev=86.47 > clat percentiles (usec): > | 1.00th=[ 4896], 5.00th=[ 4896], 10.00th=[ 4896], 20.00th=[ > 4896], > | 30.00th=[ 4896], 40.00th=[ 4960], 50.00th=[ 4960], 60.00th=[ > 4960], > | 70.00th=[ 4960], 80.00th=[ 4960], 90.00th=[ 4960], 95.00th=[ > 4960], > | 99.00th=[ 4960], 99.50th=[ 4960], 99.90th=[ 4960], 99.95th=[ > 4960], > | 99.99th=[ 8640] > bw (MB /s): min= 3209, max= 3237, per=99.96%, avg=3235.57, > stdev= 6.18 > lat (msec) : 2=0.03%, 4=0.03%, 10=99.94% > cpu : usr=1.80%, sys=16.99%, ctx=64755, majf=0, minf=528 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, > >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, > >=64=0.0% > issued : total=r=64767/w=0/d=0, short=r=0/w=0/d=0, > drop=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=32 > > Run status group 0 (all jobs): > READ: io=32384MB, aggrb=3236.8MB/s, minb=3236.8MB/s, > maxb=3236.8MB/s, mint=10005msec, maxt=10005msec > > Disk stats (read/write): > nvme0n1: ios=255788/0, merge=0/0, ticks=1246388/0, > in_queue=1246788, util=99.05% > > Jeffrey Hugo > Senior Engineer > Qualcomm Datacenter Technologies, Inc. > 1-303-247-5002 > > From: Manoj Iyer [mailto:<email address hidden>] > Sent: Thursday, May 11, 2017 9:23 PM > To: Jeff Hugo <email address hidden>; <email address hidden> > Cc: Nate Watterson <email address hidden>; Timur Tabi > <email address hidden> > Subject: [Bug #1688158] Please help test kernel for > iommu.passthrough=0/1 > > Jeff, > > I have back-ported patches to address "Support SMMU passthrough using > the default domain". This adds a new kernel command line option > iommu.passthrough=0/1 to Configure DMA to bypass the IOMMU for ARM64. > http://bugs.launchpad.net/bugs/1688158 > > 0 - Use IOMMU translation for DMA. > 1 - Bypass the IOMMU for DMA. > > The test case recommended by Nate using 'fio' is: > > $ sudo apt update ; sudo apt install fio > > $ sudo fio --name=global --readonly --group_reporting --direct=1 > --ioengine=libaio --rw=read --eta-newline=1s --size=1T > --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s > --name=nvme_0 --filename=/dev/nvme0n1 > > You can find the kernel and the instructions to install the kernel in > the Centriq test PPA: > https://launchpad.net/~centriq-team/+archive/ubuntu/test/ > > Nate and I have gone back and forth on testing these patches and > finally came down to having this test run on an NVME device. > Unfortunately I cant seem to find any nvme HW at this time, we would > need to order these drive and could take time. > > Could you please test the kernel using the test case mentioned above > and reply back to this email with test case out put for both cases. > iommu.passthrough=0 and iommu.passthrough=1. We should see better > performance (iops may be) with IOMMU bypassed for DMA (ie > iommu.passthrough=1). > > The bug is cced on this email, so if you replay with a summary output > in both cases it will be automatically recorded in the bug. Once I > get your test results I will be able to make a case for SRUing these > patches. > > Thanks a ton > Manoj Iyer
Thanks Jeff/Nate I will sru these patches shortly.
On Mon, May 15, 2017 at 2:55 PM, Jeff Hugo <email address hidden> 8cfdf006971f: /home/ubuntu# fio --name=global --readonly /dev/nvme0n1 512K/512K- 512K/512K- 512K, 50754/w= 0/d=0, short=r=0/w=0/d=0, 8cfdf006971f: /home/ubuntu# /dev/nvme0n1 512K/512K- 512K/512K- 512K, 64767/w= 0/d=0, short=r=0/w=0/d=0, gh=0/1 gh=0/1 to Configure DMA to bypass the IOMMU for ARM64. bugs.launchpad. net/bugs/ 1688158 /dev/nvme0n1 /launchpad. net/~centriq- team/+archive/ ubuntu/ test/ gh=1. We should see better gh=1).
wrote:
> It’s working as expected. I see improvement with passthrough
> enabled (see below). Nate also confirmed the SMMU setup via JTAG.
>
> Passthrough=0 -
>
> root@null-
> --group_reporting --direct=1 --ioengine=libaio --rw=read
> --eta-newline=1s --size=1T --blocksize=512k --iodepth=32 --numjobs=1
> --runtime=10s --name=nvme_0 --filename=
> nvme_0: (g=0): rw=read, bs=512K-
> ioengine=libaio, iodepth=32
> fio-2.2.10
> Starting 1 process
> Jobs: 1 (f=1): [R(1)] [36.4% done] [2540MB/0KB/0KB /s] [5080/0/0
> iops] [eta 00m:07s]
> Jobs: 1 (f=1): [R(1)] [45.5% done] [2539MB/0KB/0KB /s] [5077/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [2540MB/0KB/0KB /s]
> [5079/0/0 iops] [eta 00m:05s]
> Jobs: 1 (f=1): [R(1)] [63.6% done] [2540MB/0KB/0KB /s] [5079/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [2539MB/0KB/0KB /s]
> [5078/0/0 iops] [eta 00m:03s]
> Jobs: 1 (f=1): [R(1)] [81.8% done] [2539MB/0KB/0KB /s] [5078/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [2540MB/0KB/0KB /s]
> [5080/0/0 iops] [eta 00m:01s]
> Jobs: 1 (f=1): [R(1)] [100.0% done] [2539MB/0KB/0KB /s] [5078/0/0
> iops] [eta 00m:00s]
> nvme_0: (groupid=0, jobs=1): err= 0: pid=36481: Mon May 15 17:10:26
> 2017
> read : io=25377MB, bw=2537.5MB/s, iops=5074, runt= 10001msec
> slat (usec): min=32, max=440, avg=191.74, stdev=157.14
> clat (usec): min=153, max=7641, avg=6106.54, stdev=107.40
> lat (usec): min=442, max=7904, avg=6298.45, stdev=174.49
> clat percentiles (usec):
> | 1.00th=[ 5920], 5.00th=[ 5984], 10.00th=[ 6048], 20.00th=[
> 6048],
> | 30.00th=[ 6112], 40.00th=[ 6112], 50.00th=[ 6112], 60.00th=[
> 6112],
> | 70.00th=[ 6112], 80.00th=[ 6112], 90.00th=[ 6176], 95.00th=[
> 6176],
> | 99.00th=[ 6304], 99.50th=[ 6368], 99.90th=[ 6432], 99.95th=[
> 6432],
> | 99.99th=[ 7008]
> bw (MB /s): min= 2498, max= 2540, per=99.98%, avg=2536.84,
> stdev= 9.44
> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%
> lat (msec) : 2=0.01%, 4=0.02%, 10=99.96%
> cpu : usr=3.84%, sys=96.12%, ctx=67, majf=0, minf=528
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued : total=r=
> drop=r=0/w=0/d=0
> latency : target=0, window=0, percentile=100.00%, depth=32
>
> Run status group 0 (all jobs):
> READ: io=25377MB, aggrb=2537.5MB/s, minb=2537.5MB/s,
> maxb=2537.5MB/s, mint=10001msec, maxt=10001msec
>
> Disk stats (read/write):
> nvme0n1: ios=200550/0, merge=0/0, ticks=23848/0, in_queue=23828,
> util=91.79%
>
>
>
> passthrough=1 –
>
> root@null-
> ing --direct=1 --ioengine=libaio --rw=read --eta-newline=1s
> --size=1T --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s
> --name=nvme_0 --filename=
> nvme_0: (g=0): rw=read, bs=512K-
> ioengine=libaio, iodepth=32
> fio-2.2.10
> Starting 1 process
> Jobs: 1 (f=1): [R(1)] [36.4% done] [3238MB/0KB/0KB /s] [6476/0/0
> iops] [eta 00m:07s]
> Jobs: 1 (f=1): [R(1)] [45.5% done] [3238MB/0KB/0KB /s] [6476/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [3239MB/0KB/0KB /s]
> [6477/0/0 iops] [eta 00m:05s]
> Jobs: 1 (f=1): [R(1)] [63.6% done] [3238MB/0KB/0KB /s] [6476/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [3238MB/0KB/0KB /s]
> [6476/0/0 iops] [eta 00m:03s]
> Jobs: 1 (f=1): [R(1)] [81.8% done] [3238MB/0KB/0KB /s] [6476/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [3239MB/0KB/0KB /s]
> [6477/0/0 iops] [eta 00m:01s]
> Jobs: 1 (f=1): [R(1)] [100.0% done] [3238MB/0KB/0KB /s] [6476/0/0
> iops] [eta 00m:00s]
> nvme_0: (groupid=0, jobs=1): err= 0: pid=1600: Mon May 15 17:13:46
> 2017
> read : io=32384MB, bw=3236.8MB/s, iops=6473, runt= 10005msec
> slat (usec): min=13, max=210, avg=16.19, stdev= 6.40
> clat (usec): min=1505, max=9532, avg=4925.01, stdev=87.39
> lat (usec): min=1616, max=9566, avg=4941.38, stdev=86.47
> clat percentiles (usec):
> | 1.00th=[ 4896], 5.00th=[ 4896], 10.00th=[ 4896], 20.00th=[
> 4896],
> | 30.00th=[ 4896], 40.00th=[ 4960], 50.00th=[ 4960], 60.00th=[
> 4960],
> | 70.00th=[ 4960], 80.00th=[ 4960], 90.00th=[ 4960], 95.00th=[
> 4960],
> | 99.00th=[ 4960], 99.50th=[ 4960], 99.90th=[ 4960], 99.95th=[
> 4960],
> | 99.99th=[ 8640]
> bw (MB /s): min= 3209, max= 3237, per=99.96%, avg=3235.57,
> stdev= 6.18
> lat (msec) : 2=0.03%, 4=0.03%, 10=99.94%
> cpu : usr=1.80%, sys=16.99%, ctx=64755, majf=0, minf=528
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued : total=r=
> drop=r=0/w=0/d=0
> latency : target=0, window=0, percentile=100.00%, depth=32
>
> Run status group 0 (all jobs):
> READ: io=32384MB, aggrb=3236.8MB/s, minb=3236.8MB/s,
> maxb=3236.8MB/s, mint=10005msec, maxt=10005msec
>
> Disk stats (read/write):
> nvme0n1: ios=255788/0, merge=0/0, ticks=1246388/0,
> in_queue=1246788, util=99.05%
>
> Jeffrey Hugo
> Senior Engineer
> Qualcomm Datacenter Technologies, Inc.
> 1-303-247-5002
>
> From: Manoj Iyer [mailto:<email address hidden>]
> Sent: Thursday, May 11, 2017 9:23 PM
> To: Jeff Hugo <email address hidden>; <email address hidden>
> Cc: Nate Watterson <email address hidden>; Timur Tabi
> <email address hidden>
> Subject: [Bug #1688158] Please help test kernel for
> iommu.passthrou
>
> Jeff,
>
> I have back-ported patches to address "Support SMMU passthrough using
> the default domain". This adds a new kernel command line option
> iommu.passthrou
> http://
>
> 0 - Use IOMMU translation for DMA.
> 1 - Bypass the IOMMU for DMA.
>
> The test case recommended by Nate using 'fio' is:
>
> $ sudo apt update ; sudo apt install fio
>
> $ sudo fio --name=global --readonly --group_reporting --direct=1
> --ioengine=libaio --rw=read --eta-newline=1s --size=1T
> --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s
> --name=nvme_0 --filename=
>
> You can find the kernel and the instructions to install the kernel in
> the Centriq test PPA:
> https:/
>
> Nate and I have gone back and forth on testing these patches and
> finally came down to having this test run on an NVME device.
> Unfortunately I cant seem to find any nvme HW at this time, we would
> need to order these drive and could take time.
>
> Could you please test the kernel using the test case mentioned above
> and reply back to this email with test case out put for both cases.
> iommu.passthrough=0 and iommu.passthrou
> performance (iops may be) with IOMMU bypassed for DMA (ie
> iommu.passthrou
>
> The bug is cced on this email, so if you replay with a summary output
> in both cases it will be automatically recorded in the bug. Once I
> get your test results I will be able to make a case for SRUing these
> patches.
>
> Thanks a ton
> Manoj Iyer