It’s working as expected.  I see improvement with passthrough enabled (see below).  Nate also confirmed the SMMU setup via JTAG.

Passthrough=0 -

root@null-8cfdf006971f:/home/ubuntu# fio --name=global --readonly --group_reporting --direct=1 --ioengine=libaio --rw=read --eta-newline=1s  --size=1T --blocksize=512k --iodepth=32 --numjobs=1  --runtime=10s --name=nvme_0 --filename=/dev/nvme0n1
nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio, iodepth=32
fio-2.2.10
Starting 1 process
Jobs: 1 (f=1): [R(1)] [36.4% done] [2540MB/0KB/0KB /s] [5080/0/0 iops] [eta 00m:07s]
Jobs: 1 (f=1): [R(1)] [45.5% done] [2539MB/0KB/0KB /s] [5077/0/0 iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [2540MB/0KB/0KB /s] [5079/0/0 iops] [eta 00m:05s]
Jobs: 1 (f=1): [R(1)] [63.6% done] [2540MB/0KB/0KB /s] [5079/0/0 iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [2539MB/0KB/0KB /s] [5078/0/0 iops] [eta 00m:03s]
Jobs: 1 (f=1): [R(1)] [81.8% done] [2539MB/0KB/0KB /s] [5078/0/0 iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [2540MB/0KB/0KB /s] [5080/0/0 iops] [eta 00m:01s]
Jobs: 1 (f=1): [R(1)] [100.0% done] [2539MB/0KB/0KB /s] [5078/0/0 iops] [eta 00m:00s]
nvme_0: (groupid=0, jobs=1): err= 0: pid=36481: Mon May 15 17:10:26 2017
  read : io=25377MB, bw=2537.5MB/s, iops=5074, runt= 10001msec
    slat (usec): min=32, max=440, avg=191.74, stdev=157.14
    clat (usec): min=153, max=7641, avg=6106.54, stdev=107.40
     lat (usec): min=442, max=7904, avg=6298.45, stdev=174.49
    clat percentiles (usec):
     |  1.00th=[ 5920],  5.00th=[ 5984], 10.00th=[ 6048], 20.00th=[ 6048],
     | 30.00th=[ 6112], 40.00th=[ 6112], 50.00th=[ 6112], 60.00th=[ 6112],
     | 70.00th=[ 6112], 80.00th=[ 6112], 90.00th=[ 6176], 95.00th=[ 6176],
     | 99.00th=[ 6304], 99.50th=[ 6368], 99.90th=[ 6432], 99.95th=[ 6432],
     | 99.99th=[ 7008]
    bw (MB  /s): min= 2498, max= 2540, per=99.98%, avg=2536.84, stdev= 9.44
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%
    lat (msec) : 2=0.01%, 4=0.02%, 10=99.96%
  cpu          : usr=3.84%, sys=96.12%, ctx=67, majf=0, minf=528
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=50754/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=25377MB, aggrb=2537.5MB/s, minb=2537.5MB/s, maxb=2537.5MB/s, mint=10001msec, maxt=10001msec

Disk stats (read/write):
  nvme0n1: ios=200550/0, merge=0/0, ticks=23848/0, in_queue=23828, util=91.79%



passthrough=1 –

root@null-8cfdf006971f:/home/ubuntu#
ing --direct=1 --ioengine=libaio --rw=read --eta-newline=1s  --size=1T --blocksize=512k --iodepth=32 --numjobs=1  --runtime=10s --name=nvme_0 --filename=/dev/nvme0n1
nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio, iodepth=32
fio-2.2.10
Starting 1 process
Jobs: 1 (f=1): [R(1)] [36.4% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta 00m:07s]
Jobs: 1 (f=1): [R(1)] [45.5% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [3239MB/0KB/0KB /s] [6477/0/0 iops] [eta 00m:05s]
Jobs: 1 (f=1): [R(1)] [63.6% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta 00m:03s]
Jobs: 1 (f=1): [R(1)] [81.8% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [3239MB/0KB/0KB /s] [6477/0/0 iops] [eta 00m:01s]
Jobs: 1 (f=1): [R(1)] [100.0% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta 00m:00s]
nvme_0: (groupid=0, jobs=1): err= 0: pid=1600: Mon May 15 17:13:46 2017
  read : io=32384MB, bw=3236.8MB/s, iops=6473, runt= 10005msec
    slat (usec): min=13, max=210, avg=16.19, stdev= 6.40
    clat (usec): min=1505, max=9532, avg=4925.01, stdev=87.39
     lat (usec): min=1616, max=9566, avg=4941.38, stdev=86.47
    clat percentiles (usec):
     |  1.00th=[ 4896],  5.00th=[ 4896], 10.00th=[ 4896], 20.00th=[ 4896],
     | 30.00th=[ 4896], 40.00th=[ 4960], 50.00th=[ 4960], 60.00th=[ 4960],
     | 70.00th=[ 4960], 80.00th=[ 4960], 90.00th=[ 4960], 95.00th=[ 4960],
     | 99.00th=[ 4960], 99.50th=[ 4960], 99.90th=[ 4960], 99.95th=[ 4960],
     | 99.99th=[ 8640]
    bw (MB  /s): min= 3209, max= 3237, per=99.96%, avg=3235.57, stdev= 6.18
    lat (msec) : 2=0.03%, 4=0.03%, 10=99.94%
  cpu          : usr=1.80%, sys=16.99%, ctx=64755, majf=0, minf=528
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=64767/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=32384MB, aggrb=3236.8MB/s, minb=3236.8MB/s, maxb=3236.8MB/s, mint=10005msec, maxt=10005msec

Disk stats (read/write):
  nvme0n1: ios=255788/0, merge=0/0, ticks=1246388/0, in_queue=1246788, util=99.05%

Jeffrey Hugo
Senior Engineer
Qualcomm Datacenter Technologies, Inc.
1-303-247-5002

From: Manoj Iyer [mailto:<email address hidden>]
Sent: Thursday, May 11, 2017 9:23 PM
To: Jeff Hugo <email address hidden>; <email address hidden>
Cc: Nate Watterson <email address hidden>; Timur Tabi <email address hidden>
Subject: [Bug #1688158] Please help test kernel for iommu.passthrough=0/1

Jeff,

I have back-ported patches to address "Support SMMU passthrough using the default domain". This adds a new kernel command line option iommu.passthrough=0/1 to Configure DMA to bypass the IOMMU for ARM64. http://bugs.launchpad.net/bugs/1688158

0 - Use IOMMU translation for DMA.
1 - Bypass the IOMMU for DMA.

The test case recommended by Nate using 'fio' is:

$ sudo apt update ; sudo apt install fio

$ sudo fio --name=global --readonly --group_reporting --direct=1 --ioengine=libaio --rw=read --eta-newline=1s  --size=1T --blocksize=512k --iodepth=32 --numjobs=1  --runtime=10s --name=nvme_0 --filename=/dev/nvme0n1

You can find the kernel and the instructions to install the kernel in the Centriq test PPA:
https://launchpad.net/~centriq-team/+archive/ubuntu/test/

Nate and I have gone back and forth on testing these patches and finally came down to having this test run on an NVME device. Unfortunately I cant seem to find any nvme  HW at this time, we would need to order these drive and could take time.

Could you please test the kernel using the test case mentioned above and reply back to this email with test case out put for both cases. iommu.passthrough=0 and iommu.passthrough=1. We should see better performance (iops may be) with IOMMU bypassed for DMA (ie iommu.passthrough=1).

The bug is cced on this email, so if you replay with a summary output in both cases it will be automatically recorded in the bug. Once I get your test results I will be able to make a case for SRUing these patches.

Thanks a ton
Manoj Iyer