read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16 ... read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16 write: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16 ... write: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16 fio-2.1.3 Starting 16 processes read: (groupid=0, jobs=1): err= 0: pid=1066: Wed Feb 8 15:22:55 2017 read : io=1024.0MB, bw=2720.6KB/s, iops=680, runt=385423msec slat (usec): min=2, max=900544, avg=1353.42, stdev=4273.51 clat (usec): min=209, max=920431, avg=22165.08, stdev=11767.86 lat (usec): min=221, max=929766, avg=23519.63, stdev=12331.02 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 27], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 40], | 99.00th=[ 48], 99.50th=[ 52], 99.90th=[ 60], 99.95th=[ 69], | 99.99th=[ 149] bw (KB /s): min= 901, max= 3562, per=12.60%, avg=2726.09, stdev=229.41 lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.04%, 4=0.53%, 10=8.71%, 20=35.33%, 50=54.71% lat (msec) : 100=0.64%, 250=0.01%, 1000=0.01% cpu : usr=0.35%, sys=1.00%, ctx=461657, majf=0, minf=44 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1067: Wed Feb 8 15:22:55 2017 read : io=1024.0MB, bw=2720.6KB/s, iops=680, runt=385430msec slat (usec): min=2, max=896831, avg=1352.71, stdev=4270.98 clat (usec): min=206, max=905528, avg=22167.59, stdev=11808.84 lat (usec): min=224, max=905535, avg=23521.39, stdev=12376.40 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 27], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 40], | 99.00th=[ 48], 99.50th=[ 52], 99.90th=[ 61], 99.95th=[ 70], | 99.99th=[ 196] bw (KB /s): min= 274, max= 3514, per=12.60%, avg=2725.34, stdev=230.29 lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.04%, 4=0.54%, 10=8.86%, 20=35.06%, 50=54.89% lat (msec) : 100=0.58%, 250=0.02%, 1000=0.01% cpu : usr=0.36%, sys=0.98%, ctx=461399, majf=0, minf=45 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1068: Wed Feb 8 15:22:55 2017 read : io=1024.0MB, bw=2719.8KB/s, iops=679, runt=385549msec slat (usec): min=2, max=201008, avg=1354.87, stdev=3916.61 clat (usec): min=240, max=908162, avg=22173.21, stdev=12075.76 lat (usec): min=249, max=927359, avg=23529.17, stdev=12508.09 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 27], 80.00th=[ 30], 90.00th=[ 36], 95.00th=[ 40], | 99.00th=[ 48], 99.50th=[ 52], 99.90th=[ 61], 99.95th=[ 69], | 99.99th=[ 208] bw (KB /s): min= 830, max= 3768, per=12.59%, avg=2724.54, stdev=231.88 lat (usec) : 250=0.01%, 500=0.06%, 750=0.01%, 1000=0.02% lat (msec) : 2=0.05%, 4=0.61%, 10=8.75%, 20=34.94%, 50=54.90% lat (msec) : 100=0.64%, 250=0.03%, 1000=0.01% cpu : usr=0.34%, sys=1.01%, ctx=462638, majf=0, minf=44 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1069: Wed Feb 8 15:22:55 2017 read : io=1024.0MB, bw=2704.2KB/s, iops=676, runt=387770msec slat (usec): min=2, max=900682, avg=1361.90, stdev=4289.65 clat (usec): min=223, max=928262, avg=22301.70, stdev=11860.32 lat (usec): min=329, max=928293, avg=23664.71, stdev=12440.34 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 27], 80.00th=[ 31], 90.00th=[ 36], 95.00th=[ 40], | 99.00th=[ 48], 99.50th=[ 52], 99.90th=[ 61], 99.95th=[ 68], | 99.99th=[ 139] bw (KB /s): min= 890, max= 5083, per=12.52%, avg=2707.40, stdev=265.49 lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.05%, 4=0.56%, 10=8.87%, 20=34.64%, 50=55.17% lat (msec) : 100=0.67%, 250=0.01%, 1000=0.01% cpu : usr=0.34%, sys=1.01%, ctx=463668, majf=0, minf=44 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1070: Wed Feb 8 15:22:55 2017 read : io=1024.0MB, bw=2721.3KB/s, iops=680, runt=385324msec slat (usec): min=2, max=895935, avg=1360.06, stdev=4263.76 clat (usec): min=220, max=930291, avg=22154.23, stdev=11771.28 lat (usec): min=226, max=930299, avg=23515.42, stdev=12337.00 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 27], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 40], | 99.00th=[ 48], 99.50th=[ 51], 99.90th=[ 62], 99.95th=[ 69], | 99.99th=[ 167] bw (KB /s): min= 868, max= 3617, per=12.60%, avg=2726.18, stdev=239.92 lat (usec) : 250=0.01%, 500=0.04%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.07%, 4=0.64%, 10=8.65%, 20=35.05%, 50=54.93% lat (msec) : 100=0.59%, 250=0.01%, 1000=0.01% cpu : usr=0.31%, sys=1.04%, ctx=462986, majf=0, minf=44 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1071: Wed Feb 8 15:22:55 2017 read : io=1024.0MB, bw=2737.2KB/s, iops=684, runt=383095msec slat (usec): min=2, max=903780, avg=1344.43, stdev=4248.51 clat (usec): min=240, max=917436, avg=22033.45, stdev=11718.56 lat (usec): min=277, max=923379, avg=23378.97, stdev=12286.45 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 27], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 39], | 99.00th=[ 48], 99.50th=[ 51], 99.90th=[ 60], 99.95th=[ 64], | 99.99th=[ 229] bw (KB /s): min= 972, max= 3360, per=12.68%, avg=2743.38, stdev=219.60 lat (usec) : 250=0.01%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.04%, 4=0.55%, 10=8.68%, 20=35.51%, 50=54.60% lat (msec) : 100=0.55%, 250=0.01%, 1000=0.01% cpu : usr=0.39%, sys=0.95%, ctx=459149, majf=0, minf=45 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1072: Wed Feb 8 15:22:55 2017 read : io=1024.0MB, bw=2724.5KB/s, iops=681, runt=384876msec slat (usec): min=2, max=902641, avg=1353.46, stdev=4307.59 clat (usec): min=235, max=906246, avg=22131.77, stdev=11945.60 lat (usec): min=242, max=930639, avg=23486.32, stdev=12505.83 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 27], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 40], | 99.00th=[ 48], 99.50th=[ 52], 99.90th=[ 61], 99.95th=[ 67], | 99.99th=[ 277] bw (KB /s): min= 899, max= 3425, per=12.62%, avg=2730.54, stdev=231.75 lat (usec) : 250=0.01%, 500=0.04%, 750=0.05%, 1000=0.01% lat (msec) : 2=0.04%, 4=0.56%, 10=8.80%, 20=35.01%, 50=54.84% lat (msec) : 100=0.63%, 250=0.01%, 500=0.01%, 1000=0.01% cpu : usr=0.36%, sys=0.98%, ctx=461480, majf=0, minf=44 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1073: Wed Feb 8 15:22:55 2017 read : io=1024.0MB, bw=2739.2KB/s, iops=684, runt=382816msec slat (usec): min=2, max=904742, avg=1345.38, stdev=4232.88 clat (usec): min=212, max=920239, avg=22015.90, stdev=11609.79 lat (usec): min=267, max=925612, avg=23362.38, stdev=12186.48 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 27], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 39], | 99.00th=[ 47], 99.50th=[ 51], 99.90th=[ 61], 99.95th=[ 67], | 99.99th=[ 111] bw (KB /s): min= 903, max= 3866, per=12.69%, avg=2744.98, stdev=224.24 lat (usec) : 250=0.01%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.04%, 4=0.54%, 10=8.84%, 20=35.47%, 50=54.52% lat (msec) : 100=0.56%, 250=0.01%, 1000=0.01% cpu : usr=0.33%, sys=1.02%, ctx=459349, majf=0, minf=44 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1074: Wed Feb 8 15:22:55 2017 write: io=1024.0MB, bw=2605.5KB/s, iops=651, runt=402462msec slat (usec): min=2, max=902126, avg=1407.73, stdev=4506.84 clat (usec): min=108, max=926499, avg=23152.13, stdev=12781.97 lat (usec): min=122, max=926517, avg=24560.94, stdev=13355.70 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 7], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 29], 80.00th=[ 33], 90.00th=[ 38], 95.00th=[ 43], | 99.00th=[ 52], 99.50th=[ 56], 99.90th=[ 65], 99.95th=[ 73], | 99.99th=[ 105] bw (KB /s): min= 906, max= 9832, per=12.46%, avg=2596.01, stdev=675.66 lat (usec) : 250=0.03%, 500=0.02%, 750=0.03%, 1000=0.04% lat (msec) : 2=0.18%, 4=0.85%, 10=10.20%, 20=29.94%, 50=57.42% lat (msec) : 100=1.29%, 250=0.01%, 1000=0.01% cpu : usr=0.33%, sys=1.08%, ctx=484757, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1075: Wed Feb 8 15:22:55 2017 write: io=1024.0MB, bw=2605.8KB/s, iops=651, runt=402407msec slat (usec): min=2, max=896153, avg=1405.35, stdev=4497.49 clat (usec): min=114, max=924056, avg=23150.55, stdev=12804.72 lat (usec): min=121, max=939503, avg=24557.00, stdev=13367.34 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 29], 80.00th=[ 33], 90.00th=[ 38], 95.00th=[ 43], | 99.00th=[ 52], 99.50th=[ 56], 99.90th=[ 64], 99.95th=[ 69], | 99.99th=[ 130] bw (KB /s): min= 601, max= 9240, per=12.47%, avg=2598.20, stdev=674.51 lat (usec) : 250=0.02%, 500=0.02%, 750=0.03%, 1000=0.03% lat (msec) : 2=0.14%, 4=0.75%, 10=10.51%, 20=29.96%, 50=57.17% lat (msec) : 100=1.34%, 250=0.01%, 1000=0.01% cpu : usr=0.34%, sys=1.06%, ctx=483933, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1076: Wed Feb 8 15:22:55 2017 write: io=1024.0MB, bw=2605.4KB/s, iops=651, runt=402467msec slat (usec): min=2, max=891731, avg=1403.88, stdev=4503.99 clat (usec): min=108, max=929763, avg=23156.26, stdev=12920.47 lat (usec): min=119, max=929772, avg=24561.24, stdev=13510.59 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 7], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 29], 80.00th=[ 33], 90.00th=[ 38], 95.00th=[ 43], | 99.00th=[ 52], 99.50th=[ 56], 99.90th=[ 65], 99.95th=[ 69], | 99.99th=[ 155] bw (KB /s): min= 628, max=10056, per=12.45%, avg=2595.85, stdev=698.11 lat (usec) : 250=0.04%, 500=0.02%, 750=0.02%, 1000=0.05% lat (msec) : 2=0.19%, 4=0.83%, 10=10.64%, 20=29.62%, 50=57.24% lat (msec) : 100=1.34%, 250=0.01%, 1000=0.01% cpu : usr=0.34%, sys=1.07%, ctx=482744, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1077: Wed Feb 8 15:22:55 2017 write: io=1024.0MB, bw=2611.2KB/s, iops=652, runt=401571msec slat (usec): min=2, max=899764, avg=1400.97, stdev=4474.68 clat (usec): min=154, max=924279, avg=23103.87, stdev=12630.49 lat (usec): min=311, max=936557, avg=24505.90, stdev=13187.01 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 25], | 70.00th=[ 29], 80.00th=[ 32], 90.00th=[ 38], 95.00th=[ 42], | 99.00th=[ 52], 99.50th=[ 56], 99.90th=[ 66], 99.95th=[ 70], | 99.99th=[ 111] bw (KB /s): min= 213, max= 6544, per=12.55%, avg=2615.50, stdev=638.40 lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02% lat (msec) : 2=0.06%, 4=0.51%, 10=10.17%, 20=30.86%, 50=57.05% lat (msec) : 100=1.28%, 250=0.01%, 1000=0.01% cpu : usr=0.32%, sys=1.07%, ctx=473805, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1078: Wed Feb 8 15:22:55 2017 write: io=1024.0MB, bw=2607.8KB/s, iops=651, runt=402106msec slat (usec): min=2, max=898708, avg=1404.70, stdev=4472.07 clat (usec): min=116, max=919498, avg=23132.87, stdev=12642.65 lat (usec): min=184, max=931470, avg=24538.66, stdev=13185.60 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 29], 80.00th=[ 32], 90.00th=[ 38], 95.00th=[ 42], | 99.00th=[ 51], 99.50th=[ 55], 99.90th=[ 63], 99.95th=[ 68], | 99.99th=[ 124] bw (KB /s): min= 48, max= 9384, per=12.52%, avg=2609.57, stdev=686.23 lat (usec) : 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.03% lat (msec) : 2=0.11%, 4=0.63%, 10=10.42%, 20=30.27%, 50=57.29% lat (msec) : 100=1.21%, 250=0.01%, 1000=0.01% cpu : usr=0.33%, sys=1.06%, ctx=481175, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1079: Wed Feb 8 15:22:55 2017 write: io=1024.0MB, bw=2610.5KB/s, iops=652, runt=401682msec slat (usec): min=2, max=910781, avg=1403.26, stdev=4516.65 clat (usec): min=143, max=931800, avg=23108.48, stdev=12769.19 lat (usec): min=154, max=931810, avg=24512.79, stdev=13298.32 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 25], | 70.00th=[ 29], 80.00th=[ 32], 90.00th=[ 38], 95.00th=[ 42], | 99.00th=[ 52], 99.50th=[ 56], 99.90th=[ 66], 99.95th=[ 75], | 99.99th=[ 176] bw (KB /s): min= 687, max= 6800, per=12.54%, avg=2614.03, stdev=640.72 lat (usec) : 250=0.01%, 500=0.01%, 750=0.03%, 1000=0.02% lat (msec) : 2=0.08%, 4=0.58%, 10=10.18%, 20=30.93%, 50=56.91% lat (msec) : 100=1.23%, 250=0.03%, 1000=0.01% cpu : usr=0.32%, sys=1.07%, ctx=475304, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1080: Wed Feb 8 15:22:55 2017 write: io=1024.0MB, bw=2606.5KB/s, iops=651, runt=402296msec slat (usec): min=2, max=892261, avg=1408.39, stdev=4502.88 clat (usec): min=105, max=908359, avg=23139.55, stdev=12733.13 lat (usec): min=132, max=913218, avg=24549.05, stdev=13266.68 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 29], 80.00th=[ 32], 90.00th=[ 38], 95.00th=[ 42], | 99.00th=[ 52], 99.50th=[ 56], 99.90th=[ 67], 99.95th=[ 73], | 99.99th=[ 169] bw (KB /s): min= 873, max= 9088, per=12.50%, avg=2604.49, stdev=667.10 lat (usec) : 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.03% lat (msec) : 2=0.14%, 4=0.62%, 10=10.39%, 20=30.28%, 50=57.17% lat (msec) : 100=1.31%, 250=0.01%, 1000=0.01% cpu : usr=0.36%, sys=1.04%, ctx=483440, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1081: Wed Feb 8 15:22:55 2017 write: io=1024.0MB, bw=2606.7KB/s, iops=651, runt=402273msec slat (usec): min=2, max=893620, avg=1402.17, stdev=4491.53 clat (usec): min=145, max=913834, avg=23142.88, stdev=12697.94 lat (usec): min=302, max=915530, avg=24546.14, stdev=13267.45 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 29], 80.00th=[ 32], 90.00th=[ 38], 95.00th=[ 42], | 99.00th=[ 51], 99.50th=[ 56], 99.90th=[ 65], 99.95th=[ 71], | 99.99th=[ 141] bw (KB /s): min= 548, max= 9512, per=12.50%, avg=2605.79, stdev=681.49 lat (usec) : 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.03% lat (msec) : 2=0.12%, 4=0.54%, 10=10.53%, 20=29.91%, 50=57.61% lat (msec) : 100=1.20%, 250=0.02%, 1000=0.01% cpu : usr=0.31%, sys=1.09%, ctx=482511, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=8192.0MB, aggrb=21632KB/s, minb=2704KB/s, maxb=2739KB/s, mint=382816msec, maxt=387770msec WRITE: io=8192.0MB, aggrb=20842KB/s, minb=2605KB/s, maxb=2611KB/s, mint=401571msec, maxt=402467msec Disk stats (read/write): vdb: ios=2097170/2094596, merge=0/87, ticks=12345264/12609152, in_queue=24953040, util=100.00%