read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16 ... read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16 write: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16 ... write: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16 fio-2.1.3 Starting 16 processes read: (groupid=0, jobs=1): err= 0: pid=1589: Wed Feb 8 15:03:52 2017 read : io=1024.0MB, bw=2769.3KB/s, iops=692, runt=378653msec slat (usec): min=2, max=832751, avg=1327.01, stdev=4221.42 clat (usec): min=171, max=853951, avg=21780.25, stdev=11726.95 lat (usec): min=192, max=867940, avg=23108.36, stdev=12275.70 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 39], | 99.00th=[ 48], 99.50th=[ 51], 99.90th=[ 65], 99.95th=[ 106], | 99.99th=[ 215] bw (KB /s): min= 907, max= 5384, per=12.52%, avg=2772.74, stdev=268.17 lat (usec) : 250=0.01%, 500=0.09%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.06%, 4=0.61%, 10=9.27%, 20=35.98%, 50=53.37% lat (msec) : 100=0.54%, 250=0.05%, 1000=0.01% cpu : usr=0.38%, sys=1.06%, ctx=456294, majf=0, minf=45 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1590: Wed Feb 8 15:03:52 2017 read : io=1024.0MB, bw=2776.1KB/s, iops=694, runt=377603msec slat (usec): min=2, max=825270, avg=1326.10, stdev=4194.52 clat (usec): min=207, max=872692, avg=21717.11, stdev=11587.36 lat (usec): min=240, max=872732, avg=23044.32, stdev=12131.37 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 39], | 99.00th=[ 48], 99.50th=[ 51], 99.90th=[ 62], 99.95th=[ 88], | 99.99th=[ 155] bw (KB /s): min= 623, max= 4984, per=12.56%, avg=2782.19, stdev=242.72 lat (usec) : 250=0.01%, 500=0.07%, 750=0.01%, 1000=0.02% lat (msec) : 2=0.06%, 4=0.65%, 10=9.38%, 20=36.09%, 50=53.16% lat (msec) : 100=0.52%, 250=0.04%, 1000=0.01% cpu : usr=0.37%, sys=1.06%, ctx=456254, majf=0, minf=46 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1591: Wed Feb 8 15:03:52 2017 read : io=1024.0MB, bw=2785.5KB/s, iops=696, runt=376445msec slat (usec): min=2, max=834984, avg=1321.51, stdev=4174.56 clat (usec): min=205, max=864703, avg=21651.03, stdev=11536.66 lat (usec): min=287, max=869506, avg=22973.62, stdev=12076.27 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 39], | 99.00th=[ 47], 99.50th=[ 50], 99.90th=[ 61], 99.95th=[ 70], | 99.99th=[ 243] bw (KB /s): min= 995, max= 5099, per=12.60%, avg=2791.15, stdev=223.80 lat (usec) : 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.05%, 4=0.67%, 10=9.11%, 20=36.47%, 50=53.16% lat (msec) : 100=0.48%, 250=0.02%, 500=0.01%, 1000=0.01% cpu : usr=0.36%, sys=1.07%, ctx=455608, majf=0, minf=45 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1592: Wed Feb 8 15:03:52 2017 read : io=1024.0MB, bw=2787.8KB/s, iops=696, runt=376139msec slat (usec): min=2, max=832208, avg=1321.94, stdev=4148.75 clat (usec): min=239, max=864052, avg=21631.98, stdev=11375.49 lat (usec): min=250, max=874931, avg=22954.99, stdev=11909.79 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 39], | 99.00th=[ 47], 99.50th=[ 51], 99.90th=[ 61], 99.95th=[ 68], | 99.99th=[ 169] bw (KB /s): min= 851, max= 3626, per=12.61%, avg=2793.39, stdev=197.40 lat (usec) : 250=0.01%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.07%, 4=0.61%, 10=9.18%, 20=36.75%, 50=52.80% lat (msec) : 100=0.53%, 250=0.02%, 1000=0.01% cpu : usr=0.34%, sys=1.09%, ctx=455418, majf=0, minf=45 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1593: Wed Feb 8 15:03:52 2017 read : io=1024.0MB, bw=2784.6KB/s, iops=696, runt=376573msec slat (usec): min=2, max=830095, avg=1319.41, stdev=4171.08 clat (usec): min=278, max=840026, avg=21659.91, stdev=11464.48 lat (usec): min=285, max=866329, avg=22980.33, stdev=12006.55 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 39], | 99.00th=[ 47], 99.50th=[ 50], 99.90th=[ 59], 99.95th=[ 66], | 99.99th=[ 245] bw (KB /s): min= 830, max= 4294, per=12.59%, avg=2789.64, stdev=221.78 lat (usec) : 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.05%, 4=0.65%, 10=9.42%, 20=36.12%, 50=53.28% lat (msec) : 100=0.41%, 250=0.03%, 500=0.01%, 1000=0.01% cpu : usr=0.36%, sys=1.07%, ctx=454792, majf=0, minf=45 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1594: Wed Feb 8 15:03:52 2017 read : io=1024.0MB, bw=2806.3KB/s, iops=701, runt=373661msec slat (usec): min=2, max=840930, avg=1309.63, stdev=4138.14 clat (usec): min=344, max=857705, avg=21492.12, stdev=11388.56 lat (usec): min=361, max=880798, avg=22802.81, stdev=11940.23 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 26], 80.00th=[ 30], 90.00th=[ 34], 95.00th=[ 38], | 99.00th=[ 46], 99.50th=[ 50], 99.90th=[ 59], 99.95th=[ 64], | 99.99th=[ 243] bw (KB /s): min= 824, max= 3361, per=12.69%, avg=2812.22, stdev=194.05 lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.04%, 4=0.57%, 10=9.33%, 20=37.28%, 50=52.29% lat (msec) : 100=0.43%, 250=0.02%, 500=0.01%, 1000=0.01% cpu : usr=0.34%, sys=1.09%, ctx=452356, majf=0, minf=46 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1595: Wed Feb 8 15:03:52 2017 read : io=1024.0MB, bw=2807.5KB/s, iops=701, runt=373495msec slat (usec): min=2, max=835312, avg=1312.66, stdev=4100.96 clat (usec): min=325, max=875625, avg=21479.65, stdev=11216.30 lat (usec): min=368, max=876985, avg=22793.40, stdev=11748.68 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 34], 95.00th=[ 38], | 99.00th=[ 47], 99.50th=[ 50], 99.90th=[ 57], 99.95th=[ 62], | 99.99th=[ 115] bw (KB /s): min= 937, max= 3677, per=12.70%, avg=2813.29, stdev=198.49 lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.05%, 4=0.62%, 10=9.40%, 20=36.75%, 50=52.72% lat (msec) : 100=0.42%, 250=0.01%, 1000=0.01% cpu : usr=0.35%, sys=1.08%, ctx=452558, majf=0, minf=45 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 read: (groupid=0, jobs=1): err= 0: pid=1596: Wed Feb 8 15:03:52 2017 read : io=1024.0MB, bw=2791.4KB/s, iops=697, runt=375694msec slat (usec): min=2, max=164903, avg=1314.90, stdev=3821.41 clat (usec): min=285, max=848275, avg=21610.63, stdev=11476.44 lat (usec): min=310, max=849944, avg=22926.60, stdev=11895.30 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 39], | 99.00th=[ 47], 99.50th=[ 50], 99.90th=[ 60], 99.95th=[ 67], | 99.99th=[ 167] bw (KB /s): min= 802, max= 3793, per=12.62%, avg=2796.77, stdev=201.63 lat (usec) : 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.05%, 4=0.65%, 10=9.31%, 20=36.97%, 50=52.48% lat (msec) : 100=0.49%, 250=0.02%, 1000=0.01% cpu : usr=0.35%, sys=1.09%, ctx=454697, majf=0, minf=45 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1597: Wed Feb 8 15:03:52 2017 write: io=1024.0MB, bw=2652.7KB/s, iops=663, runt=395295msec slat (usec): min=3, max=845878, avg=1380.96, stdev=4444.95 clat (usec): min=104, max=851570, avg=22739.46, stdev=12608.54 lat (usec): min=115, max=890791, avg=24121.52, stdev=13167.03 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 13], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 28], 80.00th=[ 32], 90.00th=[ 37], 95.00th=[ 42], | 99.00th=[ 52], 99.50th=[ 56], 99.90th=[ 67], 99.95th=[ 76], | 99.99th=[ 157] bw (KB /s): min= 517, max=10616, per=12.48%, avg=2647.88, stdev=752.18 lat (usec) : 250=0.06%, 500=0.02%, 750=0.03%, 1000=0.03% lat (msec) : 2=0.12%, 4=0.92%, 10=10.97%, 20=30.89%, 50=55.69% lat (msec) : 100=1.23%, 250=0.03%, 1000=0.01% cpu : usr=0.35%, sys=1.16%, ctx=486072, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1598: Wed Feb 8 15:03:52 2017 write: io=1024.0MB, bw=2666.2KB/s, iops=666, runt=393314msec slat (usec): min=3, max=822658, avg=1368.23, stdev=4384.41 clat (usec): min=141, max=870469, avg=22632.12, stdev=12223.54 lat (usec): min=156, max=881179, avg=24001.37, stdev=12768.34 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 28], 80.00th=[ 32], 90.00th=[ 37], 95.00th=[ 41], | 99.00th=[ 50], 99.50th=[ 54], 99.90th=[ 63], 99.95th=[ 68], | 99.99th=[ 176] bw (KB /s): min= 533, max= 6392, per=12.57%, avg=2667.59, stdev=620.37 lat (usec) : 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.02% lat (msec) : 2=0.07%, 4=0.53%, 10=10.41%, 20=32.02%, 50=55.94% lat (msec) : 100=0.96%, 250=0.02%, 1000=0.01% cpu : usr=0.35%, sys=1.13%, ctx=464870, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1599: Wed Feb 8 15:03:52 2017 write: io=1024.0MB, bw=2652.6KB/s, iops=663, runt=395309msec slat (usec): min=2, max=826959, avg=1378.35, stdev=4447.07 clat (usec): min=64, max=871334, avg=22744.82, stdev=12624.85 lat (usec): min=109, max=871377, avg=24124.24, stdev=13178.63 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 13], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 28], 80.00th=[ 32], 90.00th=[ 38], 95.00th=[ 42], | 99.00th=[ 51], 99.50th=[ 55], 99.90th=[ 66], 99.95th=[ 73], | 99.99th=[ 217] bw (KB /s): min= 605, max=14544, per=12.52%, avg=2656.12, stdev=833.84 lat (usec) : 100=0.01%, 250=0.07%, 500=0.02%, 750=0.03%, 1000=0.03% lat (msec) : 2=0.14%, 4=0.95%, 10=11.02%, 20=30.82%, 50=55.78% lat (msec) : 100=1.12%, 250=0.01%, 1000=0.01% cpu : usr=0.37%, sys=1.14%, ctx=485489, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1600: Wed Feb 8 15:03:52 2017 write: io=1024.0MB, bw=2652.3KB/s, iops=663, runt=395357msec slat (usec): min=2, max=828639, avg=1375.54, stdev=4432.87 clat (usec): min=97, max=877964, avg=22750.76, stdev=12747.12 lat (usec): min=113, max=878000, avg=24127.43, stdev=13340.73 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 13], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 25], | 70.00th=[ 29], 80.00th=[ 32], 90.00th=[ 38], 95.00th=[ 42], | 99.00th=[ 52], 99.50th=[ 56], 99.90th=[ 65], 99.95th=[ 70], | 99.99th=[ 190] bw (KB /s): min= 895, max=13816, per=12.48%, avg=2647.00, stdev=830.79 lat (usec) : 100=0.01%, 250=0.09%, 500=0.02%, 750=0.02%, 1000=0.15% lat (msec) : 2=0.28%, 4=0.94%, 10=11.41%, 20=29.80%, 50=55.99% lat (msec) : 100=1.28%, 250=0.01%, 1000=0.01% cpu : usr=0.36%, sys=1.16%, ctx=485452, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1601: Wed Feb 8 15:03:52 2017 write: io=1024.0MB, bw=2653.6KB/s, iops=663, runt=395155msec slat (usec): min=2, max=848383, avg=1382.02, stdev=4454.20 clat (usec): min=112, max=859676, avg=22731.80, stdev=12663.68 lat (usec): min=121, max=874836, avg=24114.92, stdev=13186.43 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 13], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 28], 80.00th=[ 32], 90.00th=[ 38], 95.00th=[ 42], | 99.00th=[ 51], 99.50th=[ 55], 99.90th=[ 65], 99.95th=[ 71], | 99.99th=[ 273] bw (KB /s): min= 423, max=10656, per=12.51%, avg=2654.80, stdev=766.89 lat (usec) : 250=0.03%, 500=0.01%, 750=0.03%, 1000=0.04% lat (msec) : 2=0.14%, 4=0.75%, 10=11.19%, 20=30.74%, 50=55.85% lat (msec) : 100=1.22%, 250=0.01%, 500=0.01%, 1000=0.01% cpu : usr=0.37%, sys=1.14%, ctx=485630, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1602: Wed Feb 8 15:03:52 2017 write: io=1024.0MB, bw=2656.4KB/s, iops=664, runt=394751msec slat (usec): min=3, max=837684, avg=1377.27, stdev=4405.77 clat (usec): min=196, max=860693, avg=22712.31, stdev=12444.72 lat (usec): min=496, max=866476, avg=24090.71, stdev=12998.70 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 13], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 28], 80.00th=[ 32], 90.00th=[ 37], 95.00th=[ 42], | 99.00th=[ 51], 99.50th=[ 55], 99.90th=[ 64], 99.95th=[ 69], | 99.99th=[ 210] bw (KB /s): min= 521, max= 9008, per=12.53%, avg=2657.47, stdev=723.53 lat (usec) : 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.02% lat (msec) : 2=0.13%, 4=0.70%, 10=11.39%, 20=30.59%, 50=56.03% lat (msec) : 100=1.09%, 250=0.01%, 1000=0.01% cpu : usr=0.38%, sys=1.12%, ctx=481726, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1603: Wed Feb 8 15:03:52 2017 write: io=1024.0MB, bw=2657.7KB/s, iops=664, runt=394559msec slat (usec): min=2, max=834338, avg=1373.82, stdev=4419.91 clat (usec): min=218, max=888774, avg=22704.23, stdev=12584.68 lat (usec): min=249, max=888789, avg=24079.11, stdev=13128.48 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 28], 80.00th=[ 32], 90.00th=[ 37], 95.00th=[ 42], | 99.00th=[ 51], 99.50th=[ 56], 99.90th=[ 66], 99.95th=[ 75], | 99.99th=[ 184] bw (KB /s): min= 172, max= 9040, per=12.54%, avg=2661.16, stdev=704.46 lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.03% lat (msec) : 2=0.08%, 4=0.73%, 10=10.97%, 20=31.31%, 50=55.69% lat (msec) : 100=1.13%, 250=0.03%, 1000=0.01% cpu : usr=0.37%, sys=1.12%, ctx=479159, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 write: (groupid=0, jobs=1): err= 0: pid=1604: Wed Feb 8 15:03:52 2017 write: io=1024.0MB, bw=2654.6KB/s, iops=663, runt=395012msec slat (usec): min=2, max=830644, avg=1384.09, stdev=4427.06 clat (usec): min=143, max=859554, avg=22721.39, stdev=12536.86 lat (usec): min=189, max=865917, avg=24106.61, stdev=13025.02 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 13], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 28], 80.00th=[ 32], 90.00th=[ 38], 95.00th=[ 42], | 99.00th=[ 51], 99.50th=[ 56], 99.90th=[ 64], 99.95th=[ 69], | 99.99th=[ 167] bw (KB /s): min= 960, max= 9448, per=12.52%, avg=2656.85, stdev=746.24 lat (usec) : 250=0.01%, 500=0.02%, 750=0.02%, 1000=0.04% lat (msec) : 2=0.14%, 4=0.73%, 10=11.38%, 20=30.78%, 50=55.64% lat (msec) : 100=1.23%, 250=0.01%, 1000=0.01% cpu : usr=0.33%, sys=1.18%, ctx=485514, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=8192.0MB, aggrb=22153KB/s, minb=2769KB/s, maxb=2807KB/s, mint=373495msec, maxt=378653msec WRITE: io=8192.0MB, aggrb=21217KB/s, minb=2652KB/s, maxb=2666KB/s, mint=393314msec, maxt=395357msec Disk stats (read/write): vdb: ios=2097152/2096033, merge=0/84, ticks=12121364/12354172, in_queue=24473904, util=100.00%