Comment 6 for bug 1031260

Hi, I've tested with 3.5 (it's still running). Took another one than the one you linked, as you linked -rc7 and the official release was on the site too (so just 3.5.0 no rc/prerelease). It seems somewhat resolved (note - I'm no kernel dev ;). With that I mean the values are much higher now, however, the max_hw_sector_kb has a value of 16383 which is an odd number and is 16MiB - 1 kiB. The minus 1kiB does not make sense to me whatsoever.

Have done some simple tests with values like 512 (default after boot), 2048 and most notably 12288 seems to perform best (64k chunk size, 7 disk RAID-5 (6 data disks thus) * 32 (makes 64*6*32)).

That has doubled my write performance w/o touching any other settings whatsoever.

This is not very extensive, just a copy/paste from my console. Had a 'while (true); do killall -USR1 dd; sleep 20; done' running in another terminal to have dd print statistics in between.

At the time of testing nothing else was accessing any of the disks in the RAID array. There was sufficient time between the dd's for the kernel to flush it's buffers. Didn't copy the output for free, but checked the size of the buffers with that before starting the next dd. Wasn't more than a couple of MiB's before starting with over 3G RAM free, which explains the initial (way) higher speeds.

root@datavault:/sys/block/md4/queue# cat max_sectors_kb
512
root@datavault:/sys/block/md4/queue# cat max_hw_sectors_kb
16383
root@datavault:/sys/block/md4/queue# dd if=/dev/zero of=/dev/md4p2 bs=1M count=20480
2482+0 records in
2482+0 records out
2602565632 bytes (2.6 GB) copied, 27.5633 s, 94.4 MB/s
3795+0 records in
3795+0 records out
3979345920 bytes (4.0 GB) copied, 47.6073 s, 83.6 MB/s
4980+0 records in
4980+0 records out
5221908480 bytes (5.2 GB) copied, 67.7112 s, 77.1 MB/s
5383+0 records in
5383+0 records out
5644484608 bytes (5.6 GB) copied, 87.6912 s, 64.4 MB/s
5931+0 records in
5931+0 records out
6219104256 bytes (6.2 GB) copied, 107.683 s, 57.8 MB/s
6827+0 records in
6827+0 records out
7158628352 bytes (7.2 GB) copied, 127.683 s, 56.1 MB/s
7920+0 records in
7920+0 records out
8304721920 bytes (8.3 GB) copied, 147.699 s, 56.2 MB/s
8594+0 records in
8594+0 records out
9011462144 bytes (9.0 GB) copied, 167.704 s, 53.7 MB/s
9157+0 records in
9157+0 records out
9601810432 bytes (9.6 GB) copied, 187.723 s, 51.1 MB/s
9709+0 records in
9709+0 records out
10180624384 bytes (10 GB) copied, 207.755 s, 49.0 MB/s
11159+0 records in
11159+0 records out
11701059584 bytes (12 GB) copied, 227.724 s, 51.4 MB/s
12307+0 records in
12307+0 records out
12904824832 bytes (13 GB) copied, 247.744 s, 52.1 MB/s
12867+0 records in
12867+0 records out
13492027392 bytes (13 GB) copied, 267.759 s, 50.4 MB/s
13634+0 records in
13634+0 records out
14296285184 bytes (14 GB) copied, 287.759 s, 49.7 MB/s
14287+0 records in
14287+0 records out
14981005312 bytes (15 GB) copied, 307.784 s, 48.7 MB/s
15118+0 records in
15118+0 records out
15852371968 bytes (16 GB) copied, 327.795 s, 48.4 MB/s
15699+0 records in
15699+0 records out
16461594624 bytes (16 GB) copied, 347.775 s, 47.3 MB/s
16325+0 records in
16325+0 records out
17118003200 bytes (17 GB) copied, 367.835 s, 46.5 MB/s
16938+0 records in
16938+0 records out
17760780288 bytes (18 GB) copied, 387.863 s, 45.8 MB/s
17476+0 records in
17476+0 records out
18324914176 bytes (18 GB) copied, 407.855 s, 44.9 MB/s
18234+0 records in
18234+0 records out
19119734784 bytes (19 GB) copied, 427.859 s, 44.7 MB/s
18887+0 records in
18887+0 records out
19804454912 bytes (20 GB) copied, 447.867 s, 44.2 MB/s
19393+0 records in
19393+0 records out
20335034368 bytes (20 GB) copied, 467.911 s, 43.5 MB/s
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB) copied, 511.516 s, 42.0 MB/s
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB) copied, 511.516 s, 42.0 MB/s

root@datavault:/sys/block/md4/queue# mdadm --detail /dev/md4
/dev/md4:
        Version : 0.90
  Creation Time : Thu Oct 20 23:48:39 2011
     Raid Level : raid5
     Array Size : 11721086976 (11178.10 GiB 12002.39 GB)
  Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
   Raid Devices : 7
  Total Devices : 8
Preferred Minor : 4
    Persistence : Superblock is persistent

    Update Time : Wed Aug 1 15:32:48 2012
          State : clean
 Active Devices : 7
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 2213e62a:723f32ab:c4e8ec2d:2a80eff0 (local to host datavault)
         Events : 0.3449563

    Number Major Minor RaidDevice State
       0 8 64 0 active sync /dev/sde
       1 8 80 1 active sync /dev/sdf
       2 8 96 2 active sync /dev/sdg
       3 8 112 3 active sync /dev/sdh
       4 8 128 4 active sync /dev/sdi
       5 8 176 5 active sync /dev/sdl
       6 8 160 6 active sync /dev/sdk

       7 8 144 - spare /dev/sdj

root@datavault:/sys/block/md4/queue# echo 12288 > max_sectors_kb
root@datavault:/sys/block/md4/queue# dd if=/dev/zero of=/dev/md4p2 bs=1M count=20480
381+0 records in
381+0 records out
399507456 bytes (400 MB) copied, 0.255935 s, 1.6 GB/s
1946+0 records in
1946+0 records out
2040528896 bytes (2.0 GB) copied, 20.2837 s, 101 MB/s
3318+0 records in
3318+0 records out
3479175168 bytes (3.5 GB) copied, 40.2759 s, 86.4 MB/s
4853+0 records in
4853+0 records out
5088739328 bytes (5.1 GB) copied, 60.2838 s, 84.4 MB/s
6282+0 records in
6282+0 records out
6587154432 bytes (6.6 GB) copied, 80.3035 s, 82.0 MB/s
7811+0 records in
7811+0 records out
8190427136 bytes (8.2 GB) copied, 100.3 s, 81.7 MB/s
9402+0 records in
9402+0 records out
9858711552 bytes (9.9 GB) copied, 120.324 s, 81.9 MB/s
10800+0 records in
10800+0 records out
11324620800 bytes (11 GB) copied, 140.324 s, 80.7 MB/s
12399+0 records in
12399+0 records out
13001293824 bytes (13 GB) copied, 160.336 s, 81.1 MB/s
13904+0 records in
13904+0 records out
14579400704 bytes (15 GB) copied, 180.352 s, 80.8 MB/s
15215+0 records in
15215+0 records out
15954083840 bytes (16 GB) copied, 200.356 s, 79.6 MB/s
16250+0 records in
16250+0 records out
17039360000 bytes (17 GB) copied, 220.36 s, 77.3 MB/s
17082+0 records in
17082+0 records out
17911775232 bytes (18 GB) copied, 240.376 s, 74.5 MB/s
17559+0 records in
17559+0 records out
18411945984 bytes (18 GB) copied, 260.412 s, 70.7 MB/s
18276+0 records in
18276+0 records out
19163774976 bytes (19 GB) copied, 280.404 s, 68.3 MB/s
19474+0 records in
19474+0 records out
20419969024 bytes (20 GB) copied, 300.404 s, 68.0 MB/s
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB) copied, 333.692 s, 64.4 MB/s
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB) copied, 333.692 s, 64.4 MB/s

root@datavault:/sys/block/md4/queue# echo 2048 > max_sectors_kb
root@datavault:/sys/block/md4/queue# dd if=/dev/zero of=/dev/md4p2 bs=1M count=20480
1465+0 records in
1465+0 records out
1536163840 bytes (1.5 GB) copied, 12.9843 s, 118 MB/s
2763+0 records in
2763+0 records out
2897215488 bytes (2.9 GB) copied, 33.0076 s, 87.8 MB/s
4138+0 records in
4138+0 records out
4339007488 bytes (4.3 GB) copied, 53.0197 s, 81.8 MB/s
5228+0 records in
5228+0 records out
5481955328 bytes (5.5 GB) copied, 73.0554 s, 75.0 MB/s
5958+0 records in
5958+0 records out
6247415808 bytes (6.2 GB) copied, 93.0753 s, 67.1 MB/s
6699+0 records in
6699+0 records out
7024410624 bytes (7.0 GB) copied, 113.08 s, 62.1 MB/s
8030+0 records in
8030+0 records out
8420065280 bytes (8.4 GB) copied, 133.083 s, 63.3 MB/s
8791+0 records in
8791+0 records out
9218031616 bytes (9.2 GB) copied, 153.131 s, 60.2 MB/s
9313+0 records in
9313+0 records out
9765388288 bytes (9.8 GB) copied, 173.183 s, 56.4 MB/s
9741+0 records in
9741+0 records out
10214178816 bytes (10 GB) copied, 193.195 s, 52.9 MB/s
10929+0 records in
10929+0 records out
11459887104 bytes (11 GB) copied, 213.208 s, 53.7 MB/s
11848+0 records in
11848+0 records out
12423528448 bytes (12 GB) copied, 233.231 s, 53.3 MB/s
12383+0 records in
12383+0 records out
12984516608 bytes (13 GB) copied, 253.235 s, 51.3 MB/s
12915+0 records in
12915+0 records out
13542359040 bytes (14 GB) copied, 273.255 s, 49.6 MB/s
13693+0 records in
13693+0 records out
14358151168 bytes (14 GB) copied, 293.249 s, 49.0 MB/s
15156+0 records in
15156+0 records out
15892217856 bytes (16 GB) copied, 313.276 s, 50.7 MB/s
16215+0 records in
16215+0 records out
17002659840 bytes (17 GB) copied, 333.315 s, 51.0 MB/s
16827+0 records in
16827+0 records out
17644388352 bytes (18 GB) copied, 353.283 s, 49.9 MB/s
17899+0 records in
17899+0 records out
18768461824 bytes (19 GB) copied, 373.315 s, 50.3 MB/s
19061+0 records in
19061+0 records out
19986907136 bytes (20 GB) copied, 393.359 s, 50.8 MB/s
19847+0 records in
19847+0 records out
20811087872 bytes (21 GB) copied, 413.375 s, 50.3 MB/s
20437+0 records in
20437+0 records out
21429747712 bytes (21 GB) copied, 433.379 s, 49.4 MB/s
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB) copied, 456.783 s, 47.0 MB/s
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB) copied, 456.783 s, 47.0 MB/s