On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg
[ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912 [ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write [ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write [ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write [ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write [ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write [ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write [ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write [ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write [ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write [ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write [ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912 <snip> [ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744 [ 1012.755396] buffer_io_error: 194552 callbacks suppressed [ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write [ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write [ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write [ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write [ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write
Able to replicate this with a bonnie++ stress test.
bonnie++ -d /mnt/test/ -r 1000
Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg
[ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912
[ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write
[ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write
[ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write
[ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write
[ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write
[ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write
[ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write
[ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write
[ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write
[ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write
[ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912
<snip>
[ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744
[ 1012.755396] buffer_io_error: 194552 callbacks suppressed
[ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write
[ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write
[ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write
[ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write
[ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write
Able to replicate this with a bonnie++ stress test.
bonnie++ -d /mnt/test/ -r 1000
Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux