Verification with bionic-proposed of the I/O Error path. All good, working as expected (see comments #11 to #16). # uname -rv 4.15.0-56-generic #62-Ubuntu SMP Wed Jul 24 20:18:55 UTC 2019 test 1 ------ # ./setup.sh >/dev/null 2>&1 [ 369.375820] bcache: register_bdev() registered backing device dm-0 [ 369.395195] bcache: run_cache_set() invalidating existing data [ 369.410278] bcache: register_cache() registered cache device dm-1 [ 371.393391] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set c1126837-e029-4d08-bad3-38ff8bc08054 # lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop └─fake-loop0 253:0 0 1024M 0 dm └─bcache0 251:0 0 1024M 0 disk loop1 7:1 0 1G 0 loop └─fake-loop1 253:1 0 1024M 0 dm └─bcache0 251:0 0 1024M 0 disk On another shell: # fio --name=write --rw=randwrite --filename=/dev/bcache0 --bs=4k --iodepth=8 --ioengine=libaio --runtime=300s --continue_on_error=all # [ 425.656209] bcache: bch_count_io_errors() dm-1: IO error on writing btree, recovering [ 425.684837] bcache: error on c1126837-e029-4d08-bad3-38ff8bc08054: [ 425.684840] journal io error [ 425.686537] , disabling caching [ 425.688849] Buffer I/O error on dev bcache0, logical block 2807, lost async page write [ 425.691541] Buffer I/O error on dev bcache0, logical block 2808, lost async page write [ 425.694131] bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache0 is "auto" and cache is clean, keep it alive. [ 425.698343] Buffer I/O error on dev bcache0, logical block 2810, lost async page write [ 425.702522] Buffer I/O error on dev bcache0, logical block 2812, lost async page write [ 425.705326] Buffer I/O error on dev bcache0, logical block 2813, lost async page write [ 425.707896] Buffer I/O error on dev bcache0, logical block 2814, lost async page write [ 425.710692] Buffer I/O error on dev bcache0, logical block 2816, lost async page write [ 425.713524] Buffer I/O error on dev bcache0, logical block 2817, lost async page write [ 425.716512] Buffer I/O error on dev bcache0, logical block 2818, lost async page write [ 425.719156] Buffer I/O error on dev bcache0, logical block 2819, lost async page write [ 425.742817] bcache: cached_dev_detach_finish() Caching disabled for dm-0 [ 425.746933] bcache: bch_count_io_errors() dm-1: IO error on writing btree, recovering [ 425.750502] bcache: cache_set_free() Cache set c1126837-e029-4d08-bad3-38ff8bc08054 unregistered fio finished: Run status group 0 (all jobs): WRITE: bw=212MiB/s (222MB/s), 212MiB/s-212MiB/s (222MB/s-222MB/s), io=1024MiB (1074MB), run=4830-4830msec bcache not on top of caching device: # lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop └─fake-loop0 253:0 0 1024M 0 dm └─bcache0 251:0 0 1024M 0 disk loop1 7:1 0 1G 0 loop fake-loop1 253:1 0 1G 0 dm test 2 ------ # ./setup.sh >/dev/null 2>&1 [ 23.946411] bcache: register_bdev() registered backing device dm-0 [ 23.952262] bcache: run_cache_set() invalidating existing data [ 23.966564] bcache: register_cache() registered cache device dm-1 [ 25.949934] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set d7a3c644-e21e-49bb-bcee-e14709a65745 # lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop └─fake-loop0 253:0 0 1024M 0 dm └─bcache0 251:0 0 1024M 0 disk loop1 7:1 0 1G 0 loop └─fake-loop1 253:1 0 1024M 0 dm └─bcache0 251:0 0 1024M 0 disk # echo writeback > /sys/block/bcache0/bcache/cache_mode # dd if=/dev/zero of=/dev/bcache0 bs=4k dd: error writing '/dev/bcache0': No space left on device 262142+0 records in 262141+0 records out 1073729536 bytes (1.1 GB, 1.0 GiB) copied, 4.2152 s, 255 MB/s # ./dm_fake_dev.sh /dev/loop1 bad [ 55.515809] Buffer I/O error on dev dm-1, logical block 262128, async page read [ 55.522181] Buffer I/O error on dev dm-1, logical block 262128, async page read [ 55.528425] bcache: register_bcache() error /dev/dm-1: device already registered # [ 74.716322] bcache: bch_count_io_errors() dm-1: IO error on reading dirty data from cache, recovering [ 74.719766] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 74.808148] bcache: bch_count_io_errors() dm-1: IO error on reading dirty data from cache, recovering [ 74.813328] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 74.964134] bcache: bch_count_io_errors() dm-1: IO error on reading dirty data from cache, recovering ... [ 85.184111] bcache: bch_count_io_errors() dm-1: IO error on reading dirty data from cache, recovering [ 85.189769] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 85.364144] bcache: bch_count_io_errors() dm-1: IO error on reading dirty data from cache, recovering [ 85.369008] bcache: bch_cached_dev_error() stop bcache0: too many IO errors on backing device dm-0 [ 85.369008] [ 85.374890] bcache: bch_count_io_errors() dm-1: IO error on reading dirty data from cache, recovering [ 85.379004] bcache: bcache_device_free() bcache0 stopped bcache device is removed # lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop └─fake-loop0 253:0 0 1024M 0 dm loop1 7:1 0 1G 0 loop fake-loop1 253:1 0 1G 0 dm test 3 ------ # ./setup.sh >/dev/null 2>&1 [ 31.671004] bcache: register_bdev() registered backing device dm-0 [ 31.676981] bcache: run_cache_set() invalidating existing data [ 31.686989] bcache: register_cache() registered cache device dm-1 [ 33.678658] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set 12ed8698-10f0-4a6d-bee1-4f11d5d6bf20 # lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop └─fake-loop0 253:0 0 1024M 0 dm └─bcache0 251:0 0 1024M 0 disk loop1 7:1 0 1G 0 loop └─fake-loop1 253:1 0 1024M 0 dm └─bcache0 251:0 0 1024M 0 disk # echo 1 > /sys/block/bcache0/bcache/detach # [ 43.394627] bcache: cached_dev_detach_finish() Caching disabled for dm-0 # lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop └─fake-loop0 253:0 0 1024M 0 dm └─bcache0 251:0 0 1024M 0 disk loop1 7:1 0 1G 0 loop └─fake-loop1 253:1 0 1024M 0 dm # dd if=/dev/zero of=/dev/bcache0 bs=4k dd: error writing '/dev/bcache0': No space left on device 262142+0 records in 262141+0 records out 1073729536 bytes (1.1 GB, 1.0 GiB) copied, 3.69032 s, 291 MB/s # sync # iostat -xy 1 10 /dev/bcache0 | grep bcache0 bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 # test 4 ------ # ./setup.sh >/dev/null 2>&1 [ 19.867266] bcache: register_bdev() registered backing device dm-0 [ 20.032234] bcache: run_cache_set() invalidating existing data [ 20.310143] bcache: register_cache() registered cache device dm-1 [ 22.034294] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set 5304f479-8f93-487e-a5cb-eb940e21751a # echo writeback > /sys/block/bcache0/bcache/cache_mode # cat /sys/block/bcache0/bcache/cache_mode writethrough [writeback] writearound none # ./dm_fake_dev.sh /dev/loop0 bad [ 39.618330] Buffer I/O error on dev dm-0, logical block 262128, async page read [ 39.623830] Buffer I/O error on dev dm-0, logical block 262128, async page read [ 39.631524] bcache: register_bcache() error /dev/dm-0: device already registered (emitting change event) [ 39.633892] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 39.639076] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 39.642480] Buffer I/O error on dev bcache0, logical block 262112, async page read # [ 39.648190] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 39.651837] Buffer I/O error on dev bcache0, logical block 262112, async page read [ 39.654309] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 39.657289] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 39.660116] Buffer I/O error on dev bcache0, logical block 1, async page read # dd if=/dev/zero of=/dev/bcache0 bs=4k [ 60.268137] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 60.271183] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 60.273467] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 60.275693] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 60.277945] Buffer I/O error on dev bcache0, logical block 0, lost async page write [ 60.279908] Buffer I/O error on dev bcache0, logical block 1, lost async page write [ 60.280160] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 60.281866] Buffer I/O error on dev bcache0, logical block 2, lost async page write [ 60.286340] Buffer I/O error on dev bcache0, logical block 4, lost async page write ... [ 60.456411] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 60.459153] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 60.461861] bcache: bch_cached_dev_error() stop bcache0: too many IO errors on backing device dm-0 [ 60.461861] dd: error writing '/dev/bcache0': No space left on device 262142+0 records in 262141+0 records out 1073729536 bytes (1.1 GB, 1.0 GiB) copied, 2.12236 s, 506 MB/s [ 60.931271] bcache: bcache_device_free() bcache0 stopped # lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop loop1 7:1 0 1G 0 loop └─fake-loop1 253:1 0 1024M 0 dm fake-loop0 253:0 0 1G 0 dm # ls /dev/bcache0 ls: cannot access '/dev/bcache0': No such file or directory test 5 ------ # modprobe scsi_debug dev_size_mb=1024 # dmesg | tail [ 72.008739] scsi host2: scsi_debug: version 1.86 [20160430] dev_size_mb=1024, opts=0x0, submit_queues=1, statistics=0 [ 72.009273] scsi 2:0:0:0: Direct-Access Linux scsi_debug 0186 PQ: 0 ANSI: 7 [ 72.010159] sd 2:0:0:0: Power-on or device reset occurred [ 72.012001] sd 2:0:0:0: Attached scsi generic sg0 type 0 [ 72.018462] sd 2:0:0:0: [sda] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB) [ 72.022566] sd 2:0:0:0: [sda] Write Protect is off [ 72.022571] sd 2:0:0:0: [sda] Mode Sense: 73 00 10 08 [ 72.030790] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 72.121972] sd 2:0:0:0: [sda] Attached SCSI disk # ./setup-sda.sh >/dev/null 2>&1 [ 83.685946] bcache: run_cache_set() invalidating existing data [ 83.697155] bcache: register_cache() registered cache device dm-0 [ 83.795953] bcache: register_bdev() registered backing device sda [ 84.638672] bcache: bch_cached_dev_attach() Caching sda as bcache0 on set de4439a0-7a3e-42dc-9415-c10525eb1e6d # lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop └─fake-loop0 253:0 0 1024M 0 dm └─bcache0 251:0 0 1024M 0 disk sda 8:0 0 1G 0 disk └─bcache0 251:0 0 1024M 0 disk # echo writeback > /sys/block/sda/bcache/cache_mode # echo 1 > /sys/block/sda/device/delete [ 99.172506] sd 2:0:0:0: [sda] Synchronizing SCSI cache # sleep 10 [ 104.284209] bcache: cached_dev_status_update() sda: device offline for 5 seconds [ 104.287916] bcache: cached_dev_status_update() bdev0: disable I/O request due to backing device offline [ 104.292588] bcache: bcache_device_free() bcache0 stopped # # lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop └─fake-loop0 253:0 0 1024M 0 dm # ls /dev/bcache0 ls: cannot access '/dev/bcache0': No such file or directory test 6 ------ # ./setup-two-bcache-one-cache.sh >/dev/null 2>&1 [ 23.100810] bcache: register_bdev() registered backing device dm-0 [ 23.114149] bcache: run_cache_set() invalidating existing data [ 23.123286] bcache: register_cache() registered cache device dm-2 [ 23.123552] bcache: register_bdev() registered backing device dm-1 [ 24.130116] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set 866e8bce-38c8-491c-a845-097cc5373ff6 [ 24.136297] bcache: bch_cached_dev_attach() Caching dm-1 as bcache1 on set 866e8bce-38c8-491c-a845-097cc5373ff6 # lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop └─fake-loop0 253:0 0 1024M 0 dm └─bcache0 251:0 0 1024M 0 disk loop1 7:1 0 1G 0 loop └─fake-loop1 253:1 0 1024M 0 dm └─bcache1 251:128 0 1024M 0 disk loop2 7:2 0 1G 0 loop └─fake-loop2 253:2 0 1024M 0 dm ├─bcache0 251:0 0 1024M 0 disk └─bcache1 251:128 0 1024M 0 disk e writebackic-bcache:~# echo writeback | tee /sys/block/bcache*/bcache/cache_mode e_set_failedbcache:~# echo always | tee /sys/block/bcache*/bcache/stop_when_cache always # ./dm_fake_dev.sh /dev/loop0 bad [ 54.190905] Buffer I/O error on dev dm-0, logical block 262128, async page read [ 54.196046] Buffer I/O error on dev dm-0, logical block 262128, async page read [ 54.202079] bcache: register_bcache() error /dev/dm-0: device already registered (emitting change event) [ 54.204484] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 54.209396] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 54.212415] Buffer I/O error on dev bcache0, logical block 262112, async page read # [ 54.217664] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 54.221146] Buffer I/O error on dev bcache0, logical block 262112, async page read [ 54.223735] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 54.226221] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 54.228466] Buffer I/O error on dev bcache0, logical block 1, async page read root@guest-bcache:~# dd if=/dev/zero of=/dev/bcache1 bs=4k & dd if=/dev/zero of=/dev/bcache0 bs=4k & [1] 1385 [2] 1386 # [ 66.991499] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 67.003045] Buffer I/O error on dev bcache0, logical block 0, lost async page write [ 67.005222] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 67.008246] Buffer I/O error on dev bcache0, logical block 1, lost async page write [ 67.010836] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 67.013786] Buffer I/O error on dev bcache0, logical block 2, lost async page write [ 67.016412] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 67.019387] Buffer I/O error on dev bcache0, logical block 3, lost async page write [ 67.021986] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 67.025013] Buffer I/O error on dev bcache0, logical block 4, lost async page write [ 67.027636] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 67.030614] Buffer I/O error on dev bcache0, logical block 5, lost async page write [ 67.033269] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 67.036262] Buffer I/O error on dev bcache0, logical block 6, lost async page write [ 67.038893] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 67.041868] Buffer I/O error on dev bcache0, logical block 7, lost async page write [ 67.044524] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 67.047529] Buffer I/O error on dev bcache0, logical block 8, lost async page write [ 67.050115] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 67.053136] Buffer I/O error on dev bcache0, logical block 9, lost async page write [ 67.056327] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 67.059360] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable ... [ 67.206725] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 67.210109] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 67.212578] bcache: bch_cached_dev_error() stop bcache0: too many IO errors on backing device dm-0 [ 67.212578] dd: error writing '/dev/bcache0': No space left on device 262142+0 records[ 68.393613] bcache: bcache_device_free() bcache0 stopped in 262141+0 records out 1073729536 bytes (1.1 GB, 1.0 GiB) copied, 2.32385 s, 462 MB/s dd: error writing '/dev/bcache1': No space left on device 262142+0 records in 262141+0 records out 1073729536 bytes (1.1 GB, 1.0 GiB) copied, 4.5021 s, 238 MB/s # lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop loop1 7:1 0 1G 0 loop └─fake-loop1 253:1 0 1024M 0 dm └─bcache1 251:128 0 1024M 0 disk loop2 7:2 0 1G 0 loop └─fake-loop2 253:2 0 1024M 0 dm └─bcache1 251:128 0 1024M 0 disk fake-loop0 253:0 0 1G 0 dm only bcache0 was stopped. bcache1 remains working. # reboot # ./setup-two-bcache-one-cache.reboot.sh >/dev/null 2>&1 [ 17.697259] bcache: register_bdev() registered backing device dm-0 [ 17.770242] bcache: register_bdev() registered backing device dm-1 [ 17.902012] bcache: bch_journal_replay() journal replay done, 57093 keys in 34 entries, seq 161 [ 17.908344] bcache: bch_cached_dev_attach() Caching dm-1 as bcache1 on set 866e8bce-38c8-491c-a845-097cc5373ff6 [ 17.914819] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set 866e8bce-38c8-491c-a845-097cc5373ff6 [ 17.918484] bcache: register_cache() registered cache device dm-2 # lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop └─fake-loop0 253:0 0 1024M 0 dm └─bcache0 251:0 0 1024M 0 disk loop1 7:1 0 1G 0 loop └─fake-loop1 253:1 0 1024M 0 dm └─bcache1 251:128 0 1024M 0 disk loop2 7:2 0 1G 0 loop └─fake-loop2 253:2 0 1024M 0 dm ├─bcache0 251:0 0 1024M 0 disk └─bcache1 251:128 0 1024M 0 disk both bcache devices reattached after reboot.