I/O Error Test 6 (for the Disco kernel) ================ commit: 'Revert "bcache: set CACHE_SET_IO_DISABLE in bch_cached_dev_error()"' Problem: if one backing device hits I/O errors the cache device is disabled, but if that cache device is shared by other bcache devices they stop too (even with non-failing backing devices). Original kernel: all bcache devices that share cache device with failing backing device are stopped. Modified kernel: only the bcache device with the failing backing device is stopped. Original kernel: --------------- root@bionic-bcache:~# uname -rv 5.0.0-21-generic #22-Ubuntu SMP Tue Jul 2 13:27:33 UTC 2019 root@bionic-bcache:~# ./setup-two-bcache-one-cache.sh >/dev/null 2>&1 [ 23.323929] bcache: register_bdev() registered backing device dm-0 [ 23.330821] bcache: register_bdev() registered backing device dm-1 [ 23.335493] bcache: run_cache_set() invalidating existing data [ 23.347255] bcache: register_cache() registered cache device dm-2 [ 24.335738] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set f816e09d-f744-4fc9-b3bd-239f3d5093c6 [ 24.342388] bcache: bch_cached_dev_attach() Caching dm-1 as bcache1 on set f816e09d-f744-4fc9-b3bd-239f3d5093c6 root@bionic-bcache:~# lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop └─fake-loop0 253:0 0 1024M 0 dm └─bcache0 251:0 0 1024M 0 disk loop1 7:1 0 1G 0 loop └─fake-loop1 253:1 0 1024M 0 dm └─bcache1 251:128 0 1024M 0 disk loop2 7:2 0 1G 0 loop └─fake-loop2 253:2 0 1024M 0 dm ├─bcache0 251:0 0 1024M 0 disk └─bcache1 251:128 0 1024M 0 disk # echo writeback | tee /sys/block/bcache*/bcache/cache_mode writeback # echo always | tee /sys/block/bcache*/bcache/stop_when_cache_set_failed always root@bionic-bcache:~# ./dm_fake_dev.sh /dev/loop0 bad [ 58.915344] Buffer I/O error on dev dm-0, logical block 262128, async page read [ 58.921948] Buffer I/O error on dev dm-0, logical block 262128, async page read [ 58.928886] bcache: register_bcache() error /dev/dm-0: device already registered (emitting change event) [ 58.931006] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 58.936386] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 58.939346] Buffer I/O error on dev bcache0, logical block 262112, async page read root@bionic-bcache:~# [ 58.944685] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 58.948468] Buffer I/O error on dev bcache0, logical block 262112, async page read [ 58.951078] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 58.954231] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 58.957216] Buffer I/O error on dev bcache0, logical block 1, async page read # ./dm_fake_dev.sh /dev/loop0 bad [ 167.341298] Buffer I/O error on dev dm-0, logical block 262128, async page read [ 167.347802] Buffer I/O error on dev dm-0, logical block 262128, async page read [ 167.354959] bcache: register_bcache() error /dev/dm-0: device already registered (emitting change event) [ 167.356585] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 167.364784] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 167.369083] Buffer I/O error on dev bcache0, logical block 262112, async page read root@bionic-bcache:~# [ 167.376976] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 167.381644] Buffer I/O error on dev bcache0, logical block 262112, async page read [ 167.384195] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 167.387144] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 167.390040] Buffer I/O error on dev bcache0, logical block 1, async page read root@guest-bcache:~# dd if=/dev/zero of=/dev/bcache1 bs=4k & dd if=/dev/zero of=/dev/bcache0 bs=4k & [1] 1464 [2] 1465 root@bionic-bcache:~# [ 178.103060] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 178.107790] Buffer I/O error on dev bcache0, logical block 0, lost async page write [ 178.111814] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 178.116428] Buffer I/O error on dev bcache0, logical block 1, lost async page write [ 178.119286] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 178.122070] Buffer I/O error on dev bcache0, logical block 2, lost async page write [ 178.122601] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 178.128535] Buffer I/O error on dev bcache0, logical block 3, lost async page write [ 178.132472] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 178.136169] Buffer I/O error on dev bcache0, logical block 4, lost async page write [ 178.139426] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 178.143021] Buffer I/O error on dev bcache0, logical block 5, lost async page write [ 178.146279] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 178.149876] Buffer I/O error on dev bcache0, logical block 6, lost async page write [ 178.153119] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 178.156697] Buffer I/O error on dev bcache0, logical block 7, lost async page write [ 178.159941] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 178.163519] Buffer I/O error on dev bcache0, logical block 8, lost async page write [ 178.166783] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 178.170706] Buffer I/O error on dev bcache0, logical block 9, lost async page write [ 178.173933] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 178.177574] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 178.181235] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable ... [ 178.362803] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 178.366412] bcache: bch_cached_dev_error() stop bcache0: too many IO errors on backing device dm-0 [ 178.366412] [ 178.501362] bcache: bch_cache_set_error() CACHE_SET_IO_DISABLE already set [ 178.504932] bcache: bch_cache_set_error() bcache: error on f816e09d-f744-4fc9-b3bd-239f3d5093c6: [ 178.509390] journal io error [ 178.509391] bcache: bch_cache_set_error() , disabling caching [ 178.509391] [ 178.517586] bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache0 is "always", stop it for failed cache set f816 e09d-f744-4fc9-b3bd-239f3d5093c6. [ 178.524925] bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache1 is "always", stop it for failed cache set f816 e09d-f744-4fc9-b3bd-239f3d5093c6. [ 178.562349] bcache: cached_dev_detach_finish() Caching disabled for dm-1 dd: error writing '/dev/bcache0': No space left on device 262142+0 records in 262141+0 records out 1073729536 bytes (1.1 GB, 1.0 GiB) copied, 2.89317 s, 371 MB/s [ 180.186818] bcache: bcache_device_free() bcache0 stopped [ 180.188875] bcache: bch_count_io_errors() dm-2: IO error on writing btree. [ 180.214681] bcache: cache_set_free() Cache set f816e09d-f744-4fc9-b3bd-239f3d5093c6 unregistered dd: error writing '/dev/bcache1': No space left on device [ 181.732575] bcache: bcache_device_free() bcache1 stopped 262142+0 records in 262141+0 records out 1073729536 bytes (1.1 GB, 1.0 GiB) copied, 4.44023 s, 242 MB/s root@bionic-bcache:~# lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop loop1 7:1 0 1G 0 loop └─fake-loop1 253:1 0 1024M 0 dm loop2 7:2 0 1G 0 loop └─fake-loop2 253:2 0 1024M 0 dm fake-loop0 253:0 0 1G 0 dm both bcache0 and bcache1 devices removed. Modified kernel: --------------- root@bionic-bcache:~# uname -rv 5.0.0-21-generic #22+test20190707build1 SMP Mon Jul 8 01:50:31 UTC 2019 root@bionic-bcache:~# ./setup-two-bcache-one-cache.sh >/dev/null 2>&1 [ 25.668092] bcache: register_bdev() registered backing device dm-0 [ 25.680959] bcache: register_bdev() registered backing device dm-1 [ 25.686178] bcache: run_cache_set() invalidating existing data [ 25.695269] bcache: register_cache() registered cache device dm-2 [ 26.691859] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set b3823d82-8753-44ef-a7df-e1271b667021 [ 26.698108] bcache: bch_cached_dev_attach() Caching dm-1 as bcache1 on set b3823d82-8753-44ef-a7df-e1271b667021 root@bionic-bcache:~# lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop └─fake-loop0 253:0 0 1024M 0 dm └─bcache1 251:128 0 1024M 0 disk loop1 7:1 0 1G 0 loop └─fake-loop1 253:1 0 1024M 0 dm └─bcache0 251:0 0 1024M 0 disk loop2 7:2 0 1G 0 loop └─fake-loop2 253:2 0 1024M 0 dm ├─bcache0 251:0 0 1024M 0 disk └─bcache1 251:128 0 1024M 0 disk # echo writeback | tee /sys/block/bcache*/bcache/cache_mode writeback # echo always | tee /sys/block/bcache*/bcache/stop_when_cache_set_failed always root@bionic-bcache:~# ./dm_fake_dev.sh /dev/loop0 bad [ 49.073126] Buffer I/O error on dev dm-0, logical block 262128, async page read [ 49.079509] Buffer I/O error on dev dm-0, logical block 262128, async page read [ 49.086012] bcache: register_bcache() error /dev/dm-0: device already registered (emitting change event) [ 49.088466] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 49.093359] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 49.096298] Buffer I/O error on dev bcache0, logical block 262112, async page read root@bionic-bcache:~# [ 49.100583] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 49.103578] Buffer I/O error on dev bcache0, logical block 262112, async page read [ 49.107542] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 49.111926] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 49.116200] Buffer I/O error on dev bcache0, logical block 1, async page read root@guest-bcache:~# dd if=/dev/zero of=/dev/bcache1 bs=4k & dd if=/dev/zero of=/dev/bcache0 bs=4k & [1] 1453 [2] 1454 root@bionic-bcache:~# [ 55.398092] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 55.404433] Buffer I/O error on dev bcache0, logical block 0, lost async page write [ 55.409868] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 55.414151] Buffer I/O error on dev bcache0, logical block 1, lost async page write [ 55.417134] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 55.420521] Buffer I/O error on dev bcache0, logical block 2, lost async page write [ 55.423094] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 55.428977] Buffer I/O error on dev bcache0, logical block 3, lost async page write [ 55.433236] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 55.436400] Buffer I/O error on dev bcache0, logical block 4, lost async page write [ 55.439314] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable ... [ 55.720661] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 55.726927] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 55.734921] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 55.743469] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 55.747248] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 55.750829] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 55.754349] bcache: bch_count_backing_io_errors() dm-0: IO error on backing device, unrecoverable [ 55.757972] bcache: bch_cached_dev_error() stop bcache0: too many IO errors on backing device dm-0 [ 55.757972] dd: error writing '/dev/bcache1': No space left on device dd: error writing '/dev/bcache0': No space left on device 262142+0 records in 262141+0 records out 1073729536 bytes (1.1 GB, 1.0 GiB) copied, 3.62916 s, 296 MB/s [ 58.188089] bcache: bcache_device_free() bcache0 stopped root@bionic-bcache:~# lsblk -e 252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop loop1 7:1 0 1G 0 loop └─fake-loop1 253:1 0 1024M 0 dm └─bcache1 251:128 0 1024M 0 disk loop2 7:2 0 1G 0 loop └─fake-loop2 253:2 0 1024M 0 dm └─bcache1 251:128 0 1024M 0 disk fake-loop0 253:0 0 1G 0 dm bcache0 is removed, bcache1 is still avaiable.