under heavy IO encounter TOCTOU bug with bcache removal

Bug #1700564 reported by Ryan Harper
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
curtin
Medium
Ryan Harper
curtin (Ubuntu)
Medium
Unassigned

Bug Description

Here we see a check if a path exists, but when we attempt to remove it; it's gone (as the bcache device has been removed).

We need to handle the kernel async removal and not fail if something that should be removed is removed.

[ 26.387965] cloud-init[1430]: Current device storage tree:
[ 26.388370] cloud-init[1430]: vdb
[ 26.389001] cloud-init[1430]: |-- vdb1
[ 26.389441] cloud-init[1430]: `-- vdb2
[ 26.389930] cloud-init[1430]: `-- bcache0
[ 26.390170] cloud-init[1430]: vdc
[ 26.400560] cloud-init[1430]: `-- bcache0
[ 26.400881] cloud-init[1430]: shutdown running on holder type: 'bcache' syspath: '/sys/class/block/bcache0'
[ 26.401256] cloud-init[1430]: stopping bcache cacheset at: /sys/fs/bcache/661527fd-09b0-48e5-987d-1175ce570a76
[ 26.403018] cloud-init[1430]: waiting for /sys/fs/bcache/661527fd-09b0-48e5-987d-1175ce570a76 to be removed
[ 26.404351] cloud-init[1430]: /sys/fs/bcache/661527fd-09b0-48e5-987d-1175ce570a76 has been removed
[ 26.406967] cloud-init[1430]: Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
[ 26.407310] cloud-init[1430]: os.path.exists on blockdevs:
[ 26.407667] cloud-init[1430]: [('/sys/class/block/bcache0', True), ('/sys/class/block/vdc/bcache', True), ('/sys/class/block/vdb/vdb2/bcache', False)]
[ 26.407996] cloud-init[1430]: stopping bcache backing device at: /sys/class/block/bcache0/bcache
[ 26.413071] cloud-init[1430]: finish: cmd-install/stage-partitioning/builtin/cmd-block-meta: FAIL: removing previous storage devices
[ 26.418728] cloud-init[1430]: finish: cmd-install/stage-partitioning/builtin/cmd-block-meta: FAIL: curtin command block-meta
[ 26.421744] cloud-init[1430]: Traceback (most recent call last):
[ 26.423874] cloud-init[1430]: File "/curtin/curtin/commands/main.py", line 215, in main
[ 26.424290] cloud-init[1430]: ret = args.func(args)
[ 26.424635] cloud-init[1430]: File "/curtin/curtin/commands/block_meta.py", line 67, in block_meta
[ 26.425253] cloud-init[1430]: meta_custom(args)
[ 26.425632] cloud-init[1430]: File "/curtin/curtin/commands/block_meta.py", line 1186, in meta_custom
[ 26.425925] cloud-init[1430]: clear_holders.clear_holders(disk_paths)
[ 26.428596] cloud-init[1430]: File "/curtin/curtin/block/clear_holders.py", line 481, in clear_holders
[ 26.428945] cloud-init[1430]: shutdown_function(dev_info['device'])
[ 26.430698] cloud-init[1430]: File "/curtin/curtin/block/clear_holders.py", line 166, in shutdown_bcache
[ 26.431769] cloud-init[1430]: '1', mode=None)
[ 26.432781] cloud-init[1430]: File "/curtin/curtin/util.py", line 435, in write_file
[ 26.433855] cloud-init[1430]: with open(filename, omode) as fp:
[ 26.435271] cloud-init[1430]: FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/block/bcache0/bcache/stop'
[ 26.436969] cloud-init[1430]: [Errno 2] No such file or directory: '/sys/class/block/bcache0/bcache/stop'
[ 26.463377] cloud-init[1430]: builtin command failed
[ 26.468229] cloud-init[1430]: finish: cmd-install/stage-partitioning/builtin: FAIL: running 'curtin block-meta simple'
[ 26.468534] cloud-init[1430]: builtin took 1.020 seconds

Related branches

Scott Moser (smoser)
Changed in curtin:
status: New → Confirmed
importance: Undecided → Medium
status: Confirmed → In Progress
assignee: nobody → Ryan Harper (raharper)
Revision history for this message
Scott Moser (smoser) wrote : Fixed in Curtin 17.1

This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in curtin:
status: In Progress → Fix Released
Revision history for this message
Scott Moser (smoser) wrote :

A fix for this bug is being SRU to Ubuntu 16.04 and 17.10 under bug 1743618.

Changed in curtin (Ubuntu):
status: New → Fix Released
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers