under heavy IO encounter TOCTOU bug with bcache removal
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
curtin |
Fix Released
|
Medium
|
Ryan Harper | ||
curtin (Ubuntu) |
Fix Released
|
Medium
|
Unassigned |
Bug Description
Here we see a check if a path exists, but when we attempt to remove it; it's gone (as the bcache device has been removed).
We need to handle the kernel async removal and not fail if something that should be removed is removed.
[ 26.387965] cloud-init[1430]: Current device storage tree:
[ 26.388370] cloud-init[1430]: vdb
[ 26.389001] cloud-init[1430]: |-- vdb1
[ 26.389441] cloud-init[1430]: `-- vdb2
[ 26.389930] cloud-init[1430]: `-- bcache0
[ 26.390170] cloud-init[1430]: vdc
[ 26.400560] cloud-init[1430]: `-- bcache0
[ 26.400881] cloud-init[1430]: shutdown running on holder type: 'bcache' syspath: '/sys/class/
[ 26.401256] cloud-init[1430]: stopping bcache cacheset at: /sys/fs/
[ 26.403018] cloud-init[1430]: waiting for /sys/fs/
[ 26.404351] cloud-init[1430]: /sys/fs/
[ 26.406967] cloud-init[1430]: Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
[ 26.407310] cloud-init[1430]: os.path.exists on blockdevs:
[ 26.407667] cloud-init[1430]: [('/sys/
[ 26.407996] cloud-init[1430]: stopping bcache backing device at: /sys/class/
[ 26.413071] cloud-init[1430]: finish: cmd-install/
[ 26.418728] cloud-init[1430]: finish: cmd-install/
[ 26.421744] cloud-init[1430]: Traceback (most recent call last):
[ 26.423874] cloud-init[1430]: File "/curtin/
[ 26.424290] cloud-init[1430]: ret = args.func(args)
[ 26.424635] cloud-init[1430]: File "/curtin/
[ 26.425253] cloud-init[1430]: meta_custom(args)
[ 26.425632] cloud-init[1430]: File "/curtin/
[ 26.425925] cloud-init[1430]: clear_holders.
[ 26.428596] cloud-init[1430]: File "/curtin/
[ 26.428945] cloud-init[1430]: shutdown_
[ 26.430698] cloud-init[1430]: File "/curtin/
[ 26.431769] cloud-init[1430]: '1', mode=None)
[ 26.432781] cloud-init[1430]: File "/curtin/
[ 26.433855] cloud-init[1430]: with open(filename, omode) as fp:
[ 26.435271] cloud-init[1430]: FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/
[ 26.436969] cloud-init[1430]: [Errno 2] No such file or directory: '/sys/class/
[ 26.463377] cloud-init[1430]: builtin command failed
[ 26.468229] cloud-init[1430]: finish: cmd-install/
[ 26.468534] cloud-init[1430]: builtin took 1.020 seconds
Related branches
- Server Team CI bot: Approve (continuous-integration)
- Chad Smith: Approve
- Scott Moser (community): Needs Information
-
Diff: 150 lines (+100/-4)2 files modifiedcurtin/block/clear_holders.py (+18/-4)
tests/unittests/test_clear_holders.py (+82/-0)
- curtin developers: Pending requested
-
Diff: 1202 lines (+350/-270)36 files modifiedcurtin/block/__init__.py (+11/-5)
curtin/block/clear_holders.py (+33/-4)
curtin/commands/block_meta.py (+23/-20)
curtin/commands/block_wipe.py (+4/-0)
curtin/commands/clear_holders.py (+1/-1)
debian/changelog (+17/-0)
examples/tests/bcache-wipe-xfs.yaml (+74/-0)
examples/tests/uefi_basic.yaml (+23/-0)
tests/unittests/test_clear_holders.py (+82/-0)
tests/unittests/test_commands_block_meta.py (+11/-15)
tests/vmtests/__init__.py (+14/-15)
tests/vmtests/releases.py (+0/-11)
tests/vmtests/test_basic.py (+0/-48)
tests/vmtests/test_bcache_basic.py (+0/-4)
tests/vmtests/test_bcache_bug1718699.py (+21/-0)
tests/vmtests/test_iscsi.py (+0/-4)
tests/vmtests/test_lvm.py (+0/-8)
tests/vmtests/test_lvm_iscsi.py (+0/-4)
tests/vmtests/test_mdadm_iscsi.py (+0/-4)
tests/vmtests/test_network.py (+0/-5)
tests/vmtests/test_network_alias.py (+0/-5)
tests/vmtests/test_network_bonding.py (+0/-8)
tests/vmtests/test_network_enisource.py (+0/-6)
tests/vmtests/test_network_ipv6.py (+0/-5)
tests/vmtests/test_network_ipv6_enisource.py (+0/-7)
tests/vmtests/test_network_ipv6_static.py (+0/-5)
tests/vmtests/test_network_ipv6_vlan.py (+0/-16)
tests/vmtests/test_network_mtu.py (+1/-6)
tests/vmtests/test_network_static.py (+0/-6)
tests/vmtests/test_network_static_routes.py (+0/-6)
tests/vmtests/test_network_vlan.py (+0/-16)
tests/vmtests/test_nvme.py (+0/-5)
tests/vmtests/test_raid5_bcache.py (+0/-5)
tests/vmtests/test_uefi_basic.py (+1/-15)
tools/launch (+10/-10)
tools/xkvm (+24/-1)
Changed in curtin: | |
status: | New → Confirmed |
importance: | Undecided → Medium |
status: | Confirmed → In Progress |
assignee: | nobody → Ryan Harper (raharper) |
This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New
Thank you.