curtin mishandles removing dm-multipath partition removal
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu on IBM z Systems |
Fix Released
|
Critical
|
Canonical Server | ||
curtin |
Fix Released
|
Undecided
|
Ryan Harper | ||
subiquity |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
With a closer look at the install we discovered that curtin
was attempting to clear a partition which was still controlled
by device-mapper; this prevents writes to the underlying partition.
Curtin will need to look up a partition by it's parent device,
discover the multipath-id of the parent, construct the correct
device-mapper DM_NAME value, and examine the output from
dmsetup ls to map the DM_NAME to the device-mapper device (dm-XX).
While working on this bug, we discovered a subiquity bug,
LP: #1860444
-- Original Description --
While trying Ubuntu focal ISO image with timestamp Dec 19 that includes subiquity 19.12.2+
2019-12-19 16:17:15,732 WARNING curtin:1242 Extract storage config does not validate.
2019-12-19 16:17:15,734 ERROR block-discover:115 block probing failed restricted=True
Traceback (most recent call last):
File "/snap/
await asyncio.
File "/snap/
return fut.result()
File "/snap/
self.
File "/snap/
self.reset()
File "/snap/
config = storage_
File "/snap/
tree = get_config_
File "/snap/
for dep in find_item_
File "/snap/
_validate_
File "/snap/
'Invalid dep_id (%s) not in storage config' % dep_id)
ValueError: Invalid dep_id (disk-sda) not in storage config
2019-12-19 16:17:15,741 INFO subiquity.core:193 saving crash report 'block probing crashed with ValueError' to /var/crash/
I did two installations that failed this way.
I've attached two sets of logs/crash files.
Related branches
- Server Team CI bot: Needs Fixing (continuous-integration)
- Dan Watkins (community): Approve
-
Diff: 16912 lines (+12842/-697)95 files modifiedHACKING.rst (+3/-4)
Makefile (+5/-1)
curtin/__init__.py (+1/-1)
curtin/block/__init__.py (+50/-6)
curtin/block/clear_holders.py (+88/-21)
curtin/block/deps.py (+1/-1)
curtin/block/multipath.py (+74/-2)
curtin/block/schemas.py (+8/-2)
curtin/block/zfs.py (+24/-0)
curtin/commands/block_meta.py (+138/-55)
curtin/commands/curthooks.py (+229/-53)
curtin/commands/install.py (+5/-1)
curtin/deps/__init__.py (+1/-1)
curtin/distro.py (+25/-2)
curtin/net/deps.py (+3/-3)
curtin/storage_config.py (+108/-22)
curtin/udev.py (+15/-7)
curtin/util.py (+23/-8)
debian/changelog (+81/-0)
debian/control (+1/-1)
doc/topics/config.rst (+6/-1)
doc/topics/integration-testing.rst (+13/-6)
doc/topics/storage.rst (+17/-2)
examples/tests/basic-dasd.yaml (+2/-2)
examples/tests/bcache-ceph-nvme-simple.yaml (+107/-0)
examples/tests/bcache-wipe-xfs.yaml (+4/-3)
examples/tests/bridging_network_v2.yaml (+38/-0)
examples/tests/dirty_disks_config.yaml (+9/-1)
examples/tests/network_mtu_networkd.yaml (+93/-0)
examples/tests/reuse-raid-member-partition.yaml (+73/-0)
examples/tests/reuse-raid-member-wipe.yaml (+71/-0)
helpers/common (+84/-13)
tests/data/probert_storage_multipath.json (+1059/-0)
tests/data/probert_storage_raid1_partitions.json (+951/-0)
tests/data/probert_storage_zlp6.json (+6980/-0)
tests/unittests/test_block.py (+59/-0)
tests/unittests/test_block_dasd.py (+1/-1)
tests/unittests/test_block_multipath.py (+87/-6)
tests/unittests/test_block_zfs.py (+26/-0)
tests/unittests/test_clear_holders.py (+131/-11)
tests/unittests/test_commands_block_meta.py (+550/-35)
tests/unittests/test_commands_collect_logs.py (+3/-1)
tests/unittests/test_curthooks.py (+338/-31)
tests/unittests/test_distro.py (+71/-0)
tests/unittests/test_storage_config.py (+152/-10)
tests/unittests/test_udev.py (+15/-14)
tests/unittests/test_util.py (+120/-1)
tests/vmtests/__init__.py (+244/-34)
tests/vmtests/helpers.py (+11/-6)
tests/vmtests/image_sync.py (+10/-3)
tests/vmtests/releases.py (+34/-0)
tests/vmtests/test_apt_config_cmd.py (+1/-8)
tests/vmtests/test_basic.py (+69/-89)
tests/vmtests/test_basic_dasd.py (+7/-16)
tests/vmtests/test_bcache_basic.py (+4/-3)
tests/vmtests/test_bcache_bug1718699.py (+4/-3)
tests/vmtests/test_bcache_ceph.py (+30/-3)
tests/vmtests/test_bcache_partitions.py (+4/-3)
tests/vmtests/test_fs_battery.py (+4/-3)
tests/vmtests/test_iscsi.py (+5/-3)
tests/vmtests/test_journald_reporter.py (+4/-7)
tests/vmtests/test_lvm.py (+14/-21)
tests/vmtests/test_lvm_iscsi.py (+4/-3)
tests/vmtests/test_lvm_raid.py (+3/-3)
tests/vmtests/test_lvm_root.py (+16/-1)
tests/vmtests/test_mdadm_bcache.py (+86/-40)
tests/vmtests/test_mdadm_iscsi.py (+4/-3)
tests/vmtests/test_multipath.py (+5/-5)
tests/vmtests/test_network.py (+3/-3)
tests/vmtests/test_network_alias.py (+4/-3)
tests/vmtests/test_network_bonding.py (+3/-3)
tests/vmtests/test_network_bridging.py (+22/-3)
tests/vmtests/test_network_ipv6.py (+0/-4)
tests/vmtests/test_network_ipv6_static.py (+3/-3)
tests/vmtests/test_network_ipv6_vlan.py (+7/-3)
tests/vmtests/test_network_mtu.py (+41/-13)
tests/vmtests/test_network_static.py (+3/-3)
tests/vmtests/test_network_static_routes.py (+5/-5)
tests/vmtests/test_network_vlan.py (+11/-3)
tests/vmtests/test_nvme.py (+4/-7)
tests/vmtests/test_old_apt_features.py (+3/-2)
tests/vmtests/test_pollinate_useragent.py (+4/-3)
tests/vmtests/test_preserve.py (+4/-3)
tests/vmtests/test_preserve_raid.py (+4/-3)
tests/vmtests/test_raid5_bcache.py (+4/-3)
tests/vmtests/test_reuse_raid_member.py (+63/-0)
tests/vmtests/test_simple.py (+10/-10)
tests/vmtests/test_uefi_basic.py (+8/-7)
tests/vmtests/test_zfsroot.py (+20/-8)
tools/block-discover-to-config (+3/-1)
tools/jenkins-runner (+50/-0)
tools/launch (+10/-0)
tools/vmtest-filter (+12/-1)
tools/vmtest-sync-images (+15/-13)
tools/xkvm (+20/-4)
- Server Team CI bot: Needs Fixing (continuous-integration)
- Dan Watkins (community): Approve
-
Diff: 16912 lines (+12842/-697)95 files modifiedHACKING.rst (+3/-4)
Makefile (+5/-1)
curtin/__init__.py (+1/-1)
curtin/block/__init__.py (+50/-6)
curtin/block/clear_holders.py (+88/-21)
curtin/block/deps.py (+1/-1)
curtin/block/multipath.py (+74/-2)
curtin/block/schemas.py (+8/-2)
curtin/block/zfs.py (+24/-0)
curtin/commands/block_meta.py (+138/-55)
curtin/commands/curthooks.py (+229/-53)
curtin/commands/install.py (+5/-1)
curtin/deps/__init__.py (+1/-1)
curtin/distro.py (+25/-2)
curtin/net/deps.py (+3/-3)
curtin/storage_config.py (+108/-22)
curtin/udev.py (+15/-7)
curtin/util.py (+23/-8)
debian/changelog (+81/-0)
debian/control (+1/-1)
doc/topics/config.rst (+6/-1)
doc/topics/integration-testing.rst (+13/-6)
doc/topics/storage.rst (+17/-2)
examples/tests/basic-dasd.yaml (+2/-2)
examples/tests/bcache-ceph-nvme-simple.yaml (+107/-0)
examples/tests/bcache-wipe-xfs.yaml (+4/-3)
examples/tests/bridging_network_v2.yaml (+38/-0)
examples/tests/dirty_disks_config.yaml (+9/-1)
examples/tests/network_mtu_networkd.yaml (+93/-0)
examples/tests/reuse-raid-member-partition.yaml (+73/-0)
examples/tests/reuse-raid-member-wipe.yaml (+71/-0)
helpers/common (+84/-13)
tests/data/probert_storage_multipath.json (+1059/-0)
tests/data/probert_storage_raid1_partitions.json (+951/-0)
tests/data/probert_storage_zlp6.json (+6980/-0)
tests/unittests/test_block.py (+59/-0)
tests/unittests/test_block_dasd.py (+1/-1)
tests/unittests/test_block_multipath.py (+87/-6)
tests/unittests/test_block_zfs.py (+26/-0)
tests/unittests/test_clear_holders.py (+131/-11)
tests/unittests/test_commands_block_meta.py (+550/-35)
tests/unittests/test_commands_collect_logs.py (+3/-1)
tests/unittests/test_curthooks.py (+338/-31)
tests/unittests/test_distro.py (+71/-0)
tests/unittests/test_storage_config.py (+152/-10)
tests/unittests/test_udev.py (+15/-14)
tests/unittests/test_util.py (+120/-1)
tests/vmtests/__init__.py (+244/-34)
tests/vmtests/helpers.py (+11/-6)
tests/vmtests/image_sync.py (+10/-3)
tests/vmtests/releases.py (+34/-0)
tests/vmtests/test_apt_config_cmd.py (+1/-8)
tests/vmtests/test_basic.py (+69/-89)
tests/vmtests/test_basic_dasd.py (+7/-16)
tests/vmtests/test_bcache_basic.py (+4/-3)
tests/vmtests/test_bcache_bug1718699.py (+4/-3)
tests/vmtests/test_bcache_ceph.py (+30/-3)
tests/vmtests/test_bcache_partitions.py (+4/-3)
tests/vmtests/test_fs_battery.py (+4/-3)
tests/vmtests/test_iscsi.py (+5/-3)
tests/vmtests/test_journald_reporter.py (+4/-7)
tests/vmtests/test_lvm.py (+14/-21)
tests/vmtests/test_lvm_iscsi.py (+4/-3)
tests/vmtests/test_lvm_raid.py (+3/-3)
tests/vmtests/test_lvm_root.py (+16/-1)
tests/vmtests/test_mdadm_bcache.py (+86/-40)
tests/vmtests/test_mdadm_iscsi.py (+4/-3)
tests/vmtests/test_multipath.py (+5/-5)
tests/vmtests/test_network.py (+3/-3)
tests/vmtests/test_network_alias.py (+4/-3)
tests/vmtests/test_network_bonding.py (+3/-3)
tests/vmtests/test_network_bridging.py (+22/-3)
tests/vmtests/test_network_ipv6.py (+0/-4)
tests/vmtests/test_network_ipv6_static.py (+3/-3)
tests/vmtests/test_network_ipv6_vlan.py (+7/-3)
tests/vmtests/test_network_mtu.py (+41/-13)
tests/vmtests/test_network_static.py (+3/-3)
tests/vmtests/test_network_static_routes.py (+5/-5)
tests/vmtests/test_network_vlan.py (+11/-3)
tests/vmtests/test_nvme.py (+4/-7)
tests/vmtests/test_old_apt_features.py (+3/-2)
tests/vmtests/test_pollinate_useragent.py (+4/-3)
tests/vmtests/test_preserve.py (+4/-3)
tests/vmtests/test_preserve_raid.py (+4/-3)
tests/vmtests/test_raid5_bcache.py (+4/-3)
tests/vmtests/test_reuse_raid_member.py (+63/-0)
tests/vmtests/test_simple.py (+10/-10)
tests/vmtests/test_uefi_basic.py (+8/-7)
tests/vmtests/test_zfsroot.py (+20/-8)
tools/block-discover-to-config (+3/-1)
tools/jenkins-runner (+50/-0)
tools/launch (+10/-0)
tools/vmtest-filter (+12/-1)
tools/vmtest-sync-images (+15/-13)
tools/xkvm (+20/-4)
- Server Team CI bot: Needs Fixing (continuous-integration)
- Dan Watkins (community): Approve
-
Diff: 15603 lines (+12330/-488)87 files modifiedHACKING.rst (+3/-4)
curtin/__init__.py (+1/-1)
curtin/block/clear_holders.py (+88/-21)
curtin/block/multipath.py (+74/-2)
curtin/block/schemas.py (+7/-1)
curtin/block/zfs.py (+24/-0)
curtin/commands/block_meta.py (+99/-46)
curtin/commands/curthooks.py (+200/-53)
curtin/commands/install.py (+5/-1)
curtin/distro.py (+25/-2)
curtin/net/deps.py (+3/-3)
curtin/storage_config.py (+105/-20)
curtin/udev.py (+15/-7)
curtin/util.py (+12/-5)
debian/changelog (+63/-0)
debian/control (+1/-1)
doc/topics/config.rst (+6/-1)
doc/topics/storage.rst (+17/-2)
examples/tests/basic-dasd.yaml (+2/-2)
examples/tests/bcache-ceph-nvme-simple.yaml (+107/-0)
examples/tests/bcache-wipe-xfs.yaml (+4/-3)
examples/tests/bridging_network_v2.yaml (+38/-0)
examples/tests/dirty_disks_config.yaml (+9/-1)
examples/tests/network_mtu_networkd.yaml (+93/-0)
examples/tests/reuse-raid-member-partition.yaml (+73/-0)
examples/tests/reuse-raid-member-wipe.yaml (+71/-0)
helpers/common (+84/-13)
tests/data/probert_storage_multipath.json (+1059/-0)
tests/data/probert_storage_raid1_partitions.json (+951/-0)
tests/data/probert_storage_zlp6.json (+6980/-0)
tests/unittests/test_block.py (+26/-0)
tests/unittests/test_block_dasd.py (+1/-1)
tests/unittests/test_block_multipath.py (+87/-6)
tests/unittests/test_block_zfs.py (+26/-0)
tests/unittests/test_clear_holders.py (+131/-11)
tests/unittests/test_commands_block_meta.py (+377/-36)
tests/unittests/test_commands_collect_logs.py (+3/-1)
tests/unittests/test_curthooks.py (+311/-26)
tests/unittests/test_distro.py (+71/-0)
tests/unittests/test_storage_config.py (+114/-9)
tests/unittests/test_udev.py (+15/-14)
tests/unittests/test_util.py (+41/-1)
tests/vmtests/__init__.py (+231/-23)
tests/vmtests/helpers.py (+11/-6)
tests/vmtests/releases.py (+34/-0)
tests/vmtests/test_apt_config_cmd.py (+1/-4)
tests/vmtests/test_basic.py (+71/-81)
tests/vmtests/test_basic_dasd.py (+8/-10)
tests/vmtests/test_bcache_basic.py (+5/-0)
tests/vmtests/test_bcache_bug1718699.py (+5/-0)
tests/vmtests/test_bcache_ceph.py (+31/-0)
tests/vmtests/test_bcache_partitions.py (+5/-0)
tests/vmtests/test_fs_battery.py (+5/-0)
tests/vmtests/test_iscsi.py (+6/-0)
tests/vmtests/test_journald_reporter.py (+5/-4)
tests/vmtests/test_lvm.py (+15/-18)
tests/vmtests/test_lvm_iscsi.py (+5/-0)
tests/vmtests/test_lvm_raid.py (+4/-0)
tests/vmtests/test_lvm_root.py (+16/-1)
tests/vmtests/test_mdadm_bcache.py (+92/-12)
tests/vmtests/test_mdadm_iscsi.py (+5/-0)
tests/vmtests/test_multipath.py (+6/-0)
tests/vmtests/test_network.py (+4/-0)
tests/vmtests/test_network_alias.py (+5/-0)
tests/vmtests/test_network_bonding.py (+4/-0)
tests/vmtests/test_network_bridging.py (+23/-0)
tests/vmtests/test_network_ipv6_static.py (+4/-0)
tests/vmtests/test_network_ipv6_vlan.py (+8/-0)
tests/vmtests/test_network_mtu.py (+42/-10)
tests/vmtests/test_network_static.py (+4/-0)
tests/vmtests/test_network_static_routes.py (+5/-0)
tests/vmtests/test_network_vlan.py (+12/-0)
tests/vmtests/test_nvme.py (+5/-0)
tests/vmtests/test_old_apt_features.py (+3/-2)
tests/vmtests/test_pollinate_useragent.py (+5/-0)
tests/vmtests/test_preserve.py (+5/-0)
tests/vmtests/test_preserve_raid.py (+5/-0)
tests/vmtests/test_raid5_bcache.py (+5/-0)
tests/vmtests/test_reuse_raid_member.py (+63/-0)
tests/vmtests/test_simple.py (+16/-1)
tests/vmtests/test_uefi_basic.py (+11/-1)
tests/vmtests/test_zfsroot.py (+18/-2)
tools/block-discover-to-config (+3/-1)
tools/jenkins-runner (+50/-0)
tools/vmtest-filter (+12/-1)
tools/vmtest-sync-images (+15/-13)
tools/xkvm (+20/-4)
- Dan Watkins (community): Approve
- Server Team CI bot: Approve (continuous-integration)
-
Diff: 640 lines (+236/-25)6 files modifiedcurtin/block/clear_holders.py (+32/-11)
curtin/block/multipath.py (+74/-2)
examples/tests/dirty_disks_config.yaml (+8/-0)
tests/unittests/test_block_multipath.py (+87/-6)
tests/unittests/test_clear_holders.py (+34/-6)
tests/vmtests/test_multipath.py (+1/-0)
Changed in ubuntu-z-systems: | |
importance: | High → Critical |
Changed in curtin: | |
status: | Incomplete → New |
Changed in ubuntu-z-systems: | |
assignee: | nobody → Canonical Server Team (canonical-server) |
Changed in ubuntu-z-systems: | |
status: | New → In Progress |
Changed in curtin: | |
status: | New → In Progress |
summary: |
- subiquity on s390x fails with curtin: block-discover probing + curtin mishandles removing dm-multipath partition removal |
description: | updated |
Changed in curtin: | |
assignee: | nobody → Ryan Harper (raharper) |
Changed in ubuntu-z-systems: | |
status: | In Progress → Fix Committed |
I think the underlying issue here is that curtin failed during block-meta due to /dev/sda1 being busy:
finish: cmd-install/ stage-partition ing/builtin/ cmd-block- meta/clear- holders: FAIL: removing previous storage devices stage-partition ing/builtin/ cmd-block- meta: FAIL: curtin command block-meta subiquity/ 1413/lib/ python3. 6/site- packages/ curtin/ commands/ main.py" , line 202, in main subiquity/ 1413/lib/ python3. 6/site- packages/ curtin/ log.py" , line 97, in wrapper subiquity/ 1413/lib/ python3. 6/site- packages/ curtin/ log.py" , line 79, in log_time subiquity/ 1413/lib/ python3. 6/site- packages/ curtin/ commands/ block_meta. py", line 79, in block_meta clear(devices, state.get( 'report_ stack_prefix' , '')) subiquity/ 1413/lib/ python3. 6/site- packages/ curtin/ commands/ block_meta. py", line 1662, in meta_clear holders. clear_holders( devices) subiquity/ 1413/lib/ python3. 6/site- packages/ curtin/ block/clear_ holders. py", line 624, in clear_holders function( dev_info[ 'device' ]) subiquity/ 1413/lib/ python3. 6/site- packages/ curtin/ block/clear_ holders. py", line 288, in wipe_superblock superblock( blockdev) subiquity/ 1413/lib/ python3. 6/site- packages/ curtin/ block/clear_ holders. py", line 333, in _wipe_superblock exclusive, strict=strict) subiquity/ 1413/lib/ python3. 6/site- packages/ curtin/ block/_ _init__ .py", line 1127, in wipe_volume zero(path, partitions=False, exclusive= exclusive, strict=strict) subiquity/ 1413/lib/ python3. 6/site- packages/ curtin/ block/_ _init__ .py", line 1041, in quick_zero exclusive, strict=strict) subiquity/ 1413/lib/ python3. 6/site- packages/ curtin/ block/_ _init__ .py", line 1061, in zero_file_ at_offsets open(path, exclusive= exclusive) as fp: subiquity/ 1413/usr/ lib/python3. 6/contextlib. py", line 81, in __enter__ subiquity/ 1413/lib/ python3. 6/site- packages/ curtin/ block/_ _init__ .py", line 954, in exclusive_open
TIMED BLOCK_META: 10.963
finish: cmd-install/
Traceback (most recent call last):
File "/snap/
ret = args.func(args)
File "/snap/
return log_time("TIMED %s: " % msg, func, *args, **kwargs)
File "/snap/
return func(*args, **kwargs)
File "/snap/
meta_
File "/snap/
clear_
File "/snap/
shutdown_
File "/snap/
_wipe_
File "/snap/
exclusive=
File "/snap/
quick_
File "/snap/
exclusive=
File "/snap/
with exclusive_
File "/snap/
return next(self.gen)
File "/snap/
fd = os.open(path, flags)
OSError: [Errno 16] Device or resource busy: '/dev/sda1'
[Errno 16] Device or resource busy: '/dev/sda1'
And I think that curtin thinks that /dev/sda1 shouldn't be busy because fuser returns non-zero and so doesn't report any users:
wiping /dev/sda1 attempt 4/4
wiping 1M on /dev/sda1 at offsets [0, -1048576]
Failed to exclusively open path: /dev/sda1
devname '/dev/sda1' had holders: []
Device holders with exclusive access: []
Device mounts: []
Running command ['fuser', '--verbose', '--mount', '/dev/sda1'] with allowed return codes [0] (capture=True)
fuser returned non-zero: ''
Possible users of /dev/sda1:
None
Playing around locally, fuser emits its error messages on stderr, so perhaps we're not capturing stderr here to get the full error message? I'm unsure.