partitions on multipathed disk confuse extract_storage_config

Bug #1839915 reported by Michael Hudson-Doyle on 2019-08-13
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
curtin
High
Unassigned

Bug Description

probe-data-mp.json is attached:

(master)mwhudson@ringil:/opt/opensource/curtin$ python3
Python 3.6.8 (default, Jan 14 2019, 11:02:34)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, json; from curtin import log, storage_config
>>> log.basicConfig(verbosity=2, stream=sys.stdout)
>>> c = json.load(open('probe-data-mp.json'))
>>> storage_config.extract_storage_config(c)
Extracting storage config from probe data
Sorting extracted configurations
Validating extracted storage config components
Extracted (unmerged) storage config:
storage:
- id: disk-sda
    path: /dev/sda
    ptable: gpt
    serial: '30000000000000064'
    type: disk
    wwn: '0x0000000000000064'
- id: disk-sdb
    path: /dev/sdb
    ptable: gpt
    serial: '30000000000000064'
    type: disk
    wwn: '0x0000000000000064'
- device: disk-sda
    flag: bios_grub
    id: partition-sda1
    number: 1
    offset: 1048576
    size: 1048576
    type: partition
- device: disk-sda
    flag: linux
    id: partition-sda2
    number: 2
    offset: 2097152
    size: 10734272512
    type: partition
- device: disk-sdb
    flag: bios_grub
    id: partition-sdb1
    number: 1
    offset: 1048576
    size: 1048576
    type: partition
- device: disk-sdb
    flag: linux
    id: partition-sdb2
    number: 2
    offset: 2097152
    size: 10734272512
    type: partition
- fstype: ext4
    id: format-partition-sda2
    type: format
    uuid: 80ca57a0-60b7-4328-be41-afedf915effd
    volume: partition-sda2
- fstype: ext4
    id: format-partition-sdb2
    type: format
    uuid: 80ca57a0-60b7-4328-be41-afedf915effd
    volume: partition-sdb2
- fstype: ext4
    id: format-disk-dm-2
    type: format
    uuid: 80ca57a0-60b7-4328-be41-afedf915effd
    volume: disk-dm-2

Generating storage config dependencies
Validate: partition-sda1:SourceType:partition -> (DepId:disk-sda DepType:disk) in SourceDeps:{'disk', 'raid', 'bcache', 'partition'} ? result=True
Validate: partition-sda2:SourceType:partition -> (DepId:disk-sda DepType:disk) in SourceDeps:{'disk', 'raid', 'bcache', 'partition'} ? result=True
Validate: partition-sdb1:SourceType:partition -> (DepId:disk-sdb DepType:disk) in SourceDeps:{'disk', 'raid', 'bcache', 'partition'} ? result=True
Validate: partition-sdb2:SourceType:partition -> (DepId:disk-sdb DepType:disk) in SourceDeps:{'disk', 'raid', 'bcache', 'partition'} ? result=True
Validate: format-partition-sda2:SourceType:format -> (DepId:partition-sda2 DepType:partition) in SourceDeps:{'dm_crypt', 'disk', 'lvm_partition', 'bcache', 'partition', 'raid'} ? result=True
Validate: partition-sda2:SourceType:partition -> (DepId:disk-sda DepType:disk) in SourceDeps:{'disk', 'raid', 'bcache', 'partition'} ? result=True
Validate: format-partition-sdb2:SourceType:format -> (DepId:partition-sdb2 DepType:partition) in SourceDeps:{'dm_crypt', 'disk', 'lvm_partition', 'bcache', 'partition', 'raid'} ? result=True
Validate: partition-sdb2:SourceType:partition -> (DepId:disk-sdb DepType:disk) in SourceDeps:{'disk', 'raid', 'bcache', 'partition'} ? result=True
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/opensource/curtin/curtin/storage_config.py", line 1171, in extract_storage_config
    tree = get_config_tree(cfg.get('id'), final_config)
  File "/opt/opensource/curtin/curtin/storage_config.py", line 274, in get_config_tree
    for dep in find_item_dependencies(item, sconfig):
  File "/opt/opensource/curtin/curtin/storage_config.py", line 244, in find_item_dependencies
    _validate_dep_type(item_id, dep_key, dep, config)
  File "/opt/opensource/curtin/curtin/storage_config.py", line 192, in _validate_dep_type
    'Invalid dep_id (%s) not in storage config' % dep_id)
ValueError: Invalid dep_id (disk-dm-2) not in storage config
>>>

The immediate problem here I guess is that curtin hasn't figured out that dm-2 is a partition of dm-0. But I'm not really sure in general what should be happening here -- it seems to me that, in general, it would be more useful to emit one disk action with path=/dev/mapper/mpatha, wwn=30000000000000064, serial=serial-a (and have some way in info to see which devices are part of it) and have the partition etc actions refer to that, rather than having disk actions with /dev/sdX paths and duplicate serials etc. Happy to work on this if you agree (and ideally tell me where to get started!)

Related branches

Michael Hudson-Doyle (mwhudson) wrote :
Ryan Harper (raharper) wrote :

Thanks for filing the bug. I had some probert data that could reproduce as well.

One of the challenges, as you see, is that multipath "partitions" are really additional dm devices, except, they are actual partitions on the underlying disks.

so, you'll have dm2 "partition" on dm0 (mpath device) which is on top of sda which will have an actual sda2.

So curtin generally wants to ignore the mpath device itself, and deal with _one_ path.

We have a couple of options; MAAS currently dumps *all* of the disks it found in the config, and constructs a layout from just *one* of the disks since MAAS and curtin don't *construct* multipath devices (the construction is how the devices are wired up).

Subiquity can also use this method, including all of the paths, but creating partitions/devices against just a single path. Alternatively, we can exclude any of the additional paths after selecting one of the paths and curtin will detect that the underlying disk is part of a multipath and ensure that the target OS has multipath enabled/configured.

Changed in curtin:
importance: Undecided → High
status: New → Triaged
Michael Hudson-Doyle (mwhudson) wrote :

I guess given current curtin practice, it would be easiest to just do some simple filtering in subiquity to make sure that only one member of a multipath device is presented and to indicate the "multipathiness" of the device somehow. IOW what your linked branch does, modulo the oddity of partition formats being ascribed to the disk.

This bug is fixed with commit 826e89d4 to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=826e89d4

Changed in curtin:
status: Triaged → Fix Committed
Changed in curtin:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers