subiquity on s390x doesn't handle pristine DASD disks that need an initial low-level format

[Frank Heimes]

In case a DASD (ECKD) disk is used for the very first time, it needs to be initially low-level formatted with dasdfmt (this is needed only once, at the very first usage).

I found a DASD disk in my environment that was not yet used before (262f) and the installer should have identified that and should have either called dasdfmt automatically or should have provided an option in the menu for doing the low-level format.

But I couldn't see any of the two options, instead I saw a crash saying that mke2fs failed - presumably because the disk was not low-level formatted before and does not have a partition with vtoc on top:

 mkfs /dev/dasda info: {'fstype': 'ext4', 'volume': 'disk-dasda', 'preserve': False, 'type': 'format', 'id': 'format-0'}
 Running command ['lsblk', '--noheadings', '--bytes', '--pairs', '--output=ALIGNMENT,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,FSTYPE,GROUP,KNAME,LABEL,LOG-SEC,MAJ:MIN,MIN-IO,MODE,MODEL,MOUNTPOINT,NAME,OPT-IO,OWNER,PHY-SEC,RM,RO,ROTA,RQ-SIZE,SIZE,STATE,TYPE,UUID', '/dev/dasda'] with allowed return codes [0] (capture=True)
 get_blockdev_sector_size: (log=512
 , phys=512
 )
 Running command ['mkfs.ext4', '-F', '-U', 'bd2d8254-9273-4f64-b6e6-d6c89ce0a1c5', '/dev/dasda'] with allowed return codes [0] (capture=True)
 An error occured handling 'format-0': ProcessExecutionError - Unexpected error while running command.
 Command: ['mkfs.ext4', '-F', '-U', 'bd2d8254-9273-4f64-b6e6-d6c89ce0a1c5', '/dev/dasda']
 Exit code: 1
 Reason: -
 Stdout: ''
 Stderr: mke2fs 1.45.5 (07-Jan-2020)
         mkfs.ext4: Device size reported to be zero. Invalid partition specified, or
                partition table wasn't reread after running fdisk, due to
                a modified partition being busy and in use. You may need to reboot
                to re-read your partition table.

 finish: cmd-install/stage-partitioning/builtin/cmd-block-meta: FAIL: configuring format: format-0
 TIMED BLOCK_META: 1.794
 finish: cmd-install/stage-partitioning/builtin/cmd-block-meta: FAIL: curtin command block-meta
 Traceback (most recent call last):
   File "/snap/subiquity/1946/lib/python3.6/site-packages/curtin/commands/main.py", line 202, in main
     ret = args.func(args)
   File "/snap/subiquity/1946/lib/python3.6/site-packages/curtin/log.py", line 97, in wrapper
     return log_time("TIMED %s: " % msg, func, *args, **kwargs)
   File "/snap/subiquity/1946/lib/python3.6/site-packages/curtin/log.py", line 79, in log_time
     return func(*args, **kwargs)
   File "/snap/subiquity/1946/lib/python3.6/site-packages/curtin/commands/block_meta.py", line 111, in block_meta
     return meta_custom(args)
   File "/snap/subiquity/1946/lib/python3.6/site-packages/curtin/commands/block_meta.py", line 1911, in meta_custom
     handler(command, storage_config_dict)
   File "/snap/subiquity/1946/lib/python3.6/site-packages/curtin/commands/block_meta.py", line 988, in format_handler
     mkfs.mkfs_from_config(volume_path, info)
   File "/snap/subiquity/1946/lib/python3.6/site-packages/curtin/block/mkfs.py", line 235, in mkfs_from_config
     label=info.get('label'), extra_options=info.get('extra_options'))
   File "/snap/subiquity/1946/lib/python3.6/site-packages/curtin/block/mkfs.py", line 211, in mkfs
     util.subp(cmd, capture=True)
   File "/snap/subiquity/1946/lib/python3.6/site-packages/curtin/util.py", line 275, in subp
     return _subp(*args, **kwargs)
   File "/snap/subiquity/1946/lib/python3.6/site-packages/curtin/util.py", line 141, in _subp
     cmd=args)
 curtin.util.ProcessExecutionError: Unexpected error while running command.
 Command: ['mkfs.ext4', '-F', '-U', 'bd2d8254-9273-4f64-b6e6-d6c89ce0a1c5', '/dev/dasda']
 Exit code: 1
 Reason: -
 Stdout: ''

In the storage configuration screen:

  USED DEVICES

    DEVICE TYPE SIZE ┌─────────────────────────┐
  [ /dev/dasda local disk 0B ▸│◂ (close) │
    unused │ Info ▸│
                                        │ Reformat ▸│
                                        │ Add VTOC Partition ▸│
                                 [ Done │ Format ▸│
                                 [ Reset│ Remove from RAID/LVM │
                                 [ Back └─────────────────────────┘

'Format' was the only active option (that wasn't grayed out), but this seem to be for the Linux filesystem format only, not for the low-level dasdfmt.

Btw. I used subiquity 20.06.1+git7.317648ae (1946) - used the 20.04 GA live installer image, but updated to the latest installer version.

The attached doc shows the (relevant) steps I performed during the installation.
I'll also attach the logs ...

[Dimitri John Ledkov (xnox)]

From logs:

storage:
  config:
  - {path: /dev/dasda, wipe: superblock, preserve: false, name: '', grub_device: false,
    device_id: 0.0.262f, type: disk, id: disk-dasda}
  - {fstype: ext4, volume: disk-dasda, preserve: false, type: format, id: format-0}
  - {device: format-0, path: /, type: mount, id: mount-0}
  version: 1

I think this is missing dasd configuration, specifically the superblock wipe command should not be there, and instead there should be two `type: dasd` commands.

- id: dasd_root
  type: dasd
  device_id: 0.0.1520
  blocksize: 4096
  disk_layout: cdl
  label: 0X1520
  mode: quick
- id: disk0
  type: disk
  ptable: vtoc
  serial: 0X1520
  name: root_disk
  wipe: superblock

[Ryan Harper (raharper)]

I'm not sure there's a curtin issue here: the probe returns what's expected AFAICT:

2020-07-15 12:43:18,223 DEBUG curtin:1358 Merged storage config:
storage:
    config:
    - blocksize: 512
        device_id: 0.0.262f
        disk_layout: not-formatted
        id: dasd-dasda
        mode: full
        type: dasd
    - device_id: 0.0.262f
        id: disk-dasda
        path: /dev/dasda
        type: disk
    version: 1

wipe: superblock is not valid on type: dasd

##### Validation Errors #####
Additional properties are not allowed ('wipe' was unexpected) in examples/tests/basic-dasd.yaml
{
 "blocksize": 4096,
 "device_id": "0.0.1520",
 "disk_layout": "cdl",
 "id": "dasd_spare",
 "mode": "full",
 "type": "dasd",
 "wipe": "superblock"
}
-----------------------------

Curtin checks if the target disk matches (label, layout, blocksize) and if they don't match the config, it will issue dasdfmt command.

[Ryan Harper (raharper)]

I'm marking the curtin task invalid for now. If you find out more information indicating that curtin needs to be changed, please mark this task back to new.

[Michael Hudson-Doyle (mwhudson)]

This is a bit confusing in the UI, but it looks like you formatted and mounted the entire disk:

 2020-07-15 12:46:05,766 DEBUG subiquity.ui.filesystem.add_partition:599 Format Entire Result: {'fstype': 'ext4', 'mount': '/', 'use_swap': False}

vs putting a partition on it. Is that supported? Either we should make this work or we should grey out the option for dasds. Let me know which you want :)

[Frank Heimes (fheimes)]

Well, not sure if I understood this correctly.
It is not about the Linux filesystem format (mkfs.ext4), that would indeed require a partition.
It's about the low-level format (dasdfmt) of an ECKD DASD, that is needed prior to the very first usage - and at that point in time there cannot be a partition.
The partitioning of the ECKD DASD (with fdasd) is only possible after the low-level format with dasdfmt was done.

Not sure how the ll format with dasdfmt is handled - since it takes quite some time (depending on the disk size several minutes), a screen indicating this would be great to have / to see. Just to avoid overhasty actions of impatient users.

But since I wasn't able to get a ll going, I have no idea if that's the case or if it's just done silently ...

The logs may show me trying to get the disk ll formatted in one or the other way - but I wasn't able to achieve this...

[Pedro Principeza (pprincipeza)]

Adding a minor mention in the LP Bug, as we had a customer being affected by the issue while installing 20.04.1 on a z14, atop of z/VM 6.4, using unformatted 3390-M9s.

The workaround provided by Herr Heimes on LP Bug #1878596, Comment #21, has proven to be a workaround for the issue.

I believe we should either check for the current format of the disk on z/VM (that's not filesystem formatting, as Heimes pointed out, but rather a low-level format of the ECKD to be used) using lszdev/lsdasd then use dasdfmt, or point out in the docs we should have someone run CPFMTXA in z/VM to add CDL to the DASD, enabling its usage by the Guest.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1887669

[Michael Hudson-Doyle (mwhudson)]

Argh I think what's going on here is that the type: dasd object in subiquity is not properly being recorded as a dependency of the type: disk object, which means that it doesn't get emitted in the generated config :(

[Michael Hudson-Doyle (mwhudson)]

I've pushed a potential fix for this to the edge/mwhudson-hack branch, if you could test that it would be great.

[Michael Hudson-Doyle (mwhudson)]

Hm are this and bug 1878596 duplicates? I find that bug pretty confusing :)

[Frank Heimes (fheimes)]

@mwhudson
LP 1878596 is not exactly a duplicate, since some more issues and concerns are raised there, like:
a) lack of information/knowledge on subiquity and the differences compared to d-i (at that point in time <20.04 GA> a parmfile was required for installations)
b) the use/support of very special DASD ECKD disks (3390-ModA EAV and EAV-II DASDs)
c) and the fact that DASD ECKD disks need to be pre-formatted (as of today)
Only c) is a duplicate to this one here (LP 1887669).
Fortunately the a) and b) are sorted out - so only c) is left.

Since this bug here (LP 1887669) is exclusively about the low-level format bug of pristine DASD (ECKD) disks, I would suggest that is used as anchor for these activities.
Once it's solved/fixed we can close the other one (LP1878596), too.

[Frank Heimes (fheimes)]

@mwhudson
I re-tired with the 'edge/mwhudson-hack' branch, like mentioned in comment #11, but it doesn't seem to change much (other than the system doesn't break or crashes).
I end up with this screen:

Guided storage configuration

Block probing did not discover any disks big enough to support guided
storage configuration. Manual configuration may still be possible.

and the following storage configuration screen is just with no disk:

▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
  Storage configuration [ Help ]
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
  To continue you need to: Mount a filesystem at /

  AVAILABLE DEVICES

    No available devices

  [ Create software RAID (md) ▸ ]
  [ Create volume group (LVM) ▸ ]

  USED DEVICES

    DEVICE TYPE SIZE
  [ /dev/dasda local disk 0B ▸ ]
    unused

                       [ Done ]
                       [ Reset ]
                       [ Back ]

At that point in time lsdasd still shows the disk as "n/f" (non formatted):
lsdasd
Bus-ID Status Name Device Type BlkSz Size Blocks
================================================================================
0.0.262f n/f dasda 94:0 ECKD

For some more details please see the attached partial console and command log.

(I left the system in that stage - in case of any further analysis.)

[Frank Heimes (fheimes)]

and here is the content of /var/log and /var/crash

[Frank Heimes (fheimes)]

This bug is fixed with
https://github.com/CanonicalLtd/subiquity/releases/tag/21.01.1
for Ubuntu releases H, G and F.

I btw. tested this also with a IBM EAV (Extended Address Volume) 3390-A 180 (= 180 x Mod1).
Just keep in mind that a dasdfmt that subiquity does under the hood may take a very long time (depending on your system setup).