curtin fails to open/erase disk with existing MDADM RAID configuration when deploying Windows images only

[Trent Lloyd]

Windows image installation fails during curtin if the disks have an existing MDADM RAID configuration leftover on-disk.

Deployment does not fail when deploying Ubuntu 16.04 - despite seemingly the same deployment kernel/initrd and same curtin version in use.

Reproduced on MAAS 2.4.0 (6981-g011e51b7a-0ubuntu1~18.04.1)

Curtin in all cases claims to be curtin: Installation started. (18.1-17-gae48e86f-0ubuntu1~18.04.1) - the observed kernel booting is ga-16.04/daily in both cases also.

A quick erase on MAAS 2.4 will successfully clear the superblock and allow the installation to proceed. However apparently this wasn't always the case and was fixed previously (LP #1708052) so unclear (and untested) if this works on MAAS 2.3 and earlier.

In my case (2.4) curtin does not know the possible users of the device, however in a customer case (2.0-2.3 as yet unknown version on 16.04) it does know the users but still otherwise fails with the same error. See error outputs below for examples of both cases.

 - The issue will occur whether sda, sdb or both drives have an mdadm superblock - even when windows isn't using sdb and sdb is not configured with any partitions etc. (Just has the leftover mdadm info)
 - The issue occurs regardless of the current storage layout in MAAS (though installing windows seems to just use a flat layout regardless of the config)

====
Error log from 2.4
====

curtin: Installation started. (18.1-17-gae48e86f-0ubuntu1~18.04.1)
Failed to exclusively open path: /dev/sda
Device holders with exclusive access: []
Device mounts: []
Possible users of /dev/sda:
None
[Errno 16] Device or resource busy: '/dev/sda'
curtin: Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'block-meta', 'custom']
Exit code: 3
Reason: -
Stdout: Failed to exclusively open path: /dev/sda
        Device holders with exclusive access: []
        Device mounts: []
        Possible users of /dev/sda:
        None
        [Errno 16] Device or resource busy: '/dev/sda'
Stderr: ''

====
Error log from customer 2.0-2.3
====
curtin: Installation started. (18.1-17-gae48e86f-0ubuntu1~16.04.1)
Failed to exclusively open path: /dev/sdbF
Device holders with exclusive access: ['md126', 'md127']D
Device mounts: []D
Possible users of /dev/sdb:P
NoneN
[Errno 16] Device or resource busy: '/dev/sdb'[
curtin: Installation failed with exception: Unexpected error while running command.c
Command: ['curtin', 'block-meta', 'custom']C
Exit code: 3E
Reason: -R
Stdout: Failed to exclusively open path: /dev/sdbS
Device holders with exclusive access: ['md126', 'md127']
Device mounts: []
Possible users of /dev/sdb:
None
[Errno 16] Device or resource busy: '/dev/sdb'

Stderr: ''S

[Andres Rodriguez (andreserl)]

Hi Trent,

Could you please attach the curtin configuration and the full log?

Also, I think this may be an issue when curtin is dealing with DD images in general and not correctly cleaning up the disk since MAAS should be sending the right configuration that includes deleting the super block.

[Trent Lloyd (lathiat)]

So it looks like wipe-superblock is specified, but it never does the other mdadm code which tears the array down. So what happens is it fails to wipe the device, because mdadm has it open.

Attaching install.log with "showtrace: true" and "verbosity:3 " from both Windows and Ubuntu 16.04 install

plus /curtin/config for both

If that isn't the right set of configs and logs, please give/point me to specific instructions for how to generate them as I am not sure outside of the above.

[Ryan Harper (raharper)]

In the windows/dd-image path while maas sends a storage config it also sends a dd-image which means curtin runs the meta-simple path. Meta-simple does not run the clear-holders as meta-simple is a no-custom-storage path.

Clear-holders needs a storage config (maas is sending one) so it can know which disks (and how to clear them w.r.t superblock wiping, or preserving). We'll need to see how to have the meta-simple path clear the disks like we do under a custom storage path, if a storage config is sent.

[Mike Pontillo (mpontillo)]

Thanks @rharper. Given that MAAS is correctly specifying 'wipe: superblock', I'll mark this Invalid for MAAS. If anything is needed from the MAAS side to fix this, we'll reopen it.

[Server Team CI bot (server-team-bot)]

This bug is fixed with commit 1ef6dfc5 to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=1ef6dfc5

[Ryan Harper (raharper)]

This bug is believed to be fixed in curtin in version 18.2. If this is still a problem for you, please make a comment and set the state back to New

Thank you.