Can't deploy CentOS with MD-RAID

[Derek DeMoss]

Running MAAS 2.9.2 (9165-g.c3e7848d1)

Attempting to deploy a machine today, but getting these errors after the machine boots off of the local disk.

/scratch is the volume on the MD-RAID, it's formatted with ext4.

```
1 A start job is running for dev-disk-by^x2did-md xZduuidx2d17b6802c:6179f2aa:fBcba0d1:2f748d64.device (13s / 1min 30s)[
33.1831981 random: crng init done
TIME 1 Timed out waiting for device dev-disk-by^xZdid-md xZduuid»x2d17b6802c:6179fZaa:fBcbabd1:2f748d64.device.
[DEPEND] Dependency failed for /scratch.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Mark the need to relabel after reboot.
[DEPEND] Dependency failed for Relabel all filesystems, if necessary,
DEPEND] Dependency failed for Migrate local SELinux policy changes from the old store structure to the new structure.
```

[Alberto Donato (ack)]

Could you please provide your exact configuration for the storage on the machine and which version of centos you're deploying?

Also, can you try updating MAAS to a more recent version?

From a quick test, deploying Centos 8 from MAAS 3.1 with a raid-backed ext4 partition worked fine.

[Heitor (heitorpbittencourt)]

In our datacenter, we use the following storage settings for all the nodes:
- `/boot/efi`: vfat - 512 MiB
- `/`: ext4 - 100 GiB
- `/scratch`: XFS on a RAID0 array over up to 4 disks

We are using MAAS 2.9.2, with official OS images.

As of updating MaaS, that is not an option for us now. The company's policy for updates are to only update to new release versions (e.g. from x.y.z to x.y.z+1) or to new minor/major versions after the first release. That means, we can only update MaaS to 2.9.3 or 3.0.1 or 3.1.1, and none of them are available at the moment. We also are running in an enterprise model of having two environments: one for production and one for staging. We only update production after extensive testing on staging.

[Alberto Donato (ack)]

So it seems that bionic is required as a commissioning series in order to deploy Centos 8.

MAAS should require it explicitly, but a workaround is to change it globally in the config.

[Heitor (heitorpbittencourt)]

This is very puzzling to us. We have been using focal as the commissioning OS so far. We changed nothing in our systems. Why do we need to downgrade it now?

[Heitor (heitorpbittencourt)]

Which leads me to the next question: can we use CentOS instead of Ubuntu for commissioning machines?

[Derek DeMoss (derek-omnivector)]

Upon testing with Bionic instead of Focal for Commissioning, it does succeed.
Very strange..

[Derek DeMoss (derek-omnivector)]

This is great progress and should allow us to proceed with deployments tomorrow, but it does bring up an issue which must be addressed for the long term.

Bionic (18.04) only has one more year of 'General Support' left on the roadmap, so we still need a resolution as to why Focal can't be used, as I assume Bionic will be removed from MAAS' commissioning options at some point

[Jerzy Husakowski (jhusakowski)]

See also https://bugs.launchpad.net/maas/+bug/1965587

[Mauricio Faria de Oliveira (mfo)]

The underlying issue here is the same as of bug 1965587
(mkfs.xfs sets '-m reflink=1' by default, which is not
supported by centos70 kernel.)

With the workaround from there, XFS on MD-RAID0 works.

 [centos@z-rotomvm22 ~]$ uname -rv
 3.10.0-1160.45.1.el7.x86_64 #1 SMP Wed Oct 13 17:20:51 UTC 2021

 [centos@z-rotomvm22 ~]$ mount | grep -w xfs
 /dev/md0 on /xfs-raid0 type xfs (rw,relatime,seclabel,attr2,inode64,sunit=1024,swidth=3072,noquota)

 [centos@z-rotomvm22 ~]$ dmesg | grep -iw xfs
 [ 3.573489] SGI XFS with ACLs, security attributes, no debug enabled
 [ 3.599751] XFS (md0): Mounting V5 Filesystem
 [ 3.625700] XFS (md0): Ending clean mount

 [centos@z-rotomvm22 ~]$ cat /proc/mdstat
 Personalities : [raid0]
 md0 : active raid0 vda4[1] vda3[0] vda5[2]
       2918400 blocks super 1.2 512k chunks

 unused devices: <none>