Unable to use fsck on RHEL/CentOS Images deployed using Bionic Ephemeral Image

[Pedro Principeza]

From Bionic on, e2fsprogs implemented the "metadata_csum" as a default feature for ext4 filesystem creation [1].

With that change in place, deployments of CentOS / RHEL images (either through MIB or other means, such as packer) fail to execute fsck against ext4 filesystems of these, given these distros do not ship e2fsprogs version higher than 1.43 (or, better said, e2fsprogs versions that support metadata_csum).

The following message is seen, upon attempting to execute fsck against the filesystem:

 Apr 19 19:25:03 TOROON63I6W systemd-fsck: /dev/mapper/vg0-lv3 has unsupported feature(s):
 metadata_csum
 Apr 19 19:25:03 TOROON63I6W systemd-fsck: e2fsck: Get a newer version of e2fsck!
 Apr 19 19:25:03 TOROON63I6W systemd-fsck: fsck failed with error code 8.

The workaround, as of now, is to deploy the CentOS/RHEL images using a Xenial Ephemeral Image, thus avoiding the e2fsprogs version that takes "metadata_csum" as a default feature.

This setup uses MAAS at:

ii maas-cli 2.4.2-7034-g2f5deb8b8-0ubuntu1 all MAAS client and command-line interface
ii maas-common 2.4.2-7034-g2f5deb8b8-0ubuntu1 all MAAS server common files
ii maas-proxy 2.4.2-7034-g2f5deb8b8-0ubuntu1 all MAAS Caching Proxy
ii maas-region-api 2.4.2-7034-g2f5deb8b8-0ubuntu1 all Region controller API service for MAAS
ii maas-region-controller 2.4.2-7034-g2f5deb8b8-0ubuntu1 all Region Controller for MAAS
ii python3-django-maas 2.4.2-7034-g2f5deb8b8-0ubuntu1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.4.2-7034-g2f5deb8b8-0ubuntu1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.4.2-7034-g2f5deb8b8-0ubuntu1 all MAAS server provisioning libraries (Python 3)

But I believe this affects MAAS at any version.

To reproduce this issue:

-- Set Bionic as the Commissioning Image to be used;
-- Deploy a CentOS/RHEL 7 image using either MIB or packer, defining at least one ext4 fileystem on Curtin.
-- Check the messages file on the firstboot of the deployed image, or attempt to run fsck against the ext4 filesystem.

I'm attaching curtin debug logs, the yaml used on the deployment, and the messages file outlining the error message on the fsck execution.

[1] https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1365874

[Adam Collard (adam-collard)]

We plan to address this as a feature that explicitly models the dependencies between target deployment image and the ephemeral image that MAAS uses to run curtin in (and install that image).

This will, for example, give MAAS the knowledge that CentOS 6 requires Xenial, CentOS 7 requires Focal etc.

Internally, see PF-3237

[Jerzy Husakowski (jhusakowski)]

On MAAS feature backlog

[Igor Gnip (igorgnip)]

Alternative solution which appears to me easier is to turn off ext4 metadata checksum using tune2fs during deployment of OS which does not have e2fsprogs recent enough to support it.
That would be rhel7 and centos7 at least.

This could be part of curtin step as well and I think it can be done in the maas preseeds.

I strongly urge / advise you not to mess with default ephemeral versions cause there are other issues potentially opening that pandora's box - example: network or storage drivers not present in xenial.

[Igor Gnip (igorgnip)]

Not being able to reliably know which ephemeral image maas runs would be much worse than the current issue.