FDE image fails to run e2fsck

[Jean-Baptiste Lallement]

After installation of the FDE image, the system fails to boot due to e2fsck failing with:

Jun 21 12:48:19 ubuntu systemd-fsck[268]: /dev/vda2 has unsupported feature(s): FEATURE_C12

----

this means that Jammy fsck fails against mantic created ext4 which is not great

Seems this is orphan_file feature / orphan_present

Also need to check if grub2 supports this as it is RO_INCOMPAT feature.

[Fix]

This SRU is disabling 2 features (orphan_file and metadata_csum_seed) from ext4 to generate filesystems compatible with older series. Fixes come from debian (fixing #1031622, #1030939). This is only disabling the feature by default. One can still generate a filesystem with this feature by explicitly listing them when invoking mkfs.ext4. The decision to disable this feature by default has already been taken for mantic and is not planned to be revised soon as far as I know. So this fix is also useful to implement a common behavior between the 2 series (and following ones).

This fix is also motivated by the fact that snapd is using mkfs.ext4 without any options. The possibility to give options for mkfs.ext4 was removed in snapd at some point for good reasons and it would be more complex to handle a special case only for lunar in snapd than disabling this feature by default in e2fsprogs I think.

[Impact]

See LP: #2028564. Generating a filesystem on lunar with e2fsprogs will use the orphan_file feature. This will prevent a jammy (or older) based OS from modifying this filesystem (in particular resize it at first boot).

[ Test Plan ]

On lunar, generate a filesystem and check the orphan_file feature is enabled:

$ dd if=/dev/zero of=test.img bs=100M count=1
$ mkfs.ext4 test.img
$ # Then check the orphan_file feature is enabled
$ dumpe2fs test.img | grep orphan

Move this .img file to a jammy based machine, and try using resize2fs on it:

$ resize2fs -d -f -M test.img

resize2fs 1.46.5 (30-Dec-2021)
resize2fs: Filesystem has unsupported feature(s) (test.img)

[Where problems could occur]

Some users on Lunar may currently rely on this feature to be enabled by default. This can still be enabled explicitly when calling mkfs.ext4 but the default behavior will be changed.

The orphan_file feature aims at improving performances when dealing with deleted files or directories. So if we disable this by default, I think it should only impact performances of generated filesystems and probably not break anything else.

It should be noted that before lunar, users did not have access to this feature and since mantic it is disabled by default, so I suspect the population expecting this feature to be enabled by default is rather small.

[Dimitri John Ledkov (xnox)]

Given new incompatible RO and RW features in ext4 that v5.15 kernels support, imho we should SRU backport of e2fsprogs on HWE grounds.

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Related debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1031622

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Julian Andres Klode (juliank)]

Newer releases of Ubuntu can always start making use of new file system feature flags. Looking at all currently supported releases (whether standard or ESM), we have never SRUed e2fsprogs before, it only ever received security updates, so I'd be inclined to not do a backport here either and resolve that in the snap side of things.

Generally for recovery purposes you will be booting an image with an equal or newer version of Ubuntu.

Regarding Dimitri's comment about new kernel feature support in 5.15, that doesn't seem related to this issue, and I'm not sure we should be changing feature flag in a stable release anyhow.

[Benjamin Drung (bdrung)]

I ran into this bug myself a while ago by doing following: I bought a new NVMe and put it into my laptop. Then I booted a recent Ubuntu live system, created partitions on the NVMe, copied everything from the old to the new, updated entries in /etc/fstab etc. When I rebooted the file system check failed with that error message. I either re-did everything with a Ubuntu jammy live system or hot-replaced fsck in the initramfs by /bin/true and upgraded Ubuntu.

[Theodore Ts'o (tytso)]

For Debian the metadata_csum_seed and orphan_file features were disabled by default by changing the /etc/mke2fs.conf file.

e2fsprogs (1.47.0-2) unstable; urgency=medium

  * Don't enable metadata_csum_seed and orhpan_file by default (Closes:
    #1031622, #1030939)

 -- Theodore Y. Ts'o <email address hidden> Sat, 04 Mar 2023 22:16:08 -0500

Ubuntu Lunar and Mantic chose to only disable metadata_csum_seed:

e2fsprogs (1.47.0-1ubuntu1) lunar; urgency=low

  * Merge from Debian unstable. Remaining changes:
    - Use 4k blocksize in all ext4 mke2fs.conf stanzas, so lvm migration
      between non-4k PVs and 4k PVs works regardless of the volume size.
  * Disable the metadata_csum_seed feature again, as grub does not yet
    support it (Closes: #866603)

 -- Julian Andres Klode <email address hidden> Fri, 17 Feb 2023 11:58:55 +0100

It should be noted that users are always free to change /etc/mke2fs.conf, since that's a configuration file, and the whole point of that file is to make it easy for distributions and users to customize how they mke2fs should behave. So if you think that Ubuntu Mantic should be able create file systems that can be handled by Jammy, it's just a matter of adjusting the mke2fs.conf file in Mantic.

As far as grub support is concerned, the metadata_csum_seed feature flag is supposed by grub2 (2.06-8) or newer. And grub doesn't care about orphan_file, since it's a COMPAT flag. That is an issue for e2fsck, but not for grub. The question was always whether a distribution supports creating a file system that can be (a) mounted and/or (b) fsck'ed by an older distribution, and if so, for how many versions back. That's not something for which people of good will can disagree, which is why it can be made configurable.

[Launchpad Janitor (janitor)]

This bug was fixed in the package e2fsprogs - 1.47.0-1ubuntu2

---------------
e2fsprogs (1.47.0-1ubuntu2) mantic; urgency=medium

  * Temporarily disable orphan_file by default, such that fsck from jammy
    can check ext4 created in mantic. Given the new incompat ext4 features
    that v5.15 & hwe kernels support, ideally e2fsprogs 1.47.0 should be
    SRUed into jammy. LP: #2025339

 -- Dimitri John Ledkov <email address hidden> Thu, 29 Jun 2023 10:14:32 +0100

[Robie Basak (racb)]

> So if we disable this by default, I think it should only impact performances of generated filesystems and probably not break anything else.

This sounds like it could be insignificant, or a major performance regression in some use cases. It's not clear which. Presumably orphan_file was introduced for a practical performance reason.

Please provide a proper analysis of the risk of performance regression here, and a documented reason why this cannot simply be worked around by adjusting how mke2fs is called (or by adjusting its configuration) in the use cases which require it, before proposing this fix for SRU.

[Robie Basak (racb)]

Also, given that this is the second time this kind of thing has happened, perhaps it is appropriate to add a dep8 test to ensure that the compatibility "level" of the default has not regressed.

[Robie Basak (racb)]

(in the development release)

[Paul Mars (upils)]

SRU abandoned to not change the expected default behavior in lunar.

[Robie Basak (racb)]

Further to a conversation in #ubuntu-devel I'm setting the Lunar task to Won't Fix. Further details: https://irclogs.ubuntu.com/2023/11/15/%23ubuntu-devel.html