Ubuntu 16.10+ installer uses ext4 feature 'metadata_csum' which is incompatible with older (LTS) e2fsprogs

[Norbert]

Ubuntu 16.10 installer sets metadata_csum option on ext4 partition which is incompatible with other LTS Ubuntu versions (12.04 LTS, 14.04 LTS, 16.04 LTS).

Steps to reproduce:
1. Download Ubuntu 16.10 installation media.
2. Install Ubuntu.
3. Try to do fsck -fy /dev/sdX1 from other supported Ubuntu distro.

Expected results:
User can check and fix errors on ext4 filesystem, created on Ubuntu 16.10.

Actual results:
User can not check and fix errors on ext4 filesystem because of lack of 'metadata_csum' option in previous LTS Ubuntu versions.
The only one working solution was to scan from 16.10 live install media.

Note:
it is known, that Dan Watkins disabled metadata_csum when creating ext4 filesystems ( see http://bazaar.launchpad.net/~daniel-thewatkins/maas-images/fix-yakkety-builds/revision/305 ). It is good solution.

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: e2fsprogs 1.43.1-1
ProcVersionSignature: Ubuntu 4.4.0-30.49-generic 4.4.13
Uname: Linux 4.4.0-30-generic i686
ApportVersion: 2.20.2-0ubuntu1
Architecture: i386
CurrentDesktop: Unity
Date: Mon Jul 11 23:42:49 2016
SourcePackage: e2fsprogs
UpgradeStatus: No upgrade log present (probably fresh install)

[Norbert (nrbrtx)]

There is a linked bug 1365874 about adding 'metadata_csum' to e2fsk in LTS versions.

[Richard Laager (rlaager)]

I don't see how turning on a new feature is a bug.

[Theodore Ts'o (tytso)]

One bit of context --- metadata_csum is not enabled by default in the official upstream e2fsprogs.tar.gz file. So with my upstream maintainer hat, I deliberately decided not to enable it by default, and mentioned in the release notes that individual distributions should decide whether they wanted to enable it. So with the upstream e2fsprogs tarball, the user can still request metadata_csum by asking for it explicitly: mke2fs -t ext4 -O metadata_csum.

With my *Debian* maintainer hat on (well, with some egging on with my upstream maintainer persona :-), I decided to enable metadata_csum by default so that in the testing and unstable branches, metadadata_csum checking would get some additional exposure, and hence testing. This gambit has worked. There have been a number of bug between 1.43 and 1.43.3 that were fixed because they were reported by Debian users. Whether I will continue leaving metadata_csum enabled right before Debian Stretch goes into final lockdown for the Debian Stable release is not something I have decided, but so far the bug report rate has been positive.

Ubuntu has apparently adopted the Debian enablement of metadata_csum by default, because it's based on the Debian 1.43.3-1 package. However, there may be some differences between Ubuntu and Debian --- the average technical sophistication of a Debian vs. a Ubuntu user, compatibility constraints with Ubuntu LTS, etc. --- that may drive a different decision with respect to mke2fs's *defaults*.

It would be good if a decision is made explicitly by Ubuntu / Canonical to decide what best makes since for Ubuntu. If you decide that Ubuntu 16.10 is a community distro, and you want to help me test metadata_csum, that's great. I have had some experiences with less-than-savvy Ubuntu users who really struggled with filing a useful bug report and participating in root causing a bug.

[Francesco Fumanti (frafu)]

I have got a problem because of the non-matching features, too: I formatted a new drive with ext4 under Ubuntu 16.10 and added it to the fstab of my precise and trusty installations.

These two LTS installations stop during the boot press giving me an error because e2fsck does not recognize the new features. (The problem does not occur for xenial.)

As Norbert and Theodor above: e2fsprogs 1.43 should be backported to precise and trusty; especially now that the new features are enabled by default.

In trusty, I am now using Daniel Mehrmann's PPA, which is offering a snapshot of e2fsck 1.43.

Daniel Mehrmann's package does not build in precise because it seems to need a newer version of libc6.

[TJ (tj)]

An issue for 18.04 too, reported in IRC.

There is a related bug "Ubuntu 12.04 LTS, 14.04 LTS, 16.04 LTS do not support ext4 metadata checksumming"

https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1365874

[Simon Iremonger (ubuntu-iremonger)]

Canonical please read Theodore Tso's Comment #4 above, and consider for Ubuntu 18.04 LTS!

This "64bit,metadata_csum" creates compatibility-issues even with 16.04 LTS, and does not seem to provide a huge benefit [>16TB fs support, slightly stronger metadata checksumming].

This creates all sorts of problems for compatibility/portability of filesystems, e.g.:-
* dual-booting 18.04, even previous LTS version cannot fsck the filesystem.
* "ext4 portable disks" created by 18.04 similarly same problem.
* Also consider how wide is the 64bit,metadata_csum support anyway, users may want disks to work with many distros/drivers? Canonical's commercial-supporters may have views herein?. -- the linked bugreport also mentions another issue with hwe-edge and LVM.

Also note, turning off the 64bit,metadata_csum when required is a total PAIN, needing filesystem offline, fsck and tune2fs in careful concert.....

In my view, the 16.04 LTS mke2fs.conf [ ext4{} stanza with
"auto_64-bit_support = 1" and NO "64bit,metadata_csum" in "features" ]
should be seriously-considered for 18.04 "LTS" (even if 18.10 onwards follow Debian).

As tytso says, this 'change' has been "Decision-by-Default" due to importing a Debian upstream package, I would like Canonical to make an 'informed and considered decision' about this please!.

Hope that helps!

[Theodore Ts'o (tytso)]

It should be noted that I recently released e2fsprogs 1.44.0 (currently in Debian Unstable, should hopefully hit Debian Testing in three more days) which turns on Metadata Checksums for everyone.

https://packages.debian.org/sid/e2fsprogs

Pulling in 1.44.0 into 18.04 LTS would be nice since it supports two new features, largedir and ea_inode, (neither turned on by default yet, but the second in particular is very useful for Samba servers since it was written specifically to efficiently support Windows ACL's and Security ID's better than any other file system by supporting xattr dedupe.)

http://e2fsprogs.sourceforge.net/e2fsprogs-release.html#1.44.0

[TJ (tj)]

I'll try to get core dev's attention this week; it'd require a Feature Freeze Exception (FFE) now to get v1.44.0 into 18.04 but as an LTS it would be better to have it from the start - and decide on the default installer/mkfs flags now rather than have users suffer later.

From that we could possibly release it to 16.04 xenial-backports. Not sure about 14.04 trusty though. The latest kernel for 14.04 is v4.4 (the GA 16.04 kernel).

[Theodore Ts'o (tytso)]

E2fsprogs 1.44.0 now depends on dpkg build-profiles, which means that getting it backported to 14.04 LTS would require adjusting debian/control and debian/rules a bit. For 14.04 LTS, I'd urge consideration of going to e2fsprogs 1.43.9. This will get you most of the latest bug fixes, including some that could cause massive file system corruption and data loss (relative to e2fsprogs 1.42.x) in the right (wrong) circumstances.

[Simon Iremonger (ubuntu-iremonger)]

Can we pay-attention, to this thread (now) being about considering the feature-flags 'used by default' in mke2fs.conf, in consideration to 18.04 -- [linked bug for e2fsprogs].

We know massive 'compatibility'/portability benefit of formatting the same as previous-LTS by default, as the required e2fsprogs was only introduced 16.10, and the needed kernel was only in 16.04 (and 14.04-HWE-kernel-update.). NB: Looks like there is a similar issue with GRUB2 which affects dual-booters, grub2 2.02beta3 may be needed to support 64bit FS. If so, again Xenial isn't compatible to detect an 18.04 in multi-boot-menu.

I'd like tytso to comment on the downsides of formatting without 64bit,metadata_csum, more specifically. From what I can (see) this only loses a little extra checksumming-integrity (against bad disks, which we've never needed before??), and the 64bit feature appears only to be needed to strengthen these checksums or support >16TB disks (bearing in mind auto_64bit_support can still be used). NB: Is this correct, -OR- is there likely going to be future annoyances with 'other' ext4 features-to-come which will ALSO expect 64bit formatting?.

My suggestion is, this hasn't been sorted-out enough in the ubuntu-world, and 64bit,metadata_csum should be disabled-by-default for 18.04 and backporting grub/e2fsprogs/etc where relevant should happen as well [see other thread, possibly another may be created for grub2].

[Robie Basak (racb)]

I posted to the mailing list about this: https://lists.ubuntu.com/archives/ubuntu-devel/2018-March/040230.html

[Simon Iremonger (ubuntu-iremonger)]

Initial comments from the email:-

a) There is some confusion over "metadata_csum" with/without "64bit". -- Those who have 'reverted' are usually reverting BOTH flags [as I've done some places], not just metadata_csum.
My understanding is 64bit is of no benefit except support >16TB fs and to strengthen metadata_csum's [if used, which help notice dodgy disks in theory.].

b) The business of including e2fsprogs-1.44.0 in 18.04 at tytso's request [NON-default extra feature support may benefit Samba users etc. later] is not addressed.

c) Its' worth pointing-out explicitly e2fsprogs only enabled 64bit,metadata_csum 'by default' for 'everybody' within last few weeks, Debian change was as-you-rightly-quote.

d) Considering the above, also think outside the ubuntu-box! What do canonical's customers/partners expect compatibility-wise with other non-ubuntu systems, virtualizers/ISCSI-hosts/etc (e2fsprogs 1.42 still very common!), let alone "backports to ubuntu LTS versions" only?.

e) My understanding from TJ in IRC is he's started doing some tests on some datacentre cases, there are particular issues with things like ISCSI hosts, the host system needs to fsck guest-FS and break otherwise!. HOPEFULLY this will appear soon and can be added to the mail-discussion.

I can try to join email/post directly to mail if appropriate/helpful.

[Steve Langasek (vorlon)]

This has been evaluated by the Foundations team and we do not intend to diverge from upstream's defaults for 18.04. The incompatibility between LTSes is not ideal, but is secondary to being able to improve the filesystem defaults across LTS releases.

[Simon Iremonger (ubuntu-iremonger)]

Steve:

Given what you say, can you consider Tytso's request to put 1.44.0 e2fsprogs into Bionic. This allows for not-enabled-by-default for support of 2 extra ext4 flags in e2fsprogs, thereby avoiding this situation happening again for 20.04 with respect to THOSE flags...

As TJ- said, be good to get it in for LTS release from the off.

https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1365874/comments/17

[maybe that should be handled in the other bug as above].

[Steve Langasek (vorlon)]

On Thu, Mar 15, 2018 at 06:08:36PM -0000, Simon Iremonger wrote:

> Given what you say, can you consider Tytso's request to put 1.44.0
> e2fsprogs into Bionic. This allows for not-enabled-by-default for
> support of 2 extra ext4 flags in e2fsprogs, thereby avoiding this
> situation happening again for 20.04 with respect to THOSE flags...

I've filed a feature freeze exception request for this issue.

  https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1756177

[Simon Iremonger (ubuntu-iremonger)]

FWIW: e2fsprogs PPA created and discussion of backports/updates should happen on this linked-bug:-
https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1365874 -- if you are interested in xenial/older e2fsprogs compatibility please subscribe to that bug #1365874 .

I also spotted e2fsprogs 1.44.2 has fix against crashing from malicious-filesystems, remains to be seen if this and further future fixes may ultimately warrant update to bionic's 1.44.1 .