EXT4: Mainstream fix for regression caused by fix CVE-2018-10876 is missing from Ubuntu Bionic Kernel
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Invalid
|
Undecided
|
Unassigned | ||
Bionic |
Fix Committed
|
Undecided
|
Unassigned |
Bug Description
Bionic kernels including the current 4.15.0-45 (and 4.15.0-46 from the latest in git) contain the following commit https:/
Given that this problem involves changes to code relating to a CVE I have marked this bug as being security related
Unfortunately this fix introduces a new bug whereby the kernel will detect corruption and take evasive action if a filesystem is remounted before the lazy background initialisation has zeroed block group zero's inode table.
There was an upstream commit made in July 2018, which claims to fix the above regression but this one has not been adopted into Ubuntu's kernel:
(https:/
Contrary to the example in the upstream fix, it doesn't really matter whether the first mount is ro or rw although if the initial mount is read-only then the kernel will not initialise in the background and a subsequent remount as rw is guaranteed to hit this problem, whereas after an initial rw mount there is a small window of time where a remount will encounter this problem.
We are hitting this issue during automated builds because Puppet is likely to issue one or more remounts against a filesystem it has only just created and mounted. This seems like inefficient behaviour by Puppet, however remounting should be a safe operation and it currently is not.
SUGGESTED FIX:
--------------
Please could the above upstream commit (874b559) to be patched into Ubuntu's kernel to alleviate this?
STEPS TO REPRODUCE:
-------------------
You either need a spare disk/volume to reformat, or a 'loop' device backed to a large file - e.g.:
# fallocate -l 25G /bigfile && losetup --find --show /bigfile
Format the device, mount and remount e.g.:
# mkfs.ext4 /dev/loop2
# mount /dev/loop7 /mnt && mount -o remount /mnt
NOTE: Contrary to the similar example in the upstream fix I am mounting rw not ro see my earlier comments on this.
The filesystem will immediately be remounted read-only with the following in /var/log/syslog:
---
Feb 21 12:25:04 host kernel: [1299993.278265] EXT4-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Feb 21 12:25:04 host kernel: [1299993.284921] EXT4-fs error (device loop2): ext4_has_
Feb 21 12:25:04 host kernel: [1299993.341356] Aborting journal on device loop2-8.
Feb 21 12:25:05 host kernel: [1299993.634358] EXT4-fs (loop2): Remounting filesystem read-only
----
By running dump2efs in a loop after first mount I was able to observe that the output for Group 0 shows in my case that it takes about 4-5 seconds after the first mount before the kernel's ext4lazyinit process zeroes Group 0's inode table.
WORKAROUND:
-----------
For the benefit of anyone else hitting this issue prior to a kernel fix, it is possible to avoid this problem by forcing mkfs to zero the inode tables immediately - e.g.
# mkfs.ext4 -E lazy_itable_init=0
This will cause mkfs to take a little longer creating the filesystem.
With the workaround all block groups on the new filesystem (as shown by dumpe2fs) will look like this until some time after the filesystem was first mounted:
Group 0: (Blocks 0-32767) csum 0x7b1a
Once initialised (either by the ext4lazyinit or by using lazy_itable_init=0 ) they acquire the ITABLE_ZEROED flag and can then be safely remounted without hitting this bug:
Group 0: (Blocks 0-32767) csum 0x7b1a [ITABLE_ZEROED]
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-signed-
ProcVersionSign
Uname: Linux 4.15.0-45-generic x86_64
NonfreeKernelMo
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/hwC0D3', '/dev/snd/hwC0D0', '/dev/snd/
CurrentDesktop: MATE
Date: Thu Feb 21 10:13:27 2019
EcryptfsInUse: Yes
HibernationDevice: RESUME=
InstallationDate: Installed on 2017-10-04 (504 days ago)
InstallationMedia: Ubuntu-MATE 17.04 "Zesty Zapus" - Release amd64 (20170412)
MachineType: Apple Inc. Macmini6,2
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=
RelatedPackageV
linux-
linux-
linux-firmware 1.173.3
SourcePackage: linux
UpgradeStatus: Upgraded to bionic on 2018-10-09 (134 days ago)
dmi.bios.date: 10/20/2016
dmi.bios.vendor: Apple Inc.
dmi.bios.version: MM61.88Z.
dmi.board.
dmi.board.name: Mac-F65AE981FFA
dmi.board.vendor: Apple Inc.
dmi.board.version: Macmini6,2
dmi.chassis.type: 16
dmi.chassis.vendor: Apple Inc.
dmi.chassis.
dmi.modalias: dmi:bvnAppleInc
dmi.product.family: Macmini
dmi.product.name: Macmini6,2
dmi.product.
dmi.sys.vendor: Apple Inc.
description: | updated |
Changed in linux (Ubuntu): | |
status: | Confirmed → Invalid |
Changed in linux (Ubuntu Bionic): | |
status: | New → Confirmed |
Thanks for the report, I'm marking this public so the kernel team can more easily work with it.
Thanks