| 2019-02-22 09:18:16 |
Tyler Hicks |
description |
Bionic kernels including the current 4.15.0-45 (and 4.15.0-46 from the latest in git) contain the following commit https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bionic/commit/?id=70b9ab218a71b56b6a8c1f4c0daf0219a523859770b9ab218a71b56b6a8c1f4c0daf0219a5238597 from June 2018 which was an (incorrect) attempt to fix CVE-2018-10876.
Given that this problem involves changes to code relating to a CVE I have marked this bug as being security related
Unfortunately this fix introduces a new bug whereby the kernel will detect corruption and take evasive action if a filesystem is remounted before the lazy background initialisation has zeroed block group zero's inode table.
There was an upstream commit made in July 2018, which claims to fix the above regression but this one has not been adopted into Ubuntu's kernel:
(https://github.com/torvalds/linux/commit/5012284700775a4e6e3fbe7eac4c543c4874b559)
Contrary to the example in the upstream fix, it doesn't really matter whether the first mount is ro or rw although if the initial mount is read-only then the kernel will not initialise in the background and a subsequent remount as rw is guaranteed to hit this problem, whereas after an initial rw mount there is a small window of time where a remount will encounter this problem.
We are hitting this issue during automated builds because Puppet is likely to issue one or more remounts against a filesystem it has only just created and mounted. This seems like inefficient behaviour by Puppet, however remounting should be a safe operation and it currently is not.
SUGGESTED FIX:
--------------
Please could the above upstream commit (874b559) to be patched into Ubuntu's kernel to alleviate this?
STEPS TO REPRODUCE:
-------------------
You either need a spare disk/volume to reformat, or a 'loop' device backed to a large file - e.g.:
# fallocate -l 25G /bigfile && losetup --find --show /bigfile
Format the device, mount and remount e.g.:
# mkfs.ext4 /dev/loop2
# mount /dev/loop7 /mnt && mount -o remount /mnt
NOTE: Contrary to the similar example in the upstream fix I am mounting rw not ro see my earlier comments on this.
The filesystem will immediately be remounted read-only with the following in /var/log/syslog:
---
Feb 21 12:25:04 host kernel: [1299993.278265] EXT4-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Feb 21 12:25:04 host kernel: [1299993.284921] EXT4-fs error (device loop2): ext4_has_uninit_itable:3135: comm mount: Inode table for bg 0 marked as needing zeroing
Feb 21 12:25:04 host kernel: [1299993.341356] Aborting journal on device loop2-8.
Feb 21 12:25:05 host kernel: [1299993.634358] EXT4-fs (loop2): Remounting filesystem read-only
----
By running dump2efs in a loop after first mount I was able to observe that the output for Group 0 shows in my case that it takes about 4-5 seconds after the first mount before the kernel's ext4lazyinit process zeroes Group 0's inode table.
WORKAROUND:
-----------
For the benefit of anyone else hitting this issue prior to a kernel fix, it is possible to avoid this problem by forcing mkfs to zero the inode tables immediately - e.g.
# mkfs.ext4 -E lazy_itable_init=0
This will cause mkfs to take a little longer creating the filesystem.
With the workaround all block groups on the new filesystem (as shown by dumpe2fs) will look like this until some time after the filesystem was first mounted:
Group 0: (Blocks 0-32767) csum 0x7b1a
Once initialised (either by the ext4lazyinit or by using lazy_itable_init=0 ) they acquire the ITABLE_ZEROED flag and can then be safely remounted without hitting this bug:
Group 0: (Blocks 0-32767) csum 0x7b1a [ITABLE_ZEROED]
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-signed-image-generic 4.15.0.45.47
ProcVersionSignature: Ubuntu 4.15.0-45.48-generic 4.15.18
Uname: Linux 4.15.0-45-generic x86_64
NonfreeKernelModules: wl
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/hwC0D3', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D8p', '/dev/snd/pcmC0D7p', '/dev/snd/pcmC0D3p', '/dev/snd/pcmC0D1c', '/dev/snd/pcmC0D1p', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CurrentDesktop: MATE
Date: Thu Feb 21 10:13:27 2019
EcryptfsInUse: Yes
HibernationDevice: RESUME=UUID=b48635fa-6372-48f1-98fd-91631831a174
InstallationDate: Installed on 2017-10-04 (504 days ago)
InstallationMedia: Ubuntu-MATE 17.04 "Zesty Zapus" - Release amd64 (20170412)
MachineType: Apple Inc. Macmini6,2
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-45-generic root=UUID=a84d64f4-c466-422f-8a7e-d586bfa4a7ed ro
RelatedPackageVersions:
linux-restricted-modules-4.15.0-45-generic N/A
linux-backports-modules-4.15.0-45-generic N/A
linux-firmware 1.173.3
SourcePackage: linux
UpgradeStatus: Upgraded to bionic on 2018-10-09 (134 days ago)
dmi.bios.date: 10/20/2016
dmi.bios.vendor: Apple Inc.
dmi.bios.version: MM61.88Z.0106.B0B.1610201654
dmi.board.asset.tag: Base Board Asset Tag#
dmi.board.name: Mac-F65AE981FFA204ED
dmi.board.vendor: Apple Inc.
dmi.board.version: Macmini6,2
dmi.chassis.type: 16
dmi.chassis.vendor: Apple Inc.
dmi.chassis.version: Mac-F65AE981FFA204ED
dmi.modalias: dmi:bvnAppleInc.:bvrMM61.88Z.0106.B0B.1610201654:bd10/20/2016:svnAppleInc.:pnMacmini6,2:pvr1.0:rvnAppleInc.:rnMac-F65AE981FFA204ED:rvrMacmini6,2:cvnAppleInc.:ct16:cvrMac-F65AE981FFA204ED:
dmi.product.family: Macmini
dmi.product.name: Macmini6,2
dmi.product.version: 1.0
dmi.sys.vendor: Apple Inc. |
Bionic kernels including the current 4.15.0-45 (and 4.15.0-46 from the latest in git) contain the following commit https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bionic/commit/?id=70b9ab218a71b56b6a8c1f4c0daf0219a5238597 from June 2018 which was an (incorrect) attempt to fix CVE-2018-10876.
Given that this problem involves changes to code relating to a CVE I have marked this bug as being security related
Unfortunately this fix introduces a new bug whereby the kernel will detect corruption and take evasive action if a filesystem is remounted before the lazy background initialisation has zeroed block group zero's inode table.
There was an upstream commit made in July 2018, which claims to fix the above regression but this one has not been adopted into Ubuntu's kernel:
(https://github.com/torvalds/linux/commit/5012284700775a4e6e3fbe7eac4c543c4874b559)
Contrary to the example in the upstream fix, it doesn't really matter whether the first mount is ro or rw although if the initial mount is read-only then the kernel will not initialise in the background and a subsequent remount as rw is guaranteed to hit this problem, whereas after an initial rw mount there is a small window of time where a remount will encounter this problem.
We are hitting this issue during automated builds because Puppet is likely to issue one or more remounts against a filesystem it has only just created and mounted. This seems like inefficient behaviour by Puppet, however remounting should be a safe operation and it currently is not.
SUGGESTED FIX:
--------------
Please could the above upstream commit (874b559) to be patched into Ubuntu's kernel to alleviate this?
STEPS TO REPRODUCE:
-------------------
You either need a spare disk/volume to reformat, or a 'loop' device backed to a large file - e.g.:
# fallocate -l 25G /bigfile && losetup --find --show /bigfile
Format the device, mount and remount e.g.:
# mkfs.ext4 /dev/loop2
# mount /dev/loop7 /mnt && mount -o remount /mnt
NOTE: Contrary to the similar example in the upstream fix I am mounting rw not ro see my earlier comments on this.
The filesystem will immediately be remounted read-only with the following in /var/log/syslog:
---
Feb 21 12:25:04 host kernel: [1299993.278265] EXT4-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Feb 21 12:25:04 host kernel: [1299993.284921] EXT4-fs error (device loop2): ext4_has_uninit_itable:3135: comm mount: Inode table for bg 0 marked as needing zeroing
Feb 21 12:25:04 host kernel: [1299993.341356] Aborting journal on device loop2-8.
Feb 21 12:25:05 host kernel: [1299993.634358] EXT4-fs (loop2): Remounting filesystem read-only
----
By running dump2efs in a loop after first mount I was able to observe that the output for Group 0 shows in my case that it takes about 4-5 seconds after the first mount before the kernel's ext4lazyinit process zeroes Group 0's inode table.
WORKAROUND:
-----------
For the benefit of anyone else hitting this issue prior to a kernel fix, it is possible to avoid this problem by forcing mkfs to zero the inode tables immediately - e.g.
# mkfs.ext4 -E lazy_itable_init=0
This will cause mkfs to take a little longer creating the filesystem.
With the workaround all block groups on the new filesystem (as shown by dumpe2fs) will look like this until some time after the filesystem was first mounted:
Group 0: (Blocks 0-32767) csum 0x7b1a
Once initialised (either by the ext4lazyinit or by using lazy_itable_init=0 ) they acquire the ITABLE_ZEROED flag and can then be safely remounted without hitting this bug:
Group 0: (Blocks 0-32767) csum 0x7b1a [ITABLE_ZEROED]
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-signed-image-generic 4.15.0.45.47
ProcVersionSignature: Ubuntu 4.15.0-45.48-generic 4.15.18
Uname: Linux 4.15.0-45-generic x86_64
NonfreeKernelModules: wl
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/hwC0D3', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D8p', '/dev/snd/pcmC0D7p', '/dev/snd/pcmC0D3p', '/dev/snd/pcmC0D1c', '/dev/snd/pcmC0D1p', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CurrentDesktop: MATE
Date: Thu Feb 21 10:13:27 2019
EcryptfsInUse: Yes
HibernationDevice: RESUME=UUID=b48635fa-6372-48f1-98fd-91631831a174
InstallationDate: Installed on 2017-10-04 (504 days ago)
InstallationMedia: Ubuntu-MATE 17.04 "Zesty Zapus" - Release amd64 (20170412)
MachineType: Apple Inc. Macmini6,2
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-45-generic root=UUID=a84d64f4-c466-422f-8a7e-d586bfa4a7ed ro
RelatedPackageVersions:
linux-restricted-modules-4.15.0-45-generic N/A
linux-backports-modules-4.15.0-45-generic N/A
linux-firmware 1.173.3
SourcePackage: linux
UpgradeStatus: Upgraded to bionic on 2018-10-09 (134 days ago)
dmi.bios.date: 10/20/2016
dmi.bios.vendor: Apple Inc.
dmi.bios.version: MM61.88Z.0106.B0B.1610201654
dmi.board.asset.tag: Base Board Asset Tag#
dmi.board.name: Mac-F65AE981FFA204ED
dmi.board.vendor: Apple Inc.
dmi.board.version: Macmini6,2
dmi.chassis.type: 16
dmi.chassis.vendor: Apple Inc.
dmi.chassis.version: Mac-F65AE981FFA204ED
dmi.modalias: dmi:bvnAppleInc.:bvrMM61.88Z.0106.B0B.1610201654:bd10/20/2016:svnAppleInc.:pnMacmini6,2:pvr1.0:rvnAppleInc.:rnMac-F65AE981FFA204ED:rvrMacmini6,2:cvnAppleInc.:ct16:cvrMac-F65AE981FFA204ED:
dmi.product.family: Macmini
dmi.product.name: Macmini6,2
dmi.product.version: 1.0
dmi.sys.vendor: Apple Inc. |
|
