After a side by side installation, resized filesystem is corrupted

Bug #1798562 reported by Jean-Baptiste Lallement on 2018-10-18
64
This bug affects 8 people
Affects Status Importance Assigned to Milestone
e2fsprogs (Ubuntu)
Critical
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned

Bug Description

[Impact]
- Users resizing filesystems using resize2fs.
- Resizing an existing Linux filesystem from the Ubuntu installer.

[Test cases]

Test Case 1:
With e2fsprogs 1.44.4-2:

Download this file system image: https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1798562/+attachment/5203159/+files/vda1b.qcow2.bz

Run the following commands:

bunzip2 vda1b.qcow2.bz
qemu-img convert vda1b.qcow2 vda1b.raw
e2fsck -f vda1b.raw
resize2fs vda1b.raw 6200M
e2fsck -f vda1b.raw

e2fsck 1.44.4 (18-Aug-2018)
Pass 1: Checking inodes, blocks, and sizes
Inode 45746 extent block passes checks, but checksum does not match extent
        (logical block 1272, physical block 466167, len 776)
Fix<y>?

Test Case 2 (Use case of the installer):
1. Perform a first installation of Ubuntu
2. Reboot into the freshly installed system and verify everything works as expected
3. Do a second installation and select "Install alonogside". Keep the size proposed by the installer and proceed to the end of installation
4. Reboot
5. On boot, in the list of installed systems, select the first system installed on the machine
6. Verify that it boots, you can login and it works as expected

Actual Result
The FS is corrupted and depending on the corruption it goes to initramfs, boots but cannot login, ...

[Regression potential]
This changes behavior in the update logic for extent blocks; as such, care should be taken to make sure that testing takes into account the possibility to introduce filesystem corruption while resizing. This is why the test cases include running e2fsck as last step.

---

Attached is the journal of the system installed on the corrupted partition.

ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: ubiquity (not installed)
ProcVersionSignature: Ubuntu 4.18.0-10.11-generic 4.18.12
Uname: Linux 4.18.0-10-generic x86_64
ApportVersion: 2.20.10-0ubuntu13
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Thu Oct 18 10:53:57 2018
InstallCmdLine: BOOT_IMAGE=/casper/vmlinuz file=/cdrom/preseed/ubuntu.seed boot=casper only-ubiquity quiet splash ---
InstallationDate: Installed on 2018-10-18 (0 days ago)
InstallationMedia: Ubuntu 18.10 "Cosmic Cuttlefish" - Release amd64 (20181017.3)
SourcePackage: ubiquity
UpgradeStatus: No upgrade log present (probably fresh install)

Jean-Baptiste Lallement (jibel) wrote :
APolihron (apolitech) wrote :

Do you get this screen?

APolihron (apolitech) wrote :

Last night this problem made me go crazy. Because if you install again 2 os's for dual boot the problem will NOT persists. I can't reproduce this problem the second time.

Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1798562

tags: added: iso-testing
Jean-Baptiste Lallement (jibel) wrote :

@Apolihron, yes but depending on the corruption the system can go into initramfs if it completely fails to boot (your screenshot), it may boot but the FS will be RO, or it may recover transparently and all sort of other side effects are possible.

Jean-Baptiste Lallement (jibel) wrote :

@APoliihron, when it happened to you what was the original operating system that you resized? Was it cosmic on cosmic or something else?

APolihron (apolitech) wrote :

Cosmic entire disk + cosmic auto-resize (install alongside)
P.S uifi version

APolihron (apolitech) wrote :

I had this problem after the uefi fix.in Ubuntu Mate 20181017.1

Rik Mills (rikmills) wrote :

In Kubuntu with Virtualbox, I end up with the resized Cosmic partition bootable and starts, but read is read only.

Plenty of KDE error not being able to write config files to the user home.

Ext4 errors shown in attached screenshot.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubiquity (Ubuntu):
status: New → Confirmed
Rik Mills (rikmills) on 2018-10-18
Changed in ubiquity (Ubuntu):
importance: Undecided → High

Apparently it's clearly reproducible with Cosmic alongside Cosmic.

Has anyone been able to reproduce with Cosmic alongside something else than Cosmic such as an earlier release, another distribution or operating system?

Łukasz Zemczak (sil2100) wrote :

Just reporting that I was unable to reproduce it in any system combination, even in the original cosmic next to cosmic case. Same for bionic on cosmic, cosmic on bionic, cosmic on fedora. So it seems the issue is not easily reproducible in all cases, must be some specific combination of components and/or race?

APolihron (apolitech) wrote :

Personally I've fone this first install entire disk ->boot in the os and restart it -> install alongside-> boot in the second os and then restart and boot in the first one.all in uefi mode.

With e2fsprogs 1.44.4-2:

bunzip2 vda1b.qcow2.bz
qemu-img convert vda1b.qcow2 vda1b.raw
e2fsck -f vda1b.raw
resize2fs vda1b.raw 6200M
e2fsck -f vda1b.raw

e2fsck 1.44.4 (18-Aug-2018)
Pass 1: Checking inodes, blocks, and sizes
Inode 45746 extent block passes checks, but checksum does not match extent
        (logical block 1272, physical block 466167, len 776)
Fix<y>?

affects: ubiquity (Ubuntu) → e2fsprogs (Ubuntu)
Changed in e2fsprogs (Ubuntu):
importance: High → Critical
tags: added: rls-dd-incoming
description: updated
Theodore Ts'o (tytso) wrote :

Thanks for the very nice repro! I've created a fix which will be going into the maint branch of the e2fsprogs tree. The commit description:

    resize2fs: update checksums in the extent tree's relocated block

    When shrinking an file system, and we need to relocate an inode, the
    checksums in its extent tree must get updated to reflect its new inode
    number. When doing this, we need to do this *after* we update the
    extent tree to reflect any blocks which need to be relocated due to
    the file system shrink operation.

    Otherwise, in the case where only an interior node of the extent tree
    needs to get relocated, and none of the entries in that node need to
    be adjusted, the checksum for that interior node is updated in the old
    copy of that block, and then after the extent tree is updated to use
    the new copy of that interior node, the extent tree is left with an
    invalid checksum.

    This is a relatively rare case, since it requires the following
    conditions to be true:

    *) The metadata checksum feature must be enabled.
    *) An inode needs to be relocated.
    *) The inode needs to have an interior node.
    *) The block for that interior node needs to be relocated.
    *) None of blocks addressed by entries in that interior node needs
        to be relocated.

    When all of these conditions are true, though, the file system is left
    with corrupted with bad checksum for the extent tree block.

    Addresses-Launchpad-Bug: 1798562

    Signed-off-by: Theodore Ts'o <email address hidden>
    Reported-by: Jean-Baptiste Lallement <email address hidden>

I've tested e2fsprogs with this change and it fixes your repro. I also have a regression test in the subsequent commit which reproduces the problem with a smaller test file system.

The attachment "resize2fs: update checksums in the extent tree's relocated block" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Rahul Negi (rahulnegi1409) wrote :

Hello , I face same issue

it shows busybox command line and says file system is corrupted ( tells to use fsck)

and after doing that from live disk ( i saw it was asking to move nodes which repairing ) the next time system reboots the kernel panics, errors were " not syncing attempting to kill" ," bad RF code "

im using windows 10 and ubuntu 18.10 alongside

i have installed it like 8-9 times and after 3-4 reboots , file system corrupts

we can increase the fs corruption time if we do software update

after updating all updates , next reboot corrupts the filesystem ( does not take more reboots )

Brian Murray (brian-murray) wrote :

This has been fixed in the e2fsprogs version in Disco.

[ 8:40AM 10668 ] [ bdmurray@impulse:/tmp/e2fsprogs-1.44.5 ]
 $ grep -B1 1798562 doc/RelNotes/v1.44.5.txt
Fix a bug where resize2fs was failing to update the extent tree
checksums in an corner case. (Addresses Launchpad Bug: #1798562)

Changed in e2fsprogs (Ubuntu):
status: Confirmed → Fix Released
description: updated

Hello Jean-Baptiste, or anyone else affected,

Accepted e2fsprogs into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/e2fsprogs/1.44.4-2ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in e2fsprogs (Ubuntu Cosmic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Changed in e2fsprogs (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed-bionic
Brian Murray (brian-murray) wrote :

Hello Jean-Baptiste, or anyone else affected,

Accepted e2fsprogs into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/e2fsprogs/1.44.1-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: id-5c49e9dab8b59b7d3bbca789
sudodus (nio-wiklund) wrote :

I did both testcases. There was no error detected in in testcase #1, and no error detected in testcase #2, so things seem to work :-)

Details:

- There was a test installation of Ubuntu 18.04.1 LTS in a Toshiba laptop with an SSD.
- I cloned a Lubuntu 18.10 iso file to a USB pendrive and installed the necessary program packages.
- I installed Lubuntu alongside Ubuntu (testcase #2)
- I ran testcase #1
- Then I rebooted into the old Ubuntu, and it worked
- Finally I rebooted into the new Lubuntu and ran 'sudo e2fsck -f /dev/sda1' and, as expected, there was no complaint.

sudodus (nio-wiklund) wrote :

I installed e2fsprogs 1.44.4-2 via cosmic-proposed, according to the instructions.

sudodus (nio-wiklund) on 2019-01-31
tags: added: verification-done-cosmic
removed: verification-needed-cosmic

Verification-done for bionic using e2fsprogs 1.44.1-1ubuntu1.1:

ubuntu@humble-cod:~$ qemu-img convert vda1b.qcow2 vda1b.raw
ubuntu@humble-cod:~$ e2fsck -f vda1b.raw
e2fsck 1.44.1 (24-Mar-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
vda1b.raw: 131875/786432 files (0.1% non-contiguous), 1115854/3145216 blocks
ubuntu@humble-cod:~$ resize2fs vda1b.raw 6200M
resize2fs 1.44.1 (24-Mar-2018)
Resizing the filesystem on vda1b.raw to 1587200 (4k) blocks.
The filesystem on vda1b.raw is now 1587200 (4k) blocks long.

ubuntu@humble-cod:~$ e2fsck -f vda1b.raw
e2fsck 1.44.1 (24-Mar-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
vda1b.raw: 131875/401408 files (0.2% non-contiguous), 1089656/1587200 blocks
ubuntu@humble-cod:~$ dpkg -l e2fsprogs libext2fs2
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=============================-===================-===================-===============================================================
ii e2fsprogs 1.44.1-1ubuntu1.1 amd64 ext2/ext3/ext4 file system utilities
ii libext2fs2:amd64 1.44.1-1ubuntu1.1 amd64 ext2/ext3/ext4 file system libraries

tags: added: verification-done-bionic
removed: verification-needed verification-needed-bionic
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package e2fsprogs - 1.44.4-2ubuntu0.2

---------------
e2fsprogs (1.44.4-2ubuntu0.2) cosmic; urgency=medium

  * d/patches/0001-resize2fs-update-checksums-in-the-extent-tree-s-relo.patch:
    do the checksum update later in extent tree relocated block to denote the
    inode number change, otherwise the checksum update might be done in the old
    copy of the block. (LP: #1798562)

 -- Mathieu Trudel-Lapierre <email address hidden> Fri, 25 Jan 2019 10:02:34 -0500

Changed in e2fsprogs (Ubuntu Cosmic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for e2fsprogs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

:Verification-done for cosmic:

ubuntu@superb-ram:~$ qemu-img convert vda1b.qcow2 vda1b.raw
ubuntu@superb-ram:~$ e2fsck -f vda1b.raw
e2fsck 1.44.4 (18-Aug-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
vda1b.raw: 131875/786432 files (0.1% non-contiguous), 1115854/3145216 blocks
ubuntu@superb-ram:~$ resize2fs vda1b.raw 6200M
resize2fs 1.44.4 (18-Aug-2018)
Resizing the filesystem on vda1b.raw to 1587200 (4k) blocks.
The filesystem on vda1b.raw is now 1587200 (4k) blocks long.

ubuntu@superb-ram:~$ e2fsck -f vda1b.raw
e2fsck 1.44.4 (18-Aug-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
vda1b.raw: 131875/401408 files (0.2% non-contiguous), 1089656/1587200 blocks
ubuntu@superb-ram:~$
ubuntu@superb-ram:~$
ubuntu@superb-ram:~$ sudo dpkg -l e2fsprogs libext2fs2
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=============================-===================-===================-===============================================================
ii e2fsprogs 1.44.4-2ubuntu0.2 amd64 ext2/ext3/ext4 file system utilities
ii libext2fs2:amd64 1.44.4-2ubuntu0.2 amd64 ext2/ext3/ext4 file system libraries

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package e2fsprogs - 1.44.1-1ubuntu1.1

---------------
e2fsprogs (1.44.1-1ubuntu1.1) bionic; urgency=medium

  * d/patches/0001-resize2fs-update-checksums-in-the-extent-tree-s-relo.patch:
    do the checksum update later in extent tree relocated block to denote the
    inode number change, otherwise the checksum update might be done in the old
    copy of the block. (LP: #1798562)

 -- Mathieu Trudel-Lapierre <email address hidden> Thu, 24 Jan 2019 18:11:28 -0500

Changed in e2fsprogs (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers