[git] file blocks duplicated at the end of the file

Bug #842647 reported by iroli on 2011-09-06
110
This bug affects 22 people
Affects Status Importance Assigned to Milestone
eCryptfs
High
Tyler Hicks
ecryptfs-utils (Ubuntu)
High
Tyler Hicks
Precise
High
Tyler Hicks

Bug Description

I am not really sure wether ecryptsfs is causing this or not, but here is my setup:
ext4 filesystem
ecryptfs $HOME
a somewhat-sized git repository in $HOME

I just issued a git -status and learnt about modifications to a file I never touched. This happens for the second time now (in 3 weeks or so). I examined the file and there are 3 block of 2042 0-bytes inserted.

I attach the corrupted file to this bugreport, but it seems as if blocks are repeated towards the end and partially 0'ed out.

I have no issues with the system indicating to harddisk or memory problems, and dmesg only brought up this:

ecryptfs_read_and_validate_header_region: Error reading header region; rc = [-4]

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: ecryptfs-utils 83-0ubuntu3.2.10.10.2
ProcVersionSignature: Ubuntu 2.6.35-30.56-generic 2.6.35.13
Uname: Linux 2.6.35-30-generic x86_64
NonfreeKernelModules: nvidia
Architecture: amd64
Date: Tue Sep 6 14:40:18 2011
EcryptfsInUse: Yes
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Release amd64 (20101007)
ProcEnviron:
 LANGUAGE=en_GB:en
 LANG=de_DE.utf8
 SHELL=/bin/bash
SourcePackage: ecryptfs-utils

iroli (roland-lezuo) wrote :
iroli (roland-lezuo) wrote :

Another file got corrupted. dmesg shows this:

[55148.424563] ecryptfs_decrypt_page: Error attempting to read lower page; rc = [-4]
[55148.424568] ecryptfs_readpage: Error decrypting page; rc = [-4]
[55148.424579] ecryptfs_decrypt_page: Error attempting to read lower page; rc = [-4]
[55148.424583] ecryptfs_readpage: Error decrypting page; rc = [-4]

Changed in ecryptfs-utils (Ubuntu):
status: New → Confirmed
iroli (roland-lezuo) wrote :

For ways to reproduce see Bug #490005

iroli (roland-lezuo) wrote :

I found out howto reproduce this on my system:

Doing a "git status -uno" and aborting (ctrl+c) give you a high chance of a corrupt file afterwards.

iroli (roland-lezuo) wrote :

Hi I upgraded to 11.10 and the bug is still somewhat there:

iroli@iroli-thinkpad:~/projects/c3pro/repo/foo_git/trunk$ git status -uno
# On branch matcher_gen_casm
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: app/compiler/src/misc/some_stuff.cpp
# modified: app/csim/check/exit_status/args.xml <!-- This one was not modified!
# modified: arch/aaa/mem_address_writeback_instructions.xml
# modified: arch/aaa/minstr_set_register_interface.xml
# modified: arch/aaa/minstr_set_stale.xml
#
no changes added to commit (use "git add" and/or "git commit -a")

iroli@iroli-thinkpad:~/projects/c3pro/repo/foo_git/trunk$ git diff app/csim/check/exit_status/args.xml
error: short read No such file or directory
error: app/csim/check/exit_status/args.xml: failed to insert into database
fatal: cannot hash app/csim/check/exit_status/args.xml

iroli@iroli-thinkpad:~/projects/c3pro/repo/foo_git/trunk$ git status -uno
# On branch matcher_gen_casm
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: app/compiler/src/misc/some_stuff.cpp
# modified: arch/aaa/mem_address_writeback_instructions.xml <!-- no the file is not marked as modified!
# modified: arch/aaa/minstr_set_register_interface.xml
# modified: arch/aaa/minstr_set_stale.xml
#
no changes added to commit (use "git add" and/or "git commit -a")

iroli@iroli-thinkpad:~/projects/c3pro/repo/foo_git/trunk$

Dustin Kirkland  (kirkland) wrote :

Tyler, sending this bug your way, hoping you can add some comments.

Changed in ecryptfs-utils (Ubuntu):
importance: Undecided → High
assignee: nobody → Tyler Hicks (tyhicks)
summary: - file blocks duplicated at the end of the file
+ [git] file blocks duplicated at the end of the file
tags: added: git
iroli (roland-lezuo) wrote :

just ran into the short read again

Tyler Hicks (tyhicks) wrote :

Thanks - I can reproduce this and have tracked down the problem. We're missing an eCryptfs metadata read, in the ecryptfs_getattr() path, after a metadata read in the lookup() path fails. This results in a stat() call returning an incorrect inode size and then subsequent read()'s not adding up to the correct file size.

I've started on a fix, but will need a bit more time, as this code is fragile and needs improvements outside of this bug.

Changed in ecryptfs-utils (Ubuntu):
status: Confirmed → In Progress
Tyler Hicks (tyhicks) wrote :

Confirmed with current 3.2-rc7 kernel.

Changed in ecryptfs:
assignee: nobody → Tyler Hicks (tyhicks)
importance: Undecided → High
status: New → In Progress
Changed in ecryptfs-utils (Ubuntu):
status: In Progress → Triaged
tags: added: rls-p-tracking
Changed in ecryptfs-utils (Ubuntu Precise):
milestone: none → ubuntu-12.04-beta-1
tags: added: rls-mgr-p-tracking
Changed in ecryptfs-utils (Ubuntu Precise):
milestone: ubuntu-12.04-beta-1 → ubuntu-12.04-beta-2
Tyler Hicks (tyhicks) wrote :

Unfortunately, I'm not able to consistently reproduce this anymore. I can trigger it once every so often, but not consistent enough to give me much confidence. I've developed a proposed patch and pushed it to a temporary location here:

http://git.kernel.org/?p=linux/kernel/git/tyhicks/ecryptfs.git;a=commitdiff;h=4b08d9cf379be83641b8d7e0eea56e14a8e20cc3

I'm in the process of building test kernels for Precise and 11.10 in hopes that I can get some testing from iroli and others affected before I push the fix upstream. I'll provide links to the test kernels when they're done building.

Tyler Hicks (tyhicks) wrote :

The patch I linked to in comment 10 breaks plaintext passthrough support. The logic around handling plaintext passthrough and the metadata-in-xattrs feature is very delicate in this part of the code. I'd love to deprecate/remove those two "features" in future kernels, as they're rarely used by anyone, but we need a fix that is backportable.

I'm going to think this through a little more and I hope to have a new patch this later this evening.

Plaintext passthrough has been broken by design for a very long time,
and practically impossible to support.

I'm in favor of deprecating the current implementation and hopefully
replacing it with a better approach eventually.

Changed in ecryptfs-utils (Ubuntu Precise):
milestone: ubuntu-12.04-beta-2 → ubuntu-12.04
Tyler Hicks (tyhicks) wrote :

Proposed fix against the current upstream kernel (3.4-rc0).

Tyler Hicks (tyhicks) wrote :

I'm still not able to consistently reproduce this issue anymore. I've built amd64 test kernels for Oneiric and Precise. For those affected, please give the kernels a try and leave a comment as to whether or not the bug is fixed.

http://people.canonical.com/~tyhicks/ecryptfs/842647/

The attachment "[PATCH] eCryptfs: Ensure i_size is set in ecryptfs_getattr()" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

tags: added: patch
Changed in ecryptfs-utils (Ubuntu Precise):
milestone: ubuntu-12.04 → none
tags: removed: rls-mgr-p-tracking rls-p-tracking
theghost (theghost) wrote :

I still get this [1] running 3.10rc7 without using git. Is there a way to find out which files are/got corrupted ?

[1] ecryptfs_decrypt_page: Error attempting to read lower page; rc = [-4]
 ecryptfs_readpage: Error decrypting page; rc = [-4]

3.11.0-13-generic (i386), same problem.

I can confirm that the following error message still happens on 3.11.0-19-generic (x86_64)

[66675.577535] ecryptfs_decrypt_page: Error attempting to read lower page; rc = [-4]
[66675.577540] ecryptfs_readpage: Error decrypting page; rc = [-4]

My workload is mostly git based

Gioele Barabucci (gioele) wrote :

It looks like the same problem is present also in Ubuntu 14.04 LTS. After stating a 190k files with `find` and `grep`, the kernel log is full of

    ecryptfs_decrypt_page: Error attempting to read lower page; rc = [-4]
    ecryptfs_readpage: Error decrypting page; rc = [-4]

Couldn't these printk from ecryptfs be augmented with some additional information? At least to make sure that we are all hitting the same bug.

Any news? The oldest bug insofar.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers