ecryptfs_encrypt_page: Error attempting to write lower page (regression)

[arty]

NOTE: A test case for this bug has been created at tests/kernel/mmap-close.sh (revno 729) in the upstream ecryptfs-utils project.

After upgrading to 3.2.0-30 I started seeing error messages like this in my syslog:

Sep 7 11:56:28 artemy-tregubenko kernel: [ 318.675991] ecryptfs_encrypt_page: Error attempting to write lower page; rc = [-5]
Sep 7 11:56:28 artemy-tregubenko kernel: [ 318.675997] ecryptfs_writepage: Error encrypting page (upper index [0x0000000000000000])
Sep 7 11:56:28 artemy-tregubenko kernel: [ 318.676058] ecryptfs_encrypt_page: Error attempting to write lower page; rc = [-5]
Sep 7 11:56:28 artemy-tregubenko kernel: [ 318.676061] ecryptfs_writepage: Error encrypting page (upper index [0x0000000000000001])
Sep 7 11:56:28 artemy-tregubenko kernel: [ 318.676119] ecryptfs_encrypt_page: Error attempting to write lower page; rc = [-5]
Sep 7 11:56:28 artemy-tregubenko kernel: [ 318.676122] ecryptfs_writepage: Error encrypting page (upper index [0x0000000000000000])
Sep 7 11:56:28 artemy-tregubenko kernel: [ 318.676179] ecryptfs_encrypt_page: Error attempting to write lower page; rc = [-5]
Sep 7 11:56:28 artemy-tregubenko kernel: [ 318.676182] ecryptfs_writepage: Error encrypting page (upper index [0x0000000000000001])

I have a one-partition setup and my whole home folder is encrypted. I did a fsck and that didn't help. I removed zero-size files from ~/.Private and that didn't help. I backed data, deleted user and its home folder, created user again and copied data back, and error was still there.

Booting to previous kernel 3.2.0-29 solved the issue. This is why I consider this bug a regression.

Now I know there's a bug #870326 which is pretty much identical to this. I couldn't find "reopen" button however and I don't know how you guys like to track regressions in launchpad. I'm sorry for creating a duplicate. Please advise me what to do? If I just add new comment to the resolved bug, won't it get lost?

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: ecryptfs-utils 96-0ubuntu3
ProcVersionSignature: Ubuntu 3.2.0-30.48-generic 3.2.27
Uname: Linux 3.2.0-30-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.0.1-0ubuntu12
Architecture: amd64
Date: Fri Sep 7 11:56:48 2012
EcryptfsInUse: Yes
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64 (20110427.1)
ProcEnviron:
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/zsh
SourcePackage: ecryptfs-utils
UpgradeStatus: Upgraded to precise on 2012-08-11 (26 days ago)

[Tyler Hicks (tyhicks)]

Hello - Thanks for reporting this.

This part of the eCryptfs code did change between 3.2.0-29 and 3.2.0-30, but I don't immediately see what the problem could be.

Can you give some more details about what you're doing (any commands that you're running, what apps you're using, etc.) when you see these messages printed to logs? It would help in reproducing this bug and finding the problem. Thanks!

[arty (me-arty)]

I noticed that bug because my IntelliJ IDEA started freezing while opening the project. Thankfully it shows some information about background processes it runs, so I knew the problem was in indexing process. That process goes through the files in the project and creates indexes of those in IDEA's folder, so it uses filesystem quite intensively. First thing I tried to do was to delete indexes folder, and that didn't work: one of the subfolders was invulnerable. I couldn't delete it neither from command line nor from Nautilus. I could move it to trash, however. That's when I started to think something's wrong with the filesystem.

While I was reporting this bug I didn't have IDEA running, and still was seeing new errors in the log. I can't name all the processes at the moment, as this was happening on my working notebook. But besides normal Ubuntu 12.04 processes I had Opera, Chrome, Skype, Dropbox, Telepathy running.

[wdesmet (kromagg)]

I have the same problem and noticed it in the same way. Only happened after the upgrade, so it must be a regressions. If it helps I believe the idea file indexer also uses memory mapping extensively.

[Etienne Neveu (neveue)]

Got the same bug:

Sep 12 15:39:16 eneveu kernel: [ 1223.824052] ecryptfs_encrypt_page: Error attempting to write lower page; rc = [-5]
Sep 12 15:39:16 eneveu kernel: [ 1223.824058] ecryptfs_writepage: Error encrypting page (upper index [0x00000000000001c7])

Here is the IntelliJ IDEA exception, in case it helps:

2012-09-12 15:39:16,601 [ 181592] ERROR - j.util.io.PersistentEnumerator - java.io.IOException: Input/output error
java.lang.RuntimeException: java.io.IOException: Input/output error
        at com.intellij.util.io.ResizeableMappedFile.resize(ResizeableMappedFile.java:60)
        at com.intellij.util.io.ResizeableMappedFile.expand(ResizeableMappedFile.java:74)
        at com.intellij.util.io.ResizeableMappedFile.ensureSize(ResizeableMappedFile.java:68)
        at com.intellij.util.io.ResizeableMappedFile.put(ResizeableMappedFile.java:172)
        at com.intellij.util.io.PersistentEnumerator.writeData(PersistentEnumerator.java:435)
        at com.intellij.util.io.PersistentEnumerator.enumerateImpl(PersistentEnumerator.java:331)
        at com.intellij.util.io.PersistentEnumerator.enumerate(PersistentEnumerator.java:233)
        at com.intellij.openapi.vfs.newvfs.persistent.FSRecords.setName(FSRecords.java:834)
        at com.intellij.openapi.vfs.newvfs.persistent.PersistentFS.a(PersistentFS.java:288)
        at com.intellij.openapi.vfs.newvfs.persistent.PersistentFS.getId(PersistentFS.java:364)
        at com.intellij.openapi.vfs.newvfs.impl.VirtualDirectoryImpl.b(VirtualDirectoryImpl.java:119)
        at com.intellij.openapi.vfs.newvfs.impl.VirtualDirectoryImpl.a(VirtualDirectoryImpl.java:63)
        at com.intellij.openapi.vfs.newvfs.impl.VirtualDirectoryImpl.refreshAndFindChild(VirtualDirectoryImpl.java:170)
        at com.intellij.openapi.vfs.newvfs.NewVirtualFileSystem.refreshAndFindFileByPath(NewVirtualFileSystem.java:105)
        at com.intellij.openapi.vfs.impl.local.LocalFileSystemBase.refreshAndFindFileByPath(LocalFileSystemBase.java:66)
        at com.intellij.openapi.vfs.impl.jar.JarHandler.refreshLocalFileForJar(JarHandler.java:78)
        at com.intellij.openapi.vfs.impl.jar.JarFileSystemImpl$2.run(JarFileSystemImpl.java:196)
        at com.intellij.openapi.application.impl.LaterInvocator$FlushQueue.run(LaterInvocator.java:332)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:251)
        [...]
Caused by: java.io.IOException: Input/output error
        at java.io.RandomAccessFile.close0(Native Method)
        at java.io.RandomAccessFile.close(RandomAccessFile.java:594)
        at com.intellij.util.io.PagedFileStorage.resizeFile(PagedFileStorage.java:312)
        at com.intellij.util.io.PagedFileStorage.resize(PagedFileStorage.java:298)
        at com.intellij.util.io.ResizeableMappedFile.resize(ResizeableMappedFile.java:57)
        ... 35 more

Going back to 3.2.0-29for now.

[Christopher Taylor (timrod)]

FWIW, I'm seeing the same issue; IDEA hangs while indexing. I did a clean install of 12.04 today and the issue persists.

[Tyler Hicks (tyhicks)]

I've built a set of amd64 test kernels, for Quantal and Precise, with a proposed fix for this bug. They can be found here:

http://people.canonical.com/~tyhicks/ecryptfs/fixes/

If any of the affected users could give this test kernel a shot and report if the bug is resolved, I'd appreciate it!

[arty (me-arty)]

Runs fine for me so far, I'll be keeping an eye on it and report failures, should any happen.

[Colin Ian King (colin-king)]

I've given this a good soak testing on Quantal, Precise + applied the patches to Oneiric and for each tested it against ext2, ext3, ext4, xfs and btrfs lower file systems, and it does solve this bug. Thanks Tyler.

[arty (me-arty)]

I haven't seen any errors during the whole day of normal usage, so I suppose the bug is fixed in this kernel build.

[Tyler Hicks (tyhicks)]

Thanks for your testing, Colin and Arty.

[Colin Ian King (colin-king)]

SRU for Oneiric and Precise, and apply to Quantal too.

== SRU Justification ==

== Impact ==

A regression was caused by commit:

821f749 eCryptfs: Revert to a writethrough cache model

That patch reverted some code (specifically, 32001d6f) that was
necessary to properly handle open() -> mmap() -> close() -> dirty pages
-> munmap(), because the lower file could be closed before the dirty
pages are written out.

This revert unfortunately causes errors:

ecryptfs_encrypt_page: Error attempting to write lower page

== Fix ==

Apply commits 7149f2558d5b5b988726662fe58b1c388337805b and
64e6651dcc10e9d2cc6230208a8e6c2cfd19ae18.

== Test Case ==

Can be tested on various file systems using the ecryptfs tests (from
lp:ecryptfs).

sudo mkdir /tmp/image /lower /upper
sudo ./tests/run_tests.sh -K -c safe -b 1000000 -D /tmp/image \
-l /lower -u /upper -f ext2,ext3,ext4,xfs,btrfs -t mmap-close.sh

Without the fix, this fails, with the fix it passes.

[Etienne Neveu (neveue)]

Not sure if it is useful to comment now, but the proposed fix also worked for me. I patched Friday afternoon, and IntelliJ hasn't crashed since.

[Tyler Hicks (tyhicks)]

This was fixed in 3.6-rc6:

http://git.kernel.org/linus/1547cb80db5587c465be609b9a15bfb2dfd73099

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.5.0-15.20

---------------
linux (3.5.0-15.20) quantal-proposed; urgency=low

  [ Tim Gardner ]

  * rebase to v3.5.4
  * SAUCE: CONFIG_HID_BATTERY_STRENGTH=y
    - LP: #1003090

  [ Upstream Kernel Changes ]

  * eCryptfs: Copy up attributes of the lower target inode after rename
    - LP: #561129
  * eCryptfs: Write out all dirty pages just before releasing the lower
    file
    - LP: #1047261
  * eCryptfs: Call lower ->flush() from ecryptfs_flush()
    - LP: #1047261
  * af_netlink: force credentials passing [CVE-2012-3520]
    - LP: #1052097
    - CVE-2012-3520
  * drm/i915: clarify IBX dp workaround
    - LP: #1011440
  * drm/i915: Implement w/a for sporadic read failures on waking from rc6
    - LP: #1011440
  * drm/i915: support Haswell force waking
    - LP: #1011440
  * drm/i915: add RPS configuration for Haswell
    - LP: #1011440
  * drm/i915: enable RC6 by default on Haswell
    - LP: #1011440
  * drm/i915: introduce haswell_init_clock_gating
    - LP: #1011440
  * drm/i915: enable RC6 workaround on Haswell
    - LP: #1011440
  * drm/i915: re-initialize DDI buffer translations after resume
    - LP: #1011440
  * drm/i915: fix PIPE_DDI_PORT_MASK
    - LP: #1011440
  * drm/i915: try to train DP even harder
    - LP: #1011440
  * drm/i915: add more Haswell PCI IDs
    - LP: #1011440
  * rebase to v3.5.4
    - LP: #1038651
 -- Leann Ogasawara <email address hidden> Mon, 17 Sep 2012 13:41:39 -0700

[Colin Ian King (colin-king)]

Tested on oneiric -proposed with ext2,ext3,ext4,xfs and btrfs lower file systems:

uname -a
Linux ubuntu 3.0.0-26-server #43-Ubuntu SMP Tue Sep 25 17:37:40 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@ubuntu:~/ecryptfs$ sudo ./tests/run_tests.sh -K -c safe -b 1000000 -D /tmp/image -l /lower -u /upper -t mmap-close.sh -f ext2,ext3,ext4,xfs,btrfs
Running eCryptfs filesystem tests on ext2
mmap-close pass
Running eCryptfs filesystem tests on ext3
mmap-close pass
Running eCryptfs filesystem tests on ext4
mmap-close pass
Running eCryptfs filesystem tests on xfs
mmap-close pass
Running eCryptfs filesystem tests on btrfs
mmap-close pass

Test Summary:
5 passed
0 failed

[Luis Henriques (henrix)]

As per comment #16, I'm tagging this bug as verified in oneiric.

[Luis Henriques (henrix)]

This bug is awaiting verification that the kernel for Precise in -proposed solves the problem (3.2.0-32.51). Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-precise' to 'verification-done-precise'.

If verification is not done by one week from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Colin Ian King (colin-king)]

Tested on Precise -proposed with ext2,ext3,ext4,xfs and btrfs lower file systems:

uname -a
Linux ubuntu 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

sudo ./tests/run_tests.sh -K -c safe -b 1000000 -D /tmp/image -l /lower -u /upper -f ext2,ext3,ext4,xfs,btrfs -t mmap-close.sh
Running eCryptfs filesystem tests on ext2
mmap-close pass
Running eCryptfs filesystem tests on ext3
mmap-close pass
Running eCryptfs filesystem tests on ext4
mmap-close pass
Running eCryptfs filesystem tests on xfs
mmap-close pass
Running eCryptfs filesystem tests on btrfs
mmap-close pass

Test Summary:
5 passed
0 failed

[Adam Conrad (adconrad)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Bartosz Radaczyński (radaczynski)]

FWIW: the workaround is to move the Idea folder to unencrypted directory and create a symlink. Not only Idea does not crash anymore, but the indexing is also much, much faster

[Jaka Hudoklin (jakahudoklin)]

I'm still seeing this bug when using vmware. It just happened when image size got over 18GB. Even the ext4 fs i have on lower gets corrupted.

Linux offlinehacker-laptop 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux,
normal laptop hard drive.

Distributor ID: Ubuntu
Description: Ubuntu 12.04.1 LTS
Release: 12.04
Codename: precise,
fully upgraded

Sorry to say, but this is not production ready for me(and it's ubuntu to blame to make it look production ready), hope it was great playing with my data, going for luks now!

[Tyler Hicks (tyhicks)]

On 2012-10-17 10:37:29, Jaka Hudoklin wrote:
> I'm still seeing this bug when using vmware. It just happened when image
> size got over 18GB. Even the ext4 fs i have on lower gets corrupted.

I can't see how this bug could ever cause corruption of the lower ext4
filesystem. I think you may have hit a different issue. Can you please
open a new bug with detailed instructions on your setup, along with what
indications there were of an eCryptfs bug and how you discovered that
the lower filesystem was corrupted? Thank you!

[somekool (somekool)]

5 years later, on ubuntu 18.10, I am seeing this same error.

```
Jan 18 11:19:47 mathieu-FMVWB3U28 kernel: [10270.829309] ecryptfs_encrypt_page: Error attempting to write lower page; rc = [-28]
Jan 18 11:19:47 mathieu-FMVWB3U28 kernel: [10270.829311] ecryptfs_write_end: Error encrypting page (upper index [0x0000000000005249])
Jan 18 11:19:47 mathieu-FMVWB3U28 kernel: [10271.004494] ecryptfs_encrypt_page: Error attempting to write lower page; rc = [-28]
Jan 18 11:19:47 mathieu-FMVWB3U28 kernel: [10271.004498] ecryptfs_write_end: Error encrypting page (upper index [0x0000000000005249])
Jan 18 11:19:47 mathieu-FMVWB3U28 kernel: [10271.004552] ecryptfs_encrypt_page: Error attempting to write lower page; rc = [-28]
Jan 18 11:19:47 mathieu-FMVWB3U28 kernel: [10271.004554] ecryptfs_write_end: Error encrypting page (upper index [0x0000000000005249])
Jan 18 11:19:47 mathieu-FMVWB3U28 kernel: [10271.004592] ecryptfs_encrypt_page: Error attempting to write lower page; rc = [-28]
```

my .Private folder is at 100%, reported by `df`, almost twice as much as what `du` says....
I am moving files away, but it does not decrease...