downloading a torrent to an encrypted home partition hangs and uses 100% CPU

[Victor Osadci]

Karmic with all updates. Home is encrypted with ecryptfs.
I have tried downloading several torrents with thansmission and rtorrent. When downloading to the encrypted home, both programs hang, use 100% CPU and I can't kill them. Downloading to /tmp, which is not encrypted works.

[Dustin Kirkland  (kirkland)]

Do you get anything in dmesg? Or elsewhere in /var/log/*?

Something from the kernel, perhaps, complaining about ecryptfs?

:-Dustin

[Victor Osadci (victor-os)]

Nothing in messages, syslog, kern.log, dmesg.

$ ps waux | grep transmission
victor 10846 82.4 0.6 82124 13712 ? Sl 20:24 5:08 transmission /home/victor/torrentname.torrent

$ killall transmission

$ ps waux | grep transmission
victor 10846 84.2 0.0 0 0 ? Zl 20:24 5:54 [transmission] <defunct>

The CPU is still 100% bussy

[u-foka (ufooka)]

Hy there!

I having the very same problem with an up2date jaunty ecryptfs home, and transmission (the included version, and after first having the issue, updated to transmissionbt ppa) as soon as transmission starts allocating the space for the torrent data, my one of my cpu cores gets 100% (transmission gets it in htop) memory cache gest filled in no time, till that the disk activity is high and after the disk stop, and transmission locks up... can only stopped by holding down the power key for several seconds, or using sysrq!! (SIGKILL does not help!)

Please lead me to provide all the information you need to fix this ass soon as possible!!

[Tyler Hicks (tyhicks)]

Hi Victor and u-foka - It looks like u-foka is correct that it occurs when rtorrent creates and extends a file big enough for the entire torrent download. If you're downloading something like a dvd iso, this causes eCryptfs to encrypt 4.7 GB of zero's and write the result to the lower filesystem.

When I tried to reproduce this, I didn't see more than 50% cpu utilization, but it did take quite some time to create and extend the file. However, rtorrent then proceeded to download the file, as expected.

You can also reproduce this behavior without rtorrent with `truncate -s 1G foo` inside of an eCryptfs mount to create and extend foo to 1 GB. It took about 1 minute and 16 seconds to complete inside of my kvm guest.

Are you all experiencing this on a machine with a relatively slow processor?

[Victor Osadci (victor-os)]

For me, this happens on a core2duo 1,6GHz notebook, so it is not exactly the fastest thing ever; I think the slow notebook hard drive has a say in this too.

I don't know how LUKS works behind the scenes, but this only happens on ecryptfs, not LUKS.

You are right, I was downloading large files, and 4,7 GB would probably take a really long time.

[Tyler Hicks (tyhicks)]

After taking a look at the call trace during a truncate that extends a file, eCryptfs could probably handle this in a more efficient way.

-> ecryptfs_setattr
 -> ecryptfs_truncate
  -> ecryptfs_write
...
   -> ecryptfs_get_locked_page
    -> ecryptfs_readpage
     -> ecryptfs_read_lower_page_segment
      -> kmap
      <- kmap
      -> ecryptfs_read_lower
      <- ecryptfs_read_lower
     <- ecryptfs_read_lower_page_segment
    <- ecryptfs_readpage
   <- ecryptfs_get_locked_page
   -> ecryptfs_encrypt_page
    -> alloc_pages_node").call
    <- alloc_pages_node").return
    -> kmap
    <- kmap
    -> ecryptfs_encrypt_extent
     -> ecryptfs_derive_iv
      -> ecryptfs_calculate_md5
      <- ecryptfs_calculate_md5
     <- ecryptfs_derive_iv
    <- ecryptfs_encrypt_extent
    -> ecryptfs_write_lower
    <- ecryptfs_write_lower
   <- ecryptfs_encrypt_page
...
  <- ecryptfs_write
 <- ecryptfs_truncate
<- ecryptfs_setattr

I don't think there's any need for the ecryptfs_readpage -> ecryptfs_read_lower_page_segment -> ecryptfs_read_lower sequence, as we know that we just need to encrypt a bunch of zeroes. Fixing that will improve performance, but the CPU is still likely to be pegged pretty high while doing all the encryption operations.

Improving the truncate path would really be a nice-to-have improvement, but I will probably not be able to devote much time to it. If someone wants to try to tackle it, I'll be more than happy to provide any help that I can.

[u-foka (ufooka)]

Hy!

I understand that it can take a while to create a 4 gb file, but what is strange, that:
- I don't think that it has to take more than ten minutes (core2duo@2ghz 2 gb ram WD3200BEVT hdd 5200 rpm)
- The disk stops working after about 3-4 seconds
- with dm-crypt (what I switched to encrypt my home instead of ecryptfs) it only takes seconds to do the job done (about 20-30) when I started to download the same file...

So I think we really have a problem, in not just slow...

[david (davidelizondo2006)]

what I think the ext4 file system
  Green is still very slow and I think it is stupid to think that besides
the stable ecryptfs that not all Czech
v ext3 and see the difference in yield figure look what happens in fedora
have many problems with ext4

  is cruel to say but I would eliminate it would cease to prueva ecryptfs
future and when ext4 is well developed that you see the 2.6.31 kernel
patches may give a temporary solution but performance would be slow despite
the efforts ....

[Dustin Kirkland  (kirkland)]

Marking invalid against ecryptfs-utils, as this is not a userspace issue.

Tyler has it on the upstream kernel side. Adding a task against Ubuntu's linux package, for tracking purposes, to make sure we pull the fix when Tyler gets it fixed upstream.

:-Dustin

[TiansHUo (tianshuo)]

I have this in top:
 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3663 user 20 0 0 0 0 Z 96 0.0 452:08.31 transmission <defunct>

killing with sudo kill -9 3663 with no avail
the parent of 3663 is init with the processid 1

the stack for 3663 is
[<c0147ea6>] do_exit+0x1c6/0x2e0
[<c0147ffa>] do_group_exit+0x3a/0xb0
[<c015403f>] get_signal_to_deliver+0x18f/0x300
[<c010304b>] do_signal+0x6b/0x160
[<c010318d>] do_notify_resume+0x4d/0x60
[<c0103448>] work_notifysig+0x13/0x1b
[<ffffffff>] 0xffffffff

in dmesg a crazy amount of the same error messages
[26030.752479] Valid eCryptfs headers not found in file header region or xattr region
[26030.752483] Either the lower file is not in a valid eCryptfs format, or the key could not be retrieved. Plaintext passthrough mode is not enabled; returning -EIO

at the very start:
 [ 60.143533] EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended

[CarloBaldassi (carlobaldassi)]

I have this problem as well, but I'm quite sure it's now just being slow, as I waited for several hours and operations didn't finish (all other operations on the encrypted filesystem take minutes at most, even with 10 GiB files). It appears that this happens when the requested file size is bigger than free RAM.

Therefore, I think this shouldn't be classified as Wishlist, but rather as a proper bug.

The description is similar to what others have observed: looking at the system monitor, it starts using CPU and disk and the cache goes up very fast; when the cache is full, disk operations stop, but the CPU keeps running at 100% and the program becomes unresponsive (both Transmission and Vuze). If the processes are killed (kill -9), they keep running as zombies at 100% and reboot is the only way to stop them (BTW from what I read this in itself could be considered a bug, as zombie processes shouldn't keep the CPU busy as far as I know). Swap is never used during the process.

My current workaround is to create an unencrypted directory under /home/ with permissions set to my user, and use that one for torrent downloads (clearly not an optimal solution, but still better than using /tmp).

Some additional details on my configuration:
I'm using Ubuntu Lucid 10.4, clean install, ext4 filesystem, encryption selected at installation, up-to-date Kernel (Linux 2.6.32-22-generic #36-Ubuntu SMP i686 GNU/Linux)

[TiansHUo (tianshuo)]

Under top, init with pid 0 is the one that has 100 cpu, so the second bug
isn't one.

On Wed, Jun 16, 2010 at 8:54 PM, CarloBaldassi <email address hidden>wrote:

> I have this problem as well, but I'm quite sure it's now just being
> slow, as I waited for several hours and operations didn't finish (all
> other operations on the encrypted filesystem take minutes at most, even
> with 10 GiB files). It appears that this happens when the requested file
> size is bigger than free RAM.
>
> Therefore, I think this shouldn't be classified as Wishlist, but rather
> as a proper bug.
>
> The description is similar to what others have observed: looking at the
> system monitor, it starts using CPU and disk and the cache goes up very
> fast; when the cache is full, disk operations stop, but the CPU keeps
> running at 100% and the program becomes unresponsive (both Transmission
> and Vuze). If the processes are killed (kill -9), they keep running as
> zombies at 100% and reboot is the only way to stop them (BTW from what I
> read this in itself could be considered a bug, as zombie processes
> shouldn't keep the CPU busy as far as I know). Swap is never used during
> the process.
>
> My current workaround is to create an unencrypted directory under /home/
> with permissions set to my user, and use that one for torrent downloads
> (clearly not an optimal solution, but still better than using /tmp).
>
> Some additional details on my configuration:
> I'm using Ubuntu Lucid 10.4, clean install, ext4 filesystem, encryption
> selected at installation, up-to-date Kernel (Linux 2.6.32-22-generic
> #36-Ubuntu SMP i686 GNU/Linux)
>
> --
> downloading a torrent to an encrypted home partition hangs and uses 100%
> CPU
> https://bugs.launchpad.net/bugs/431975
> You received this bug notification because you are a direct subscriber
> of the bug.
>

--
TiansHUo

[Arthur Țițeică (arthur-titeica)]

Just for the sake of it - a truecrypt encrypted device works smoothly with torrents and such.

[Jeremy Foshee (jeremyfoshee)]

closing the kernel task as this does not seem to be a kernel bug.

~JFo

[Dustin Kirkland  (kirkland)]

Hey JFo,

Actually, it is a kernel bug, and Tyler Hicks (the upstream maintainer) sees the problem, but hasn't had time to fix it. Ubuntu kernels are affected, though we're not working on it on the Ubuntu side. We're just waiting for upstream to commit a fix.

Thanks!
Dustin

[Lehi David Anson Toskin (fancycakes)]

I can attest to this in Lucid with ecryptfs encrypted /home/user. I'm not sure about Transmission or Vuze, but in rtorrent, there's a way to save the machine and still have large torrent downloads; add this to your ~/.rtorrent.rc:

split_file_size = 8G # or 4G or 3G, depending on your system (mine is good at 3G)

And then when the torrent is completed simply run 'cat file_1 file_2 ... > file_x' and then you're good to go.

[gcc (chris+ubuntu-qwirx)]

Please could someone upgrade this from wishlist? It is a bug, unkillable processes require a system reboot which is very annoying, this is likely to contribute to people (including me) regarding Ubuntu LTS as unstable, and I'm now regretting choosing to encrypt my homedir during the installation because of this bug.

[Arthur Țițeică (arthur-titeica)]

1.5 years after the initial bug report and Ubuntu guys still look aside and still offer this as an option in the installer with no warning.

This is NOT about torrents but about data in your home directory, you know... sh*t you work with!

[Dustin Kirkland  (kirkland)]

Constructive. Thanks.

[Brad Figg (brad-figg)]

This bug was filed against a series that is no longer supported and so is being marked as Won't Fix. If this issue still exists in a supported series, please file a new bug.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Malcolm Greaves (greaves-malcolm)]

Hi Brad,

Does this mean that drive encryption is no longer supported? Or, that drive
encryption is still supported and this bug will never be fixed?

-Malcolm
On Jul 14, 2011 1:18 PM, "Brad Figg" <email address hidden> wrote:
> This bug was filed against a series that is no longer supported and so
> is being marked as Won't Fix. If this issue still exists in a supported
> series, please file a new bug.
>
> This change has been made by an automated script, maintained by the
> Ubuntu Kernel Team.
>
> ** Changed in: linux (Ubuntu)
> Status: Triaged => Won't Fix
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/431975
>
> Title:
> downloading a torrent to an encrypted home partition hangs and uses
> 100% CPU
>
> Status in eCryptfs - Enterprise Cryptographic Filesystem:
> Triaged
> Status in “ecryptfs-utils” package in Ubuntu:
> Invalid
> Status in “linux” package in Ubuntu:
> Won't Fix
>
> Bug description:
> Karmic with all updates. Home is encrypted with ecryptfs.
> I have tried downloading several torrents with thansmission and rtorrent.
When downloading to the encrypted home, both programs hang, use 100% CPU and
I can't kill them. Downloading to /tmp, which is not encrypted works.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ecryptfs/+bug/431975/+subscriptions

[gcc (chris+ubuntu-qwirx)]

This bug DOES affect lucid which is still supported. I can't see how to mark that in Launchpad or reopen the bug. Please could someone do that?

Why would we have to open a new bug? That doesn't make any sense.

[Daniel Dietrich (shaddowy2)]

I think the automated scripts are far to aggressive while trying to "fix" the bugs. Please change the importance of this problem to something else than "wishlist", as it's a bug. In case Tyler found some time and the bug got already fixed, please close it. Maybe Dustin or Tyler can change the status? If you need some more information or help I'm pretty sure the users are willing to help you.
I always wondered what made my hard disk so slow, and after reading here and about the other trailing bug I am done with ecryptfs. I mean, you should not offer the feature on the install disc without warning since years when there are severe bugs. I will go back to luks for now.

[Sampo (sampo)]

This bug continues to exist.

What: Zombie process consuming 100% cpu, parent pid 1 (init). Unkillable and never goes away (waited 30h).

How to trigger: Use transmission (or any other software) to create over 4GB file (5GB torrent in my case) on
encrypted home directory.

It is NOT filesystem or disk performance problem (filesystem continues to perform for all other purposes and
I waited over 30h for it to possibly sort itself out). It really is something unduely stuck in kernel.

It is probably a bug in many places:

1. Bug in init for not reaping up the zombie
2. Bug in kernel for zombie process consuming resources other than process table slot.
3. Bug in ecryptfs kernel module (or possibly ext4 or their mutual interaction) for getting in the trouble in the first place

Distribution: LinuxMint 12 "lisa" based on Ubuntu 11.10 "Oneric" based on Debian?
uname -a
Linux saz 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:50:42 UTC 2011 i686 i686 i386 GNU/Linux
mount |grep ecrypt
/home/sk/.Private on /home/sk type ecryptfs (ecryptfs_check_dev_ruid,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs,ecryptfs_sig=42979cfc6e80278a,ecryptfs_fnek_sig=6261c2e7178a57d3)

It is clear that the bug manifests with officially supported distribution release so the robot
was wrong in closing the bug. Please see https://bugs.launchpad.net/launchpad/+bug/913740
for my attempt to expand the bug to new distribution.

I agree that it is reckless to advertise in install process the ability to encrypt home directories
while this bug persists. Only known workaround is not to encrypt home directory. Bugs like
this help to educate users that encryption is a bad thing - exactly opposite of what was
probably intended by including encrypt home directories feature in the first place.

Cheers,
--Sampo

[Adam Porter (alphapapa)]

This is clearly a bug, not a wish list item. I can't change the importance, but since it's clearly been incorrectly triaged, I can change the status from Triaged to Confirmed.

This is yet another example of Ubuntu pushing out software with known, serious bugs. In this case, Ubuntu's pushed out FOUR more releases with this bug. This bug causes system instability, data loss, and requires a hard reset...how much more serious can it get? Can we come up with a security angle? Maybe then it will get fixed...

Hey, at the least it's a local DoS bug. Does that qualify?

[Dustin Kirkland  (kirkland)]

Tyler has a patch for this under review on the mailing list. I'd suggest that we consider this for SRU to support Ubuntu Linux kernels once its upstream, tested and verified.

Marking in-progress.

[Dustin Kirkland  (kirkland)]

Nominating for Precise. We should be able to land this there fairly easily. Marking HIGH as this is pretty easy to reproduce, and does hang a system.

[Tyler Hicks (tyhicks)]

These commits were released in 3.3-rc2:

684a3ff7e69acc7c678d1a1394fe9e757993fd34
5e6f0d769017cc49207ef56996e42363ec26c1f0
a261a03904849c3df50bd0300efb7fb3f865137d

They address the hangs and CPU utilization mentioned in this bug. Unfortunately, they don't address is the poor performance in the truncate path.

[Amit (g2291632-amit)]

where can i get this fix...

Thanks & regards,
 Amit Garg

[Tyler Hicks (tyhicks)]

On 2012-02-08 17:55:02, Amit wrote:
> where can i get this fix...

Depends on what you need.

If you just want the patches, append the commit id's I posted in comment
28 to http://git.kernel.org/linus/ like this:

http://git.kernel.org/linus/684a3ff7e69acc7c678d1a1394fe9e757993fd34

If you want a released development kernel with the patches applied, you
can download and build 3.3-rc2.

If you want a released stable kernel with the patches applied, you'll
have to wait for 3.2.6 or 3.0.20 to be released.

If you want an official distro kernel, you'll have to wait for your
distro to pick the patches up from the -stable tree and release a kernel
update.

[Amit (g2291632-amit)]

i just want to fix ecryptfs.
what i should do .?

Thanks & regards,
 Amit Garg

[Amit (g2291632-amit)]

ok thanks got that change .......!!

--- a/fs/ecryptfs/read_write.c
+++ b/fs/ecryptfs/read_write.c
@@ -130,13 +130,13 @@ int ecryptfs_write(struct inode *ecryptfs_inode, char *data, loff_t offset,
                pgoff_t ecryptfs_page_idx = (pos >> PAGE_CACHE_SHIFT);
                size_t start_offset_in_page = (pos & ~PAGE_CACHE_MASK);
                size_t num_bytes = (PAGE_CACHE_SIZE - start_offset_in_page);
- size_t total_remaining_bytes = ((offset + size) - pos);
+ loff_t total_remaining_bytes = ((offset + size) - pos);

                if (num_bytes > total_remaining_bytes)
                        num_bytes = total_remaining_bytes;
                if (pos < offset) {
                        /* remaining zeros to write, up to destination offset */
- size_t total_remaining_zeros = (offset - pos);
+ loff_t total_remaining_zeros = (offset - pos);

                        if (num_bytes > total_remaining_zeros)
                                num_bytes = total_remaining_zeros;

[Tyler Hicks (tyhicks)]

On 2012-02-08 18:17:49, Amit wrote:
> i just want to fix ecryptfs.
> what i should do .?

You can pull down those patches, apply them, and build a kernel yourself
(something that you'd have to research and do on your own) or you'll
have to wait for your distro to release a kernel update with these
fixes.

This is getting off-topic for this bug report. If you have further
questions, please ask them in irc://irc.oftc.net/#ecryptfs

[Amit (g2291632-amit)]

 have change in kernel as you talked about some kernel change.
but still my device is hang with ecryptfs .

please help me.

Thanks & Regards,
Amit

[Amit (g2291632-amit)]

and its hanging in truncate -s 4G dummy

so please resolve this what is happening with ecryptfs.

Thanks & Regards,
Amit Garg

[Chew Jian Yue (chewjianyue)]

Sometimes it might be because of your hardware failure or overloading of the torrent application.

[Joseph Salisbury (jsalisbury)]

This bug was filed against a series that is no longer supported and so is being marked as Won't Fix. If this issue still exists in a supported series, please file a new bug.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Nelson Castillo (nelsoneci)]

Part of the issue (why it takes so long) is that ecryptfs does not support sparse files. So an operation that is fast in an unencrypted fs is slow with ecryptfs.

$ time truncate -s 2000M file.img # ecryptfs

real 0m26.239s
user 0m0.001s
sys 0m22.984s

$ time truncate -s 2000M file.img # unencrypted

real 0m0.019s
user 0m0.001s
sys 0m0.002s

I can kill the "truncate" process if I wish to. I wonder what makes the process unkillable in the scenario of this bug.

The unkillable issue also happens with qemu-img.
See https://bugs.launchpad.net/ecryptfs/+bug/936706