Rapid depletion of entropy pool

[agent 8131]

I was noticing that on several of my servers the available entropy has been exceedingly low for the last 6-7 months. My guess is this problem began with Ubuntu 9.10 and continues in Ubuntu 10.04. I came across some useful information here:

http://lkml.org/lkml/2010/4/5/19

And I confirmed that running:

watch cat /proc/sys/kernel/random/entropy_avail

will rapidly deplete the entropy pool. But running the python script:

import sys, time
while True:
  sys.stdout.write(open('/proc/sys/kernel/random/entropy_avail', 'r').read())
  time.sleep(1)

will not rapidly deplete the entropy pool. This seems to support the hypothesis that entropy is being drained with each launch of a process which has been linked to the glibc randomized stack protector. Some information about that can be found here:

http://sourceware.org/ml/libc-alpha/2008-10/msg00006.html

As many people who have run virtual servers can attest, low entropy on a server can cause a number of difficult to diagnose performance problems as processes block trying to access /dev/random. Low entropy may also lead to a reduction in security for various cryptographic services.

I'm not an expert in these matters and have limited ability to test as many of my servers are running older versions but it does appear that those older versions do not have this behavior. This could also be a kernel issue but I thought I would start here and see if others can replicate this problem and help in diagnosing the issue.

[agent 8131 (agent-8131)]

More information:

---
http://www.linuxfromscratch.org/hints/downloads/files/ssp.txt

The standard version of SSP uses /dev/urandom directly. When a whole system is
built with SSP this tends to consume all the kernel entropy. /dev/erandom is
reccomended [sic] for SSP to conserve kernel entropy.

---
http://www.linuxfromscratch.org/hints/downloads/files/entropy.txt

Frandom stands for "fast random". Erandom stands for "economical random".
They both use the arcfour algorithm

The /dev/erandom device uses the constantly changing state of frandom's pool,
in a read-only mode, for entropy. /dev/erandom consumes no entropy from
/dev/random, and is ideal for applications that want to open the device
thousands of times, such as Stack Smashing Protector.

---

So it seems that if every executable is going to use the GCC SSP feature perhaps /dev/urandom is not the best source for random numbers. It could use /dev/erandom if that device was available. And it seems it would be useful to use for random data overwrites where /dev/urandom is too slow.

[agent 8131 (agent-8131)]

And here, noticed on Gentoo a couple of months ago:

http://www.gossamer-threads.com/lists/gentoo/hardened/208081

[Kees Cook (kees)]

AT_RANDOM is used by default in the Linux kernel since 2.6.29:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=f06295b44c296c8fb08823a3118468ae343b60f2

Almost all modern Linux systems use ASLR and stack protections that need a small amount of entropy per process. Since urandom always remixes, it doesn't strictly run out, but the entropy drops. In theory, this isn't a problem since the state of the pool cannot be extracted, guessed, or replayed.

It would be nice to have another entropy pool in the kernel for these things, like Schneier's pRNG from the Practical Cryptography book.

[agent 8131 (agent-8131)]

"independently of ASLR, it would still be nice if the kernel provided a 'strong' but fast PRNG device that one could for example use to sanitize a harddrive at raw write speeds, something that isn't possible with /dev/urandom for example. if such a PRNG existed it could then of course be used for ASLR as well but ASLR itself can live with less (ditto for the SSP cookie by the way)."

http://lwn.net/Articles/334027/

So this is really an upstream Linux kernel request or, at the outside, a request to the kernel team to include an out-of-kernel patch. If such a PRNG existed it could be used for ASLR and SSP and for "(scientific) simulations, wiping the disk, stress tests on algorithms".

The erandom device seems worthy of consideration (http://www.billauer.co.il/frandom.html). It seems to reach" harddrive raw write speeds" at 155MB/s (http://ubuntuforums.org/showthread.php?t=1076959). The frandom/erandom code was rejected from the kernel in 2003 (http://lkml.indiana.edu/hypermail/linux/kernel/0310.2/0015.html) but that was before ASLR and SSP became the default on GNU/Linux systems. Perhaps it's worth revisiting that debate.

[agent 8131 (agent-8131)]

I've come across another solution that this would be useful. In the testing of filesystems with compression enabled (zfs, btrfs). Trying to figure out raw write speeds using /dev/zero is meaningless because the data is highly compressed. Using /dev/urandom is too slow (~ 6MB/s). Based on this post from stackoverflow (http://serverfault.com/questions/6440/is-there-an-alternative-to-dev-urandom) I tried:

time openssl rand 1000000000 | head -c 1000000000 > testfile

which worked ok. But being able to use dd with erandom would be ideal for this situation.

[Christopher M. Penalver (penalvch)]

agent8131, the patch you requested ( https://bugs.launchpad.net/ubuntu/+source/linux/+bug/575669/comments/4 ) is currently available in Trusty:
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-trusty.git;a=commit;h=f83ce3e6b02d5e48b3a43b001390e2b58820389d

If this doesn't work out for you, please reopen, and execute the following via a terminal:
apport-collect 575669

Thank you for your understanding.

[agent 8131 (agent-8131)]

Christopher,

Can you explain how the commit you linked to resolves this issue? The commit seems to place process permissions checks on /proc/pid/stat and /proc/pid/wchan. I do not see how this affects the available entropy unless entropy is being depleted by unauthorized processes accessing those endpoints, which does not seem very likely to me. I doubt I will have time to test trusty before it is released, but once released I will test within a virtual server to see if this issue has been resolved.

[Christopher M. Penalver (penalvch)]

agent 8131, pending the test you noted you would perform in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/575669/comments/7 and subsequent apport-collect.

[agent 8131 (agent-8131)]

Christopher,

You did not address my previous question as to why that patch would resolve this issue.

Am I correct in assuming that you are requesting that I reproduce these conditions in the unreleased 14.04? It should be noted, if not reflected in the bug status, that this is still a confirmed bug for 10.04 LTS and 12.04 LTS. Since the status of a bug may be different for different releases is it the case that the bug status on this ticket is just meant to reflect the state in the current development release? This is not made clear in any of the documentation I've read.

For people coming across this bug report who have experienced application blocking issues due to low available entropy (usually running under some sort of virtualization) one easy solution is to install the haveged package and run that daemon. Whether that provides adequate security is open for debate, but it will prevent application hangs while waiting for sufficient entropy.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[agent 8131 (agent-8131)]

This appears to no longer be a problem in Ubuntu 14.04 or at least I have not been able to replicate it in the same way. This is still a problem for 12.04 however.