Leaks file descriptors and eventually runs out

[Matt Zimmerman]

Binary package hint: acpid

I was performing a stress test which simulated part of system startup (including starting and stopping (SIGTERM) the X server) in a tight loop. After 1000 or so iterations, the X server blocked during startup in connect(), trying to connect to /var/run/acpid.socket.

The acpid log file was 2147483647 bytes and looked like this:

[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
<repeated millions of times>
[Sun Sep 14 19:43:54 2008] E

netstat and /proc/pid/fd showed that it had reached the ulimit for open file descriptors. At that point, it must have started spewing to the log, and filled it up, at which point it didn't seem to even attempt accept(2) anymore.

I could reproduce the fd leak using this simple python program:

Python 2.5.2 (r252:60911, Apr 21 2008, 11:08:23)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> help(socket)

>>> s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>> s.connect('/var/run/acpid.socket')
>>> s.close()

Each time I run this test, acpid gains another fd which never goes away.

[Matt Zimmerman (mdz)]

There seems to be a similar report and patch at https://bugzilla.altlinux.org/show_bug.cgi?id=8303

[Sebastian Urban (surban)]

We also experienced this bug on one of our hosts without stress testing. I will report back when I have further details.

[Timo Aaltonen (tjaalton)]

I've seen this occasionally on a number of hosts, so I'd like to see this fixed in hardy too..

[Timo Aaltonen (tjaalton)]

This is fixed in intrepid->

[Timo Aaltonen (tjaalton)]

uploaded to hardy-proposed.

[Martin Pitt (pitti)]

Accepted into hardy-proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Paul Elliott (omahn)]

Verification of acpid in hardy-proposed failed:

The issue originally reported was replicated using both 1.04-5ubuntu9 and 1.04-5ubuntu9.1 from hardy-proposed using the Python test program. Symptoms are the same, errors in acpid log file and lots of used file descriptors:

root@pristine804:~# tail -n3 /var/log/acpid
[Tue Nov 18 00:58:26 2008] ERR: can't accept client: Too many open files
[Tue Nov 18 00:58:26 2008] ERR: can't accept client: Too many open files
[Tue Nov 18 00:58:26 2008] ERR: can't accept client: Too many open files
root@pristine804:~# netstat | grep -c 'acpid\.socket'
1030

[Steve Langasek (vorlon)]

The reason these file descriptors remain open is that the patch only causes the sockets to be closed on EPIPE, which only happens when an acpid tries to pass on an event to the socket. So acpid running on a system that never has acpi events could leak fds until it maxes out, even with this partial fix.

A full fix requires acpid to poll the client sockets to detect client shutdowns, not just wait for acpi events to detect EPIPE.

[Launchpad Janitor (janitor)]

This bug was fixed in the package acpid - 1.0.4-5ubuntu9.1

---------------
acpid (1.0.4-5ubuntu9.1) hardy-proposed; urgency=low

  * event.c: Close the client file descriptor on disconnect. Otherwise the
    limit will be reached and the logfile is filled with errors. (LP:
    #270259)

 -- Timo Aaltonen <email address hidden> Fri, 14 Nov 2008 11:33:16 +0200

[Steve Langasek (vorlon)]

The partial fix has been copied to hardy-updates, reopening this bug for tracking of a complete fix for the leak when no acpi events are happening.

[Steve Beattie (sbeattie)]

Please don't change bug state without comment, thanks.