Leaks file descriptors and eventually runs out

Bug #270259 reported by Matt Zimmerman on 2008-09-14
92
This bug affects 1 person
Affects Status Importance Assigned to Milestone
acpid (ALT Linux)
Unknown
Unknown
acpid (Suse)
Fix Released
Critical
acpid (Ubuntu)
Medium
Unassigned
Hardy
Undecided
Unassigned

Bug Description

Binary package hint: acpid

I was performing a stress test which simulated part of system startup (including starting and stopping (SIGTERM) the X server) in a tight loop. After 1000 or so iterations, the X server blocked during startup in connect(), trying to connect to /var/run/acpid.socket.

The acpid log file was 2147483647 bytes and looked like this:

[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
[Sun Sep 14 19:43:54 2008] ERR: can't accept client: Too many open files
<repeated millions of times>
[Sun Sep 14 19:43:54 2008] E

netstat and /proc/pid/fd showed that it had reached the ulimit for open file descriptors. At that point, it must have started spewing to the log, and filled it up, at which point it didn't seem to even attempt accept(2) anymore.

I could reproduce the fd leak using this simple python program:

Python 2.5.2 (r252:60911, Apr 21 2008, 11:08:23)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> help(socket)

>>> s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>> s.connect('/var/run/acpid.socket')
>>> s.close()

Each time I run this test, acpid gains another fd which never goes away.

Matt Zimmerman (mdz) wrote :

There seems to be a similar report and patch at https://bugzilla.altlinux.org/show_bug.cgi?id=8303

Matt Zimmerman (mdz) on 2008-09-14
Changed in acpid:
importance: Undecided → Medium
status: New → Triaged
Changed in acpid:
status: Unknown → Fix Released
Sebastian Urban (surban) wrote :

We also experienced this bug on one of our hosts without stress testing. I will report back when I have further details.

Timo Aaltonen (tjaalton) wrote :

I've seen this occasionally on a number of hosts, so I'd like to see this fixed in hardy too..

Changed in acpid:
assignee: nobody → tjaalton
status: New → In Progress
Timo Aaltonen (tjaalton) wrote :

This is fixed in intrepid->

Changed in acpid:
status: Triaged → Fix Released
Timo Aaltonen (tjaalton) wrote :

uploaded to hardy-proposed.

Changed in acpid:
status: In Progress → Fix Committed
Martin Pitt (pitti) wrote :

Accepted into hardy-proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Paul Elliott (omahn) wrote :

Verification of acpid in hardy-proposed failed:

The issue originally reported was replicated using both 1.04-5ubuntu9 and 1.04-5ubuntu9.1 from hardy-proposed using the Python test program. Symptoms are the same, errors in acpid log file and lots of used file descriptors:

root@pristine804:~# tail -n3 /var/log/acpid
[Tue Nov 18 00:58:26 2008] ERR: can't accept client: Too many open files
[Tue Nov 18 00:58:26 2008] ERR: can't accept client: Too many open files
[Tue Nov 18 00:58:26 2008] ERR: can't accept client: Too many open files
root@pristine804:~# netstat | grep -c 'acpid\.socket'
1030

Martin Pitt (pitti) on 2008-11-18
Changed in acpid:
status: Fix Committed → Triaged
Timo Aaltonen (tjaalton) on 2009-01-27
Changed in acpid:
assignee: tjaalton → nobody
Steve Langasek (vorlon) wrote :

The reason these file descriptors remain open is that the patch only causes the sockets to be closed on EPIPE, which only happens when an acpid tries to pass on an event to the socket. So acpid running on a system that never has acpi events could leak fds until it maxes out, even with this partial fix.

A full fix requires acpid to poll the client sockets to detect client shutdowns, not just wait for acpi events to detect EPIPE.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package acpid - 1.0.4-5ubuntu9.1

---------------
acpid (1.0.4-5ubuntu9.1) hardy-proposed; urgency=low

  * event.c: Close the client file descriptor on disconnect. Otherwise the
    limit will be reached and the logfile is filled with errors. (LP:
    #270259)

 -- Timo Aaltonen <email address hidden> Fri, 14 Nov 2008 11:33:16 +0200

Changed in acpid:
status: Triaged → Fix Released
Steve Langasek (vorlon) wrote :

The partial fix has been copied to hardy-updates, reopening this bug for tracking of a complete fix for the leak when no acpi events are happening.

Changed in acpid:
status: Fix Released → Triaged
security vulnerability: no → yes
visibility: public → private
Steve Beattie (sbeattie) wrote :

Please don't change bug state without comment, thanks.

security vulnerability: yes → no
visibility: private → public
Changed in acpid (Ubuntu):
status: Triaged → Fix Released
Changed in acpid (Suse):
importance: Unknown → Critical
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.