Too many files opened at once

Bug #13371 reported by Sylvain Defresne on 2005-02-28
10
Affects Status Importance Assigned to Milestone
bittorrent (Debian)
New
Unknown
bittorrent (Ubuntu)
Medium
Unassigned

Bug Description

The bittorrent client opens all the file at the beginning
of the transfer, and thus can reach the configured max
open file descriptor by process when trying to download
large torrents.

It is then completely frozen since a second IOError is
raised and uncaught when the error dialog glade XML
file is opened.

The following patch fixes the problem by preventing
bittorrent from opening too many file at once (but it
is a bit more cpu / disk hungry since the files are
opened and closed more often) ...

Sylvain Defresne (sdefresne) wrote :

Created an attachment (id=1476)
The patch mentionned in the bug report

Sylvain Defresne (sdefresne) wrote :

Created an attachment (id=1478)
The patch mentionned in the bug report

The previous patch was buggy (it destroyed the file
instead of appending to them).

Matthias Klose (doko) wrote :

Is this fix worth adding? Trade off cpu load vs. having a large number of files
open?

There's another possibility to raise the number of allowed open files per
process for the desktop user,
adding a line to /etc/security/limits.conf:

<username> hard nofile 2048

maybe this can be generalized to

@users hard nofile 2048

as a default.

Sylvain Defresne (sdefresne) wrote :

I was aware of this configuration option, but I have found several
times torrents with 5000+ files and I don't really like having the
maximum number of file descriptor by process set this high.

Besides, when opening such a large torrent, the application seems
frozen. It is caused by the fact that opening the last file will
raise an exception, that is caught. In response to this exception
the program will try to read the glade description of the error
dialog box from the disk, which will result in another exception
being raised. That second exception is not caught and then the
GUI thread is dead ...

My patch was a first attempt at correcting this problem. Another
approach could be to read the maximum number of file descriptor
authorized for this process and only close some of them when there
is less than ten (or some other value large enough to be able to
load glade description file) remaining. The only problem is to
get the number of opened file descriptor currently in a portable
way ...

Sylvain Defresne (sdefresne) wrote :

(In reply to comment #4)
> Besides, when opening such a large torrent, the application seems
> frozen. It is caused by the fact that opening the last file will
> raise an exception, that is caught. In response to this exception
> the program will try to read the glade description of the error
> dialog box from the disk, which will result in another exception
> being raised. That second exception is not caught and then the
> GUI thread is dead ...

At least, it is the case when the application is gnome-btdownload
(which seems to be the "official" bittorrent client for hoary). I
haven't tested with other graphical front-end.

Matthias Klose (doko) wrote :

nofiles = len(os.listdir('/proc/%d/fd/*' % getpid()))

that may be portable enough for Linux, but I'm not aware of another method.

Matt Zimmerman (mdz) wrote :

The correct way to determine this value is to use sysconf. I am not sure
whether there is a Python API for it, and at any rate, the Python program cannot
know how many file descriptors are in use by the interpreter.

At any rate, this bug isn't serious enough to warrant an invasive change so
close to release, so let's revisit this after the Hoary release.

Matthias Klose (doko) wrote :

os.sysconf('SC_OPEN_MAX') gets you the value of the maximum allowed files. The
limits approach was to limit this change to the processes the user starts.
Another possibilty would be to keep the soft limit, but increase the hard limit,
so this value can be raised by the user by resource.setrlimit(...)

IMO it doesn't make sense if the interpreter tracks a record of all open files,
the number of files opened by extension modules and external libraries is
unknown to the interpreter and such tracking would be unreliable.

Sylvain Defresne (sdefresne) wrote :

I have checked the Bittornado code, and they let the user specify
the maximum number of file that can be opened at once by a single
Storage object. If after opening a file this limit is reached the
last recently opened file is closed.

This is better than my approach of always closing all the file,
is more tested and more portable than trying to guess how many
open file-descriptor the python interpreter has got.

And it should be relatively easy to port it from Bittornado to
Bittorrent if it is the choosen solution.

Filip Palm (filip) wrote :

I where just going to write a new bugreport about this issue but i found this one, just want to ping and say that this is still an issue in Edgy.

If i install bittorrent 5.0 (bittorrent_5.0.0_python2.4.deb) from www.bittorrent.com this problem is fixed.Why don't we just upgrade bittorrent 3.x to 5.x for Feisty?

Daniel Hahler (blueyed) on 2007-04-17
description: updated
Adam Niedling (krychek) wrote :

Is this still an issue after 3 years of the original report?

Changed in bittorrent:
status: Confirmed → Incomplete
drink (martin-espinoza) wrote :

This is still an issue on Gutsy, not sure about Hardy.

Thank you for taking the time to report this bug and helping to make Ubuntu better. You reported this bug a while ago and there hasn't been any activity in it recently. We were wondering is this still an issue for you? Can you try with latest Ubuntu release? Thanks in advance.

Sylvain Defresne (sdefresne) wrote :

I tested the btdownloadcurses from the bittorrent package from Ubuntu 8.04, and it still does exhibit this bug. But is really outdated: version available in Ubuntu is 3.4.2-11ubuntu5, when upstream version is 5.0.

Moreover, the default bittorrent client in Ubuntu 8.04 is Transmission, and it doesn't have this bug. So I think that the bittorrent package should be deprecated, or even removed (the bittornado package give a completely compatible interface from user space and does not exhibit this error).

I would like to see an official bittorrent client in Ubuntu repos, but
I would also like to see it updated. Any chance of this happening? I
like to have the reference implementation available for testing
purposes...

On 9/25/08, Sylvain Defresne <email address hidden> wrote:
> I tested the btdownloadcurses from the bittorrent package from Ubuntu
> 8.04, and it still does exhibit this bug. But is really outdated:
> version available in Ubuntu is 3.4.2-11ubuntu5, when upstream version is
> 5.0.
>
> Moreover, the default bittorrent client in Ubuntu 8.04 is Transmission,
> and it doesn't have this bug. So I think that the bittorrent package
> should be deprecated, or even removed (the bittornado package give a
> completely compatible interface from user space and does not exhibit
> this error).
>
> --
> Too many files opened at once
> https://bugs.launchpad.net/bugs/13371
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in "bittorrent" source package in Ubuntu: Incomplete
>
> Bug description:
> The bittorrent client opens all the file at the beginning
> of the transfer, and thus can reach the configured max
> open file descriptor by process when trying to download
> large torrents.
>
> It is then completely frozen since a second IOError is
> raised and uncaught when the error dialog glade XML
> file is opened.
>
> The following patch fixes the problem by preventing
> bittorrent from opening too many file at once (but it
> is a bit more cpu / disk hungry since the files are
> opened and closed more often) ...
>

@drink: you have 'transmission' as default bittorrent client (and also updated).

drink (martin-espinoza) wrote :

While I appreciate transmission (simple, works) I shouldn't have to
point out that "bittorrent" is the reference client for the bittorrent
protocol/system/networks, not transmission. In theory, it should work
no differently - which is why it's nice to have around to compare to.

Nothing is stopping me from installing bittorrent manually (and I have
done so previously) but I would prefer to have the latest in the
repository.

Is there any reason the official client shouldn't be in the repo? I
agree it doesn't need to be on the CD.

On 9/27/08, Pablo Castellano <email address hidden> wrote:
> @drink: you have 'transmission' as default bittorrent client (and also
> updated).
>
> --
> Too many files opened at once
> https://bugs.launchpad.net/bugs/13371
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in "bittorrent" source package in Ubuntu: Incomplete
>
> Bug description:
> The bittorrent client opens all the file at the beginning
> of the transfer, and thus can reach the configured max
> open file descriptor by process when trying to download
> large torrents.
>
> It is then completely frozen since a second IOError is
> raised and uncaught when the error dialog glade XML
> file is opened.
>
> The following patch fixes the problem by preventing
> bittorrent from opening too many file at once (but it
> is a bit more cpu / disk hungry since the files are
> opened and closed more often) ...
>

@drink: I didn't know about that but searching a bit it seems that unfortunately there's a license problem.
You can read more here [1] and here [2].

[1]: https://bugs.launchpad.net/ubuntu/+source/bittorrent/+bug/58917
[2]: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=298814

drink (martin-espinoza) wrote :

In that case, I give up. It might be better to eliminate it than keep
the antique edition, which only confuses the issue IMO.

papukaija (papukaija) on 2010-08-26
tags: added: patch
Changed in bittorrent (Debian):
status: Unknown → New

Thank you for posting this bug.

Is this still an issue in Lucid?

Matthias Klose (doko) wrote :

unassigning. we will demote bittorrent to universe for natty. largely replaced by bittornado

Changed in bittorrent (Ubuntu):
assignee: Matthias Klose (doko) → nobody
Bryce Harrington (bryce) wrote :

Given comments 18 and 19, and no activity in several years, I think it's clear this is not something that will be fixed here at the distro level, so I'm going to go ahead and pull the plug on it. It sounds like the issue is a legitimate one, so am setting to wontfix rather than invalid.

Changed in bittorrent (Ubuntu):
status: Incomplete → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.