Deja Dup restore fails with "too many files open" error

[Pete Goodall]

I'm trying to restore my home directory to an empty directory on the /home/ partition and running Ubuntu 9.10 Alpha 4 + updates as of 2 Sept. I have plenty of room on the HDD because I only have on partition (besides swap), but I may be limited in RAM, with only 1 GB of RAM. I get mostly through the restore (approx 27 GB) and the restore errors out with the same error for many files:

"Could not restore home/pgoodall/Source/mydroid/development/emulator/mksdcard: Too many open files"

I assume this is because I have too little RAM, but don't know.

[Michael Terry (mterry)]

Thanks for the report, Pete! This is likely a bug that was half-fixed in duplicity 0.5.18 (which you should have in Alpha 4). The rest of the fix was in 0.6.00 (which will be in Karmic+1). A possible workaround is to run:
ulimit -n 4096
or some such number to make it work.

[Michael Terry (mterry)]

However... I can't totally clear Deja Dup. Deja Dup exacerbates the problem by rarely (i.e. never) making a fresh full backup (and causing lots of open files as a result when restoring -- though I'd argue that's a duplicity issue as well). That's fixed in trunk and will be released with Deja Dup 11.0.

[Michael Terry (mterry)]

Ken, please note the linked Debian report from Gonzalo, a Debian user who is using duplicity 0.6.04 and deja-dup 10.1. Gonzalo says that even with those versions and doing just a single full backup and then restore (i.e. not a long chain of incrementals), he is getting this error.

Are there other sources of problems that would cause this error? And isn't it sort of a design flaw of duplicity that it theoretically is open to this problem? Couldn't it close the file handles on files to which it isn't actively writing? That seems like it would solve the issue.

[Kenneth Loafman (kenneth-loafman)]

Reopened this bug since it appears not to be fixed. Will look into it ASAP.

[Kenneth Loafman (kenneth-loafman)]

Just as a workaround, please try 'sudo ulimit -n 4096' before running the restore. Let me know if that works.

[Michael Terry (mterry)]

I don't think Gonzalo is subscribed to this bug, but he knows about it and may read it. I already told him to try ulimit -n 4096 in the Debian bug, but he didn't mention whether it worked for him. Since he later ran into this problem again, I'm betting he didn't try it (since that has seemed to work for everybody else with this problem).

[Kenneth Loafman (kenneth-loafman)]

Michael Terry wrote:
> I don't think Gonzalo is subscribed to this bug, but he knows about it
> and may read it. I already told him to try ulimit -n 4096 in the Debian
> bug, but he didn't mention whether it worked for him. Since he later
> ran into this problem again, I'm betting he didn't try it (since that
> has seemed to work for everybody else with this problem).

I replied to the Debian bug just now. Perhaps he'll see it. This
appears to be a totally different problem from the long incremental
backup chains.

[Kenneth Loafman (kenneth-loafman)]

I have tried to reproduce this and am unable to do so. I'm going to need more information as to what system he's on, Python versions, etc. I'm just not seeing any more than about 70 open file descriptors at any one time, and most of those are Python libraries and objects.

[Myriam Schweingruber (myriam)]

Folks, this might be a more fundamental issue, since the maximum number of open files is limited by default to 1024 on the system.

We from the Amarok team had this bug report:

https://bugs.kde.org/show_bug.cgi?id=211026

and this can be solved by adding the following lines to /etc/security/limits.conf

* soft nofile 16384
* hard nofile 16384

BTW, the same will happen on a Fedora system, which has the same limitation. Seems we inherit a limitation from the nineties here...

Please try this out.

[Michael Terry (mterry)]

Deja Dup's side of things has been fixed with 11.0 -- we now make full backups occasionally.