Restore fails with “too many open files”

Bug #1878560 reported by Martin Ueding
This bug affects 2 people
Affects Status Importance Assigned to Milestone
duplicity (Ubuntu)

Bug Description

I use duplicity with Backblaze B2, have an incremental backup every hour and a full backup every 90 days. I recently tried to restore a file to the state that it had 14 days ago. Eventually it failed saying “too many open files”.

Apparently the last full backup was 2020-03-08 15:00:05Z. So that means that _a lot_ of incremental backups needed to be resolved. These are the last lines of output:

Get: -> temp/duplicity-a7n2sb9u-tempdir/mktemp-619jt1ub-252
Verarbeitung des lokalen Manifest b'/home/mu/.cache/duplicity/17532b308f659397a23dae3345234484/' (7297)
1 Volumen in Manifest gefunden
Get: -> temp/duplicity-a7n2sb9u-tempdir/mktemp-z7996evy-253
Verarbeitung des lokalen Manifest b'/home/mu/.cache/duplicity/17532b308f659397a23dae3345234484/' (5983)
1 Volumen in Manifest gefunden
Get: -> temp/duplicity-a7n2sb9u-tempdir/mktemp-alqkws80-254
Traceback (innermost last):
  File "/usr/bin/duplicity", line 100, in <module>
  File "/usr/bin/duplicity", line 86, in with_tempdir
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 1538, in main
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 1618, in do_backup
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 723, in restore
    if not patchdir.Write_ROPaths(globals.local_path,
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 578, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 541, in integrate_patch_iters
    for patch_seq in collated:
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 407, in yield_tuples
    setrorps(overflow, elems)
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 396, in setrorps
    elems[i] = next(iter_list[i])
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 126, in difftar2path_iter
    tarinfo_list = [next(tar_iter)]
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 355, in __next__
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 348, in set_tarfile
    self.current_fp = next(self.fileobj_iter)
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 759, in get_fileobj_iter
    yield restore_get_enc_fileobj(backup_set.backend,
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 804, in restore_get_enc_fileobj
    fileobj = tdp.filtered_open_with_delete(u"rb")
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 125, in filtered_open_with_delete
    fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 802, in filtered_open
    return gpg.GPGFile(False, self, gpg_profile)
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 225, in __init__
    p1 =[u'--decrypt'], create_fhs=gnupg_fhs,
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 374, in run
    process = self._attach_fork_exec(gnupg_commands, args,
  File "/usr/lib64/python3.8/site-packages/duplicity/", line 403, in _attach_fork_exec
    pipe = os.pipe()
 OSError: [Errno 24] Too many open files

I would assume that it stumbles over the 1024 file descriptor limit and it just cannot resolve a backup with that many incremental backups. Basically with the current state of the software my incremental backups are worthless because I cannot restore. Is there something that I can do except doing less frequent incremental updates and more full backups?

Revision history for this message
Martin Ueding (martin-ueding) wrote :
Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

This is a known issue with duplicity. You will need to increase your systems number of open files since it's set so low. I'd suggest 16384 or higher if you have a lot of incrementals. I'd also suggest limiting the number of incrementals and doing more full backups.

Changed in duplicity (Ubuntu):
importance: Undecided → Low
Changed in duplicity:
status: New → Opinion
Changed in duplicity:
status: Opinion → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers