Duplicity crashes with too many files open on CentOS 7 64bit

[Shovas]

I believe I have hit this bug which was apparently fixed a long time ago:

https://lists.nongnu.org/archive/html/duplicity-talk/2009-05/msg00051.html

# duplicity --version
duplicity 0.6.24

# python --version
Python 2.7.5

# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
^ Fully updated

File system: xfs

I recently started getting these cron errors reports on a CentOS 7 64bit server:

Traceback (most recent call last):
  File "/bin/duplicity", line 1502, in <module>
  File "/bin/duplicity", line 1496, in with_tempdir
  File "/bin/duplicity", line 1345, in main
  File "/bin/duplicity", line 1478, in do_backup
  File "/bin/duplicity", line 633, in incremental_backup
  File "/bin/duplicity", line 401, in write_multivol
  File "/usr/lib64/python2.7/site-packages/duplicity/gpg.py", line 375, in GzipWriteFile
IOError: [Errno 24] Too many open files: '/tmp/duplicity-S7rZtT-tempdir/mktemp-SnpuF5-3'

Traceback (most recent call last):
  File "/bin/duplicity", line 1502, in <module>
  File "/bin/duplicity", line 1496, in with_tempdir
  File "/bin/duplicity", line 1345, in main
  File "/bin/duplicity", line 1478, in do_backup
  File "/bin/duplicity", line 633, in incremental_backup
  File "/bin/duplicity", line 401, in write_multivol
  File "/usr/lib64/python2.7/site-packages/duplicity/gpg.py", line 383, in GzipWriteFile
  File "/usr/lib64/python2.7/site-packages/duplicity/diffdir.py", line 518, in next
  File "/usr/lib64/python2.7/site-packages/duplicity/diffdir.py", line 190, in get_delta_iter
  File "/usr/lib64/python2.7/site-packages/duplicity/diffdir.py", line 271, in collate2iters
  File "/usr/lib64/python2.7/site-packages/duplicity/selection.py", line 175, in Iterate
  File "/usr/lib64/python2.7/site-packages/duplicity/selection.py", line 142, in diryield
  File "/usr/lib64/python2.7/site-packages/duplicity/robust.py", line 61, in listpath
  File "/usr/lib64/python2.7/site-packages/duplicity/robust.py", line 37, in check_common_error
  File "/usr/lib64/python2.7/site-packages/duplicity/path.py", line 525, in listdir
OSError: [Errno 24] Too many open files: '/root'

Traceback (most recent call last):
  File "/bin/duplicity", line 1502, in <module>
  File "/bin/duplicity", line 1496, in with_tempdir
  File "/bin/duplicity", line 1345, in main
  File "/bin/duplicity", line 1478, in do_backup
  File "/bin/duplicity", line 633, in incremental_backup
  File "/bin/duplicity", line 401, in write_multivol
  File "/usr/lib64/python2.7/site-packages/duplicity/gpg.py", line 375, in GzipWriteFile
IOError: [Errno 24] Too many open files: '/tmp/duplicity-nZzCbM-tempdir/mktemp-uBj_1S-3'

Here's the relevant collection-status:

Chain start time: Thu Sep 24 20:46:50 2015
Chain end time: Sat Mar 12 12:02:29 2016
Number of contained backup sets: 1017
Total number of contained volumes: 1017

The bug mentioned at the top the problem was due to pgp open files when there were many incrementals.

My workaround right now is setting `ulimit -n 4096` at the top of the backup script.

I will try to get the -v9 log as well.

[Kenneth Loafman (kenneth-loafman)]

See question #631423 for details of the fix.

[Shovas (shovas)]

Glad to see this was picked up and fixed. Thanks! :)