Duplicity crashes with too many files open on CentOS 7 64bit

Bug #1556553 reported by Shovas on 2016-03-13
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Duplicity
Medium
Unassigned

Bug Description

I believe I have hit this bug which was apparently fixed a long time ago:

https://lists.nongnu.org/archive/html/duplicity-talk/2009-05/msg00051.html

# duplicity --version
duplicity 0.6.24

# python --version
Python 2.7.5

# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
^ Fully updated

File system: xfs

I recently started getting these cron errors reports on a CentOS 7 64bit server:

Traceback (most recent call last):
  File "/bin/duplicity", line 1502, in <module>
  File "/bin/duplicity", line 1496, in with_tempdir
  File "/bin/duplicity", line 1345, in main
  File "/bin/duplicity", line 1478, in do_backup
  File "/bin/duplicity", line 633, in incremental_backup
  File "/bin/duplicity", line 401, in write_multivol
  File "/usr/lib64/python2.7/site-packages/duplicity/gpg.py", line 375, in GzipWriteFile
IOError: [Errno 24] Too many open files: '/tmp/duplicity-S7rZtT-tempdir/mktemp-SnpuF5-3'

Traceback (most recent call last):
  File "/bin/duplicity", line 1502, in <module>
  File "/bin/duplicity", line 1496, in with_tempdir
  File "/bin/duplicity", line 1345, in main
  File "/bin/duplicity", line 1478, in do_backup
  File "/bin/duplicity", line 633, in incremental_backup
  File "/bin/duplicity", line 401, in write_multivol
  File "/usr/lib64/python2.7/site-packages/duplicity/gpg.py", line 383, in GzipWriteFile
  File "/usr/lib64/python2.7/site-packages/duplicity/diffdir.py", line 518, in next
  File "/usr/lib64/python2.7/site-packages/duplicity/diffdir.py", line 190, in get_delta_iter
  File "/usr/lib64/python2.7/site-packages/duplicity/diffdir.py", line 271, in collate2iters
  File "/usr/lib64/python2.7/site-packages/duplicity/selection.py", line 175, in Iterate
  File "/usr/lib64/python2.7/site-packages/duplicity/selection.py", line 142, in diryield
  File "/usr/lib64/python2.7/site-packages/duplicity/robust.py", line 61, in listpath
  File "/usr/lib64/python2.7/site-packages/duplicity/robust.py", line 37, in check_common_error
  File "/usr/lib64/python2.7/site-packages/duplicity/path.py", line 525, in listdir
OSError: [Errno 24] Too many open files: '/root'

Traceback (most recent call last):
  File "/bin/duplicity", line 1502, in <module>
  File "/bin/duplicity", line 1496, in with_tempdir
  File "/bin/duplicity", line 1345, in main
  File "/bin/duplicity", line 1478, in do_backup
  File "/bin/duplicity", line 633, in incremental_backup
  File "/bin/duplicity", line 401, in write_multivol
  File "/usr/lib64/python2.7/site-packages/duplicity/gpg.py", line 375, in GzipWriteFile
IOError: [Errno 24] Too many open files: '/tmp/duplicity-nZzCbM-tempdir/mktemp-uBj_1S-3'

Here's the relevant collection-status:

Chain start time: Thu Sep 24 20:46:50 2015
Chain end time: Sat Mar 12 12:02:29 2016
Number of contained backup sets: 1017
Total number of contained volumes: 1017

The bug mentioned at the top the problem was due to pgp open files when there were many incrementals.

My workaround right now is setting `ulimit -n 4096` at the top of the backup script.

I will try to get the -v9 log as well.

Changed in duplicity:
assignee: nobody → Kenneth Loafman (kenneth-loafman)
importance: Undecided → Medium
milestone: none → 0.7.13
status: New → In Progress

See question #631423 for details of the fix.

Changed in duplicity:
assignee: Kenneth Loafman (kenneth-loafman) → nobody
status: In Progress → Fix Committed
Shovas (shovas) wrote :

Glad to see this was picked up and fixed. Thanks! :)

Changed in duplicity:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Related questions