Comment 42 for bug 1727653

I would bet the update will fix it. Huge manifests go into memory, in
bulk, and in lists. The lists stay around and if Python GC is not working
properly, the bulk reads do as well. During processing, there were as many
as 3 copies of the manifest in memory, one bulk, one the result of re
iteration, and the other in the collection list. That kind of misuse can
really screw up GC.

As answer to your question, changing to 'ulimit -n 10240' could help in a
very cases, but I would try both individually to see which one does the job.

If they are still on the 0.6-series, the 'ulimit' fix is the only one that
would work.

On Sun, Feb 11, 2018 at 12:34 PM, Michael Terry <email address hidden> wrote:

> I'd also be curious if this is because of duplicity <0.7.16 or because
> of the ulimit line we use for deja-dup-monitor. It's not clear to me if
> I should change the ulimit line or if we just need to update duplicity
> for those users affected.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1727653
>
> Title:
> error: can't start new thread
>
> Status in Duplicity:
> In Progress
> Status in deja-dup package in Ubuntu:
> Triaged
> Status in duplicity package in Ubuntu:
> In Progress
>
> Bug description:
> $ uname -a
> Linux mbpr13b 4.13.0-16-generic #19-Ubuntu SMP Wed Oct 11 18:35:14 UTC
> 2017 x86_64 x86_64 x86_64 GNU/Linux
>
> duplicity gets started daily via deja-dup automatically.
>
> Every now and then (approx. every third time) I get the following
> error:
>
> ------------------
>
> Failed with an unknown error.
>
> Traceback (most recent call last):
> File "/usr/bin/duplicity", line 1546, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 1540, in with_tempdir
> fn()
> File "/usr/bin/duplicity", line 1391, in main
> do_backup(action)
> File "/usr/bin/duplicity", line 1468, in do_backup
> restore(col_stats)
> File "/usr/bin/duplicity", line 731, in restore
> restore_get_patched_rop_iter(col_stats)):
> File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 560, in Write_ROPaths
> for ropath in rop_iter:
> File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 523, in integrate_patch_iters
> for patch_seq in collated:
> File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 389, in yield_tuples
> setrorps(overflow, elems)
> File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 378, in setrorps
> elems[i] = iter_list[i].next()
> File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 107, in filter_path_iter
> for path in path_iter:
> File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 121, in difftar2path_iter
> tarinfo_list = [tar_iter.next()]
> File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 339, in next
> self.set_tarfile()
> File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 333, in set_tarfile
> self.current_fp = self.fileobj_iter.next()
> File "/usr/bin/duplicity", line 768, in get_fileobj_iter
> manifest.volume_info_dict[vol_num])
> File "/usr/bin/duplicity", line 811, in restore_get_enc_fileobj
> fileobj = tdp.filtered_open_with_delete("rb")
> File "/usr/lib/python2.7/dist-packages/duplicity/dup_temp.py", line
> 119, in filtered_open_with_delete
> fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
> File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 778,
> in filtered_open
> return gpg.GPGFile(False, self, gpg_profile)
> File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 202,
> in __init__
> 'logger': self.logger_fp})
> File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py",
> line 374, in run
> create_fhs, attach_fhs)
> File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py",
> line 420, in _attach_fork_exec
> process.thread.start()
> File "/usr/lib/python2.7/threading.py", line 736, in start
> _start_new_thread(self.__bootstrap, ())
> error: can't start new thread
>
> --------------
>
> Restarting deja-dup manually fixes the problem and the next few days,
> automatic backup will run correctly until (see above).
>
> ProblemType: Bug
> DistroRelease: Ubuntu 17.10
> Package: duplicity 0.7.12-1ubuntu1
> ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4
> Uname: Linux 4.13.0-16-generic x86_64
> ApportVersion: 2.20.7-0ubuntu3
> Architecture: amd64
> CurrentDesktop: ubuntu:GNOME
> Date: Thu Oct 26 10:49:45 2017
> InstallationDate: Installed on 2017-10-20 (5 days ago)
> InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64
> (20171018)
> SourcePackage: duplicity
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1727653/+subscriptions
>