Duplicity fails with MemoryError

Bug #1722454 reported by Juerg Haefliger on 2017-10-10
80
This bug affects 18 people
Affects Status Importance Assigned to Milestone
Duplicity
Undecided
Unassigned
duplicity (Ubuntu)
Undecided
Unassigned

Bug Description

Running the following command:

/usr/bin/python /usr/bin/duplicity collection-status --exclude=/mnt/jabba/juergh/backup/gollum --include=/home/juergh/.cache/deja-dup/metadata --exclude=/home/juergh/.uk-pkg --exclude=/home/juergh/Music --exclude=/home/juergh/debug --exclude=/home/juergh/canonical/cve-matrix/cve-matrix/cve-tracker --exclude=/home/juergh/canonical/cve-matrix/cve-autotriage --exclude=/home/juergh/Desktop --exclude=/home/juergh/.cache --exclude=/home/juergh/tmp --exclude=/home/juergh/git --exclude=/home/juergh/Downloads --exclude=/home/juergh/.local/share/Trash --exclude=/home/juergh/.cache/deja-dup/tmp --exclude=/home/juergh/.thumbnails --exclude=/home/juergh/.cache/deja-dup --exclude=/home/juergh/.cache --include=/home/juergh --exclude=/sys --exclude=/run --exclude=/proc --exclude=/var/tmp --exclude=/tmp --exclude=** --gio file:///mnt/jabba/juergh/backup/gollum --no-encryption --verbosity=9 --gpg-options=--no-use-agent --archive-dir=/home/juergh/.cache/deja-dup --tempdir=/home/juergh/.cache/deja-dup/tmp --log-fd=19

results in:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1546, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1540, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1391, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1416, in do_backup
    globals.archive_dir).set_values()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 710, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 836, in get_backup_chains
    add_to_sets(f)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 824, in add_to_sets
    if set.add_filename(filename):
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 101, in add_filename
    self.set_manifest(filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 148, in set_manifest
    self.set_files_changed()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 128, in set_files_changed
    mf = self.get_manifest()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 250, in get_manifest
    return self.get_local_manifest()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 224, in get_local_manifest
    return manifest.Manifest().from_string(manifest_buffer)
  File "/usr/lib/python2.7/dist-packages/duplicity/manifest.py", line 215, in from_string
    vi = VolumeInfo().from_string(match.group(1))
  File "/usr/lib/python2.7/dist-packages/duplicity/manifest.py", line 383, in from_string
    linelist = s.strip().split("\n")
MemoryError

ProblemType: Bug
DistroRelease: Ubuntu 17.10
Package: duplicity 0.7.12-1ubuntu1
ProcVersionSignature: Ubuntu 4.13.0-12.13-generic 4.13.3
Uname: Linux 4.13.0-12-generic x86_64
ApportVersion: 2.20.7-0ubuntu2
Architecture: amd64
CurrentDesktop: GNOME
Date: Tue Oct 10 08:05:45 2017
InstallationDate: Installed on 2017-05-25 (137 days ago)
InstallationMedia: Ubuntu-GNOME 17.04 "Zesty Zapus" - Release amd64 (20170412)
SourcePackage: duplicity
UpgradeStatus: No upgrade log present (probably fresh install)

Juerg Haefliger (juergh) wrote :
Juerg Haefliger (juergh) wrote :

Workaround is to purge the cache ~/.cache/deja-dup.

Juerg Haefliger (juergh) wrote :

Happened again the day after cleaning the cache.

Juerg Haefliger (juergh) wrote :

Hmm... Why do you think this is a duplicate of #1720159? My stack trace looks different.

Download full text (4.3 KiB)

Memory errors hit when you run out of memory. Sometimes it's a different
spot in the code. This traceback was from manifest.py, so an easy call.

On Fri, Oct 27, 2017 at 2:18 AM, Juerg Haefliger <<email address hidden>
> wrote:

> *** This bug is a duplicate of bug 1720159 ***
> https://bugs.launchpad.net/bugs/1720159
>
> Hmm... Why do you think this is a duplicate of #1720159? My stack trace
> looks different.
>
> --
> You received this bug notification because you are subscribed to
> duplicity in Ubuntu.
> https://bugs.launchpad.net/bugs/1722454
>
> Title:
> Duplicity fails with MemoryError
>
> Status in Duplicity:
> New
> Status in duplicity package in Ubuntu:
> New
>
> Bug description:
> Running the following command:
>
> /usr/bin/python /usr/bin/duplicity collection-status
> --exclude=/mnt/jabba/juergh/backup/gollum
> --include=/home/juergh/.cache/deja-dup/metadata --exclude=/home/juergh
> /.uk-pkg --exclude=/home/juergh/Music --exclude=/home/juergh/debug
> --exclude=/home/juergh/canonical/cve-matrix/cve-matrix/cve-tracker
> --exclude=/home/juergh/canonical/cve-matrix/cve-autotriage
> --exclude=/home/juergh/Desktop --exclude=/home/juergh/.cache
> --exclude=/home/juergh/tmp --exclude=/home/juergh/git
> --exclude=/home/juergh/Downloads
> --exclude=/home/juergh/.local/share/Trash
> --exclude=/home/juergh/.cache/deja-dup/tmp
> --exclude=/home/juergh/.thumbnails --exclude=/home/juergh/.cache/deja-
> dup --exclude=/home/juergh/.cache --include=/home/juergh
> --exclude=/sys --exclude=/run --exclude=/proc --exclude=/var/tmp
> --exclude=/tmp --exclude=** --gio
> file:///mnt/jabba/juergh/backup/gollum --no-encryption --verbosity=9
> --gpg-options=--no-use-agent --archive-dir=/home/juergh/.cache/deja-
> dup --tempdir=/home/juergh/.cache/deja-dup/tmp --log-fd=19
>
> results in:
>
> Traceback (most recent call last):
> File "/usr/bin/duplicity", line 1546, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 1540, in with_tempdir
> fn()
> File "/usr/bin/duplicity", line 1391, in main
> do_backup(action)
> File "/usr/bin/duplicity", line 1416, in do_backup
> globals.archive_dir).set_values()
> File "/usr/lib/python2.7/dist-packages/duplicity/collections.py",
> line 710, in set_values
> self.get_backup_chains(partials + backend_filename_list)
> File "/usr/lib/python2.7/dist-packages/duplicity/collections.py",
> line 836, in get_backup_chains
> add_to_sets(f)
> File "/usr/lib/python2.7/dist-packages/duplicity/collections.py",
> line 824, in add_to_sets
> if set.add_filename(filename):
> File "/usr/lib/python2.7/dist-packages/duplicity/collections.py",
> line 101, in add_filename
> self.set_manifest(filename)
> File "/usr/lib/python2.7/dist-packages/duplicity/collections.py",
> line 148, in set_manifest
> self.set_files_changed()
> File "/usr/lib/python2.7/dist-packages/duplicity/collections.py",
> line 128, in set_files_changed
> mf = self.get_manifest()
> File "/usr/lib/python2.7/dist-packages/duplicity/collections.py",
> line 250, in get_manifest
> return self.get_local_m...

Read more...

Matthew Gregg (mcg) wrote :

Understand this is a dup ticket, but I do get the exact same traceback as posted in this ticket.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in duplicity (Ubuntu):
status: New → Confirmed
SUPER_ET_DUPER (machst) wrote :

I also get the exact same traceback

jokerejoker (jokerejoker) wrote :

I get the exact same error. Backup is started automatically every week and is only configured from the Ubuntu control panel. This only occurred after upgrading to Ubuntu 17.10.

I have an i7 processor with 16GB ram and SSD drive.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1546, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1540, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1391, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1416, in do_backup
    globals.archive_dir).set_values()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 710, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 836, in get_backup_chains
    add_to_sets(f)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 824, in add_to_sets
    if set.add_filename(filename):
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 101, in add_filename
    self.set_manifest(filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 148, in set_manifest
    self.set_files_changed()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 128, in set_files_changed
    mf = self.get_manifest()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 250, in get_manifest
    return self.get_local_manifest()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 224, in get_local_manifest
    return manifest.Manifest().from_string(manifest_buffer)
  File "/usr/lib/python2.7/dist-packages/duplicity/manifest.py", line 215, in from_string
    vi = VolumeInfo().from_string(match.group(1))
  File "/usr/lib/python2.7/dist-packages/duplicity/manifest.py", line 383, in from_string
    linelist = s.strip().split("\n")
MemoryError

beret (beret-2) wrote :

The same for me, it's look like is so long a file index. And isn't depend of real memory amount. I've 20Gb RAM and it should be enough for any operation even operations.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1546, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1540, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1391, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1521, in do_backup
    check_last_manifest(col_stats) # not needed for full backup
  File "/usr/bin/duplicity", line 1222, in check_last_manifest
    last_backup_set.check_manifests()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 199, in check_manifests
    remote_manifest = self.get_remote_manifest()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 243, in get_remote_manifest
    return manifest.Manifest().from_string(manifest_buffer)
  File "/usr/lib/python2.7/dist-packages/duplicity/manifest.py", line 215, in from_string
    vi = VolumeInfo().from_string(match.group(1))
  File "/usr/lib/python2.7/dist-packages/duplicity/manifest.py", line 383, in from_string
    linelist = s.strip().split("\n")
MemoryError

This bug report is a duplicate of: Bug #1720159: Cannot allocate memory with large manifest file since 0.7.03. It has been fixed.

Is it really a duplicate? I have now upgraded to the supposed fix of #1720159

$ apt show duplicity
Package: duplicity
Version: 0.7.16+bzr1347-0ubuntu1~ubuntu17.10.1

... but still get the same error as before, and it's not the same error report as listed in the supposed dup:
(by the way, this has real consequences. My motherboard just failed while my backups have not been happening due to this bug: much stress!)

Backup Failed

Failed with an unknown error

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1559, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1545, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1394, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1515, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 565, in full_backup
    col_stats.set_values(sig_chain_warning=None)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 720, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 846, in get_backup_chains
    add_to_sets(f)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 834, in add_to_sets
    if set.add_filename(filename):
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 102, in add_filename
    self.set_manifest(filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 153, in set_manifest
    self.set_files_changed()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 129, in set_files_changed
    mf = self.get_manifest()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 256, in get_manifest
    return self.get_local_manifest()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 234, in get_local_manifest
    return manifest.Manifest().from_string(manifest_buffer)
  File "/usr/lib/python2.7/dist-packages/duplicity/manifest.py", line 202, in from_string
    vi = VolumeInfo().from_string(match.group(1))
 MemoryError

Mikhail Kashkin (mkashkin) wrote :

Adding my own t/b. It is little different from other posts on this page.

Version info:

$ apt show duplicity
Package: duplicity
Version: 0.7.12-1ubuntu1

Error:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1546, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1540, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1391, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1468, in do_backup
    restore(col_stats)
  File "/usr/bin/duplicity", line 731, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 560, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 523, in integrate_patch_iters
    for patch_seq in collated:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 389, in yield_tuples
    setrorps(overflow, elems)
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 378, in setrorps
    elems[i] = iter_list[i].next()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 107, in filter_path_iter
    for path in path_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 121, in difftar2path_iter
    tarinfo_list = [tar_iter.next()]
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 339, in next
    self.set_tarfile()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 333, in set_tarfile
    self.current_fp = self.fileobj_iter.next()
  File "/usr/bin/duplicity", line 763, in get_fileobj_iter
    manifest = backup_set.get_manifest()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 250, in get_manifest
    return self.get_local_manifest()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 224, in get_local_manifest
    return manifest.Manifest().from_string(manifest_buffer)
  File "/usr/lib/python2.7/dist-packages/duplicity/manifest.py", line 207, in from_string
    self.files_changed = list(map(parse_fileinfo, match.group(3).split('\n')))
  File "/usr/lib/python2.7/dist-packages/duplicity/manifest.py", line 204, in parse_fileinfo
    fileinfo = line.strip().split()
MemoryError

Please upgrade to the current version of duplicity, 0.7.16. This will assure that any bugs fixed since your release are available and may fix your issue. This applies especially to duplicity versions between 0.7.03 and 0.7.14 inclusive. There was a fix in 0.7.15 that reduced memory usage drastically, and will help with memory errors and inability to start new threads.
The bug that was resolved in 0.7.15:

* Fixed bug #1720159 - Cannot allocate memory with large manifest file since 0.7.03
  - filelist is not read if --file-changed option in collection-status not present
  - This will keep memory usage lower in non collection-status operations

There are three options:
* Release tarball Install - https://launchpad.net/duplicity/+download
* Daily duplicity builds - https://launchpad.net/~duplicity-team/+archive/ubuntu/daily
* Stable duplicity builds - https://launchpad.net/~duplicity-team/+archive/ubuntu/ppa

NOTE: UNinstall duplicity first if it was installed via the distribution repository. For Ubuntu, that would be "sudo apt-get purge duplicity".

Czollli (czollli) wrote :

Hi,
my report looks different but syslog says memory error and when I set up a new S3 bucket
test works just fine with the same credentials so this must be duplicity related.

Unfortunately I do not get traceback, just the last line.

- It does the same with version 0.7.14 as well as the latest stable 0.7.17
- Cache purge workaround did not work
- also worth mentioning it ONLY happens when I do FULL backup. For some reason incrementals work.
- I am on debian jessie

Processing local manifest /root/.cache/duplicity/d34b7e48840fd480561c555df1d9c5ef/duplicity-inc.20180216T030102Z.to.20180217T030102Z.manifest (4879)
Found manifest volume 1
Found 1 volumes in manifest
File duplicity-inc.20180216T030102Z.to.20180217T030102Z.vol1.difftar.gpg is part of known set
File duplicity-inc.20180217T030102Z.to.20180218T030101Z.manifest.gpg is not part of a known set; creating new set
Processing local manifest /root/.cache/duplicity/d34b7e48840fd480561c555df1d9c5ef/duplicity-inc.20180217T030102Z.to.20180218T030101Z.manifest (2411)
Found manifest volume 1
Found 1 volumes in manifest
File duplicity-inc.20180217T030102Z.to.20180218T030101Z.vol1.difftar.gpg is part of known set
File duplicity-inc.20180218T030101Z.to.20180219T030102Z.manifest.gpg is not part of a known set; creating new set
Processing local manifest /root/.cache/duplicity/d34b7e48840fd480561c555df1d9c5ef/duplicity-inc.20180218T030101Z.to.20180219T030102Z.manifest (4426)
Found manifest volume 1
Found 1 volumes in manifest
File duplicity-inc.20180218T030101Z.to.20180219T030102Z.vol1.difftar.gpg is part of known set
File duplicity-inc.20180219T030102Z.to.20180220T030102Z.manifest.gpg is not part of a known set; creating new set
Processing local manifest /root/.cache/duplicity/d34b7e48840fd480561c555df1d9c5ef/duplicity-inc.20180219T030102Z.to.20180220T030102Z.manifest (36448)
Found manifest volume 1
Found 1 volumes in manifest
File duplicity-inc.20180219T030102Z.to.20180220T030102Z.vol1.difftar.gpg is part of known set
File duplicity-inc.20180221T030101Z.to.20180222T030101Z.manifest.gpg is not part of a known set; creating new set
Processing local manifest /root/.cache/duplicity/d34b7e48840fd480561c555df1d9c5ef/duplicity-inc.20180221T030101Z.to.20180222T030101Z.manifest (5187)
Found manifest volume 1
Found 1 volumes in manifest
File duplicity-inc.20180221T030101Z.to.20180222T030101Z.vol1.difftar.gpg is part of known set
File duplicity-inc.20180222T030101Z.to.20180223T030102Z.manifest.gpg is not part of a known set; creating new set
Processing local manifest /root/.cache/duplicity/d34b7e48840fd480561c555df1d9c5ef/duplicity-inc.20180222T030101Z.to.20180223T030102Z.manifest (11396841)
./s3Backup.sh: line 24: 25935 Killed /usr/local/bin/duplicity full -vd --full-if-older-than 30D --s3-use-new-style --s3-european-buckets --include='/home/zoltan/backups/dup-s3-backup' --exclude=** / s3://s3.eu-west-2.amazonaws.com/backup-urdongo/

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers