Duplicity adds the same files again after full backup

[Kenneth Loafman]

I run a backup on my photo library and since it is the first time duplicity does a full backup. It successfully finishes by adding all the files and creating the duplicity-full* volumes and duplicity-full-signatures.* files.

Next day I run the backup again, and now duplicity start an incremental backup. However, this time around, it also adds a lot of files again. These files were already added in the first full backup and have not changed since 2010.

On the third day, I run my backup again and now duplicity is does not add all this files (as it should).

I'm running a rather large set so the double store is a good-sized nuisance. Any suggestions? How is duplicity determining that the files change/or don't?

This is running on Ubuntu 12.04 and duplicity 0.6.18 (February 29, 2012) with Python 2.7.3.

Your advice would be greatly appreciated.

[Kenneth Loafman (kenneth-loafman)]

This bug was created from question at:
https://answers.launchpad.net/duplicity/+question/248111

[Rodrigo Alvarez (rodrigo-alvarez-i)]

This problem is still present in duplicity 0.7.06. See notes on question:
https://answers.launchpad.net/duplicity/+question/248111

[Pedro Gimeno (pgimeno)]

I'm affected by this problem too.

After a full backup, I tried to make an incremental backup, and it created (and still is creating, as I write this) many more volumes than expected. Examination of the output reveals that some files that were unchanged are being included in the incremental backup again.

The OS in which it is happening is 32 bit, and the compressed signatures file for the full backup is 6 GB. That obviously exceeds the available per-process memory space in a 32 bit OS. I suspect that to be the primary cause.

Is it possible that Duplicity does not emit any out-of-memory warning when its memory is exhausted while loading a signatures file?

Rodrigo, is/was your OS 32 bit?

[Rodrigo Alvarez (rodrigo-alvarez-i)]

It looks like yes, I was running 32-bit Ubuntu 12.04.

I'm currently running Ubuntu 18.04 but my system is still 32 bit so I can only guess that it was also running a 32-bit version back then.

My systems can run 32 and 64-bit versions, but if my memory serves me right, it is last time I tried it was not trivial to update a 32-bit system into a 64-bit system without a clean install so I've been sticking with 32-bit through the years.

I changed my backup scheme to break up large sets into smaller sets via a script so I have not been observing this problem recently.

[Pedro Gimeno (pgimeno)]

Update: I managed to make the incremental backup from a 64-bit computer, mounting the drive via SSHFS. The backup was only 1 volume this time, and I didn't see any obvious case where duplicate files were included. This looks more and more like a memory problem.

[Pedro Gimeno (pgimeno)]

A more worrying problem I've just discovered is that this also affects verification. The backup was performed on the root a running system, where some files were expected to change, notably system logs. I could indeed attest that /var/log/syslog and /var/log/auth.log had changed between backup and verification. Yet the verification showed 3,553,484 files compared and 0 differences found.

Now I've re-run the verification from the 64 bit machine via sshfs. Result: 3,553,488 files compared and 202 differences found. The differences were all expected (mostly logs, but also other stuff changed in cronjobs like the mlocate database, which should have changed in the previous verification).

I'm worried that it will also affect restoration. In any case, this looks like a blocker. People may be taking backups that they can't restore.

Maybe this can be debugged by setting ulimit -v to a low value? Or with a virtual machine, assigning it lower memory?

Off-topic: Rodrigo, I had the same problem with Debian, but things are easier now. https://wiki.debian.org/CrossGrading (don't reply)