Comment 1 for bug 914504

I experience the same effect with a somewhat similar setup:

Initial full-backup of a 570GB directory-tree, (containing 100 subdirectories and 161906 files in total.) passed happily and created 564 one-GB volumes "*.difftar.gz" (with --no-encryption and --volsize 1024) plus a two-GB .sigtar.gz and a 161k .manifest file.

Then I reran duplicity with the same settings, and the incremential backup started to push ANOTHER 445 volumes, which (according to the duplicity-inc-*.manifest) started at the file that was part of vol 119 of the previous full backup.

I have a number of further cases of such high filecount backups, and observed the following:
The intermediate "duplicity-*.sigtar.part" files grow (in my case of a 2.8TB directory-tree) up to 25GB (!) in size, but the resulting .sigtar.gz files are NEVER greater than 2147497908 Bytes (=2GB + 14260 Bytes) and the "plateau" of .sigtar.gz filesizes starts at about 2GB minus 27kB.

I gunzip'ed my 2147485382 Bytes .sigtar.gz of the above 570GB backup and searched "strings -n 10" in the tail -5000 the resulting 2170225751 Bytes .sigtar file. Now I found my original files that were part of the vol118.difftar.gz to be contained last in the (TRUNCATED?) sigtar file.

My assumption for the kindly coring supporters of this wonderfull project are:
The compression of the (large) sigtar.part file into sigtar.gz files seems to stop at about 2GB filesize. Resulting in any file of the initial full backup that was listed after the 2GB position to be recognized as "new/changed" file in an incremental backup. Additionally the files that were backuped in the full backup but have their signatures stored behind the 2GB limit cannot be rsync-updated fior small changes and maybe (untested) not even restorred at al. I.e. these files are lost and only waste space in the full-backup set.
Maybe this issue could be associated with https://bugs.launchpad.net/duplicity/+bug/385495/comments/8 (Requesting large signatrues to also be split, because of possible file-size limitation on the backup-storage.)

BTW: I successfully gziP'ed manually a 25.57GB .sigtar.part into a 25.04GB .sigtar.part.gz - so the error cause is not the gzip executable (v1.3.5) or my filesystem (ZFS).

Technical details:
Solaris 10, duplicity 0.6.15, python 2.4