Slow backups of large files, blocksize to small ?

[sander eikelenboom]

I'm using duplicity to backup large files, but backup seem to last forever.

In seems this could be related to the blocksize used, it also seems the blocksize calculated doesn't match the comment

In file diffdir.py:

def get_block_size(file_len):
    """
    Return a reasonable block size to use on files of length file_len

    If the block size is too big, deltas will be bigger than is
    necessary. If the block size is too small, making deltas and
    patching can take a really long time.
    """
    if file_len < 1024000:
        return 512 # set minimum of 512 bytes
    else:
        # Split file into about 2000 pieces, rounding to 512
        file_blocksize = long((file_len / (2000 * 512)) * 512)
        return min(file_blocksize, 2048L)

For files larger than 2000*2048 it won't split it into 2000 pieces, but use a max of 2048 as blocksize.
Shouldn't this be:
return max(file_blocksize, 2048L)

--
Sander

[sander eikelenboom (b-linux)]

Hmm with a little more thought max() isn't very smart, there should a upperlimit.
So two alternatives:
- Perhaps bump the 2048 to say 4GB / 2000 bytes ? (that would handle dvd iso's)
- Make max_blocksize configurable, depending on the wish of better delta size or delta generation speed

[Yuri D'Elia (wavexx)]

I also incurred into this problem. Very large files become a bottleneck to the point that performing a backup becomes basicly impossible if there's any file larger than a couple of GB.

As suggested, I made the following patch, introducing --max-blocksize for both duplicity and rdiffdir (defaulting to 2048 like before).

I can now saturate my server I/O by using --max-blocksize 16777216, I don't mind the increased delta.
It's only lacking proper documentation.

[Remy van Elst (raymii)]

I've emailed the mailing list about the documentation but did not receive a response (http://lists.nongnu.org/archive/html/duplicity-talk/2014-10/msg00002.html). Can someone please explain a bit more clear what this does or update the manpage?