OverflowError 2147483647 bytes

[alivar]

Hi all,

We have a lot data to backup.
After uploading backup Duplicity return error:

Attempt 1 failed. OverflowError: string longer than 2147483647 bytes
Attempt 2 failed. OverflowError: string longer than 2147483647 bytes
etc...

I running duplicity with command:
duplicity -v2 --full-if-older-than=1M --timeout 120 --volsize 512 --max-blocksize=20480 --asynchronous-upload --num-retries 40 --ssl-no-check-certificate --include /root --include /var/spool/cron --include /var/www --include /etc --exclude '**' / webdavs://***/

Tried to add max-blocksize, but it doesn't solve the problem

[Kenneth Loafman (kenneth-loafman)]

What version of duplicity? What OS and version? Please rerun with -v9 and paste the log as an attachment to this bug report. If it's too long, please paste the first 200 lines and the last 200 lines.

[alivar (zalivar)]

Good day Kenneth,

duplicity 0.7.11 (December 31, 2016), compiled from source,
OS is Debian 3.16.7 x64

First 200 lines:

Using archive dir: /root/.cache/duplicity/4b1098e1cfb965f25f34536e85fb4d39
Using backup name: 4b1098e1cfb965f25f34536e85fb4d39
Import of duplicity.backends.acdclibackend Succeeded
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.b2backend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.copycombackend Succeeded
Import of duplicity.backends.dpbxbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.lftpbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.mediafirebackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.multibackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Using WebDAV protocol http
Using WebDAV host webdav.yandex.ru port None
Using WebDAV directory /backup/rent1/
Main action: inc
================================================================================
duplicity 0.7.11 (December 31, 2016)
Args: /usr/local/bin/duplicity -v9 --full-if-older-than=1M --timeout 120 --volsize 512 --max-blocksize=20480 --asynchronous-upload --num-retries 40 --ssl-no-check-certificate --include /root --include /var/spool/cron --include /var/www --include /etc --exclude ** / webdavs://@@@@@@@webdav.yandex.ru/backup/rent1
Linux rent1 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-1 (2016-03-06) x86_64
/usr/bin/python2 2.7.9 (default, Jun 29 2016, 13:08:31)
[GCC 4.9.2]
================================================================================
Using temporary directory /home/duplicity/duplicity-d4hrlK-tempdir
Registering (mkstemp) temporary file /home/duplicity/duplicity-d4hrlK-tempdir/mkstemp-NDs679-1
Temp has 2404709822464 available, backup will use approx 1234803097.
WebDAV create connection on 'webdav.yandex.ru'
WebDAV PROPFIND /backup/rent1/ request with headers: {'Connection': 'keep-alive', 'Depth': '1'}
WebDAV data length: 95
WebDAV response status 401 with reason 'Unauthorized'.
WebDAV retry request with authentification headers.
WebDAV PROPFIND /backup/rent1/ request2 with headers: {'Connection': 'keep-alive', 'Depth': '1', 'Authorization': 'Basic @@@@@@@@@@@@'}
WebDAV data length: 95
WebDAV response2 status 207 with...

[Kenneth Loafman (kenneth-loafman)]

You've hit this bug:
https://bugs.launchpad.net/duplicity/+bug/385495https://bugs.launchpad.net/duplicity/+bug/385495

Two suggestions:

- Increase --max-blocksize to 65536 or 131072 to reduce the sigtar file size.
- Split the backup into two or more smaller runs.

[alivar (zalivar)]

I tried first solution.

But i got the same error

Attempt 39 failed. OverflowError: string longer than 2147483647 bytes
Giving up after 40 attempts. OverflowError: string longer than 2147483647 bytes

command:
duplicity -v2 --full-if-older-than=1M --timeout 120 --volsize 1024 --max-blocksize 131072 --asynchronous-upload --num-retries 40 --ssl-no-check-certificate --include /root --include /var/spool/cron --include /var/www --include /etc --exclude '**' / webdavs://***@webdav.yandex.ru <webdavs://***@webdav.yandex.ru>/** <webdavs://***@webdav.yandex.ru**>/

Is any chance to success, if I set bigger value.

> 11 марта 2017 г., в 16:43, Kenneth Loafman <email address hidden> написал(а):
>
> *** This bug is a duplicate of bug 385495 ***
> https://bugs.launchpad.net/bugs/385495
>
> You've hit this bug:
> https://bugs.launchpad.net/duplicity/+bug/385495https://bugs.launchpad.net/duplicity/+bug/385495
>
> Two suggestions:
>
> - Increase --max-blocksize to 65536 or 131072 to reduce the sigtar file size.
> - Split the backup into two or more smaller runs.
>
>
> ** This bug has been marked a duplicate of bug 385495
> Large backup signature and manifest files should be split with --volsize too
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1671755
>
> Title:
> OverflowError 2147483647 bytes
>
> Status in Duplicity:
> New
>
> Bug description:
> Hi all,
>
> We have a lot data to backup.
> After uploading backup Duplicity return error:
>
> Attempt 1 failed. OverflowError: string longer than 2147483647 bytes
> Attempt 2 failed. OverflowError: string longer than 2147483647 bytes
> etc...
>
> I running duplicity with command:
> duplicity -v2 --full-if-older-than=1M --timeout 120 --volsize 512 --max-blocksize=20480 --asynchronous-upload --num-retries 40 --ssl-no-check-certificate --include /root --include /var/spool/cron --include /var/www --include /etc --exclude '**' / webdavs://***/
>
> Tried to add max-blocksize, but it doesn't solve the problem
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1671755/+subscriptions

[Kenneth Loafman (kenneth-loafman)]

Yes, that should reduce the sigtar volsize.

On Mon, Mar 13, 2017 at 1:49 AM, alivar <email address hidden> wrote:

> *** This bug is a duplicate of bug 385495 ***
> https://bugs.launchpad.net/bugs/385495
>
> I tried first solution.
>
> But i got the same error
>
> Attempt 39 failed. OverflowError: string longer than 2147483647 bytes
> Giving up after 40 attempts. OverflowError: string longer than 2147483647
> bytes
>
> command:
> duplicity -v2 --full-if-older-than=1M --timeout 120 --volsize 1024
> --max-blocksize 131072 --asynchronous-upload --num-retries 40
> --ssl-no-check-certificate --include /root --include /var/spool/cron
> --include /var/www --include /etc --exclude '**' / webdavs://***@
> webdav.yandex.ru <webdavs://***@webdav.yandex.ru>/** <webdavs://***@
> webdav.yandex.ru**>/
>
> Is any chance to success, if I set bigger value.
>
> > 11 марта 2017 г., в 16:43, Kenneth Loafman <email address hidden>
> написал(а):
> >
> > *** This bug is a duplicate of bug 385495 ***
> > https://bugs.launchpad.net/bugs/385495
> >
> > You've hit this bug:
> > https://bugs.launchpad.net/duplicity/+bug/385495https://
> bugs.launchpad.net/duplicity/+bug/385495
> >
> > Two suggestions:
> >
> > - Increase --max-blocksize to 65536 or 131072 to reduce the sigtar file
> size.
> > - Split the backup into two or more smaller runs.
> >
> >
> > ** This bug has been marked a duplicate of bug 385495
> > Large backup signature and manifest files should be split with
> --volsize too
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1671755
> >
> > Title:
> > OverflowError 2147483647 bytes
> >
> > Status in Duplicity:
> > New
> >
> > Bug description:
> > Hi all,
> >
> > We have a lot data to backup.
> > After uploading backup Duplicity return error:
> >
> > Attempt 1 failed. OverflowError: string longer than 2147483647 bytes
> > Attempt 2 failed. OverflowError: string longer than 2147483647 bytes
> > etc...
> >
> > I running duplicity with command:
> > duplicity -v2 --full-if-older-than=1M --timeout 120 --volsize 512
> --max-blocksize=20480 --asynchronous-upload --num-retries 40
> --ssl-no-check-certificate --include /root --include /var/spool/cron
> --include /var/www --include /etc --exclude '**' / webdavs://***/
> >
> > Tried to add max-blocksize, but it doesn't solve the problem
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/duplicity/+bug/1671755/+subscriptions
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1671755
>
> Title:
> OverflowError 2147483647 bytes
>
> Status in Duplicity:
> New
>
> Bug description:
> Hi all,
>
> We have a lot data to backup.
> After uploading backup Duplicity return error:
>
> Attempt 1 failed. OverflowError: string longer than 2147483647 bytes
> Attempt 2 failed. OverflowError: string longer than 2147483647 bytes
> etc...
>
> I running duplicity with command:
> duplicity -v2 --full-if-older-than=1M --timeout 120 --volsize 512
> --max-blocksize=20480 --asynchronous-upload --num-retries 40
> --ssl-no-c...

[Maurus Cuelenaere (mcuelenaere)]

https://code.launchpad.net/~mcuelenaere/duplicity/duplicity/+merge/358411 should fix this issue.