error uploading manifest file on incremental backup

[ljesmin]

Have a problem with duplicity incremental backups to S3.

I do backup from about 30 G of small files, with exclusion of some files by
pattern to S3, eu-central-1 bucket.

I use duplicity 0.7.11 from EPEL on CentOS 7.3. Python is 2.7.5.

Full backups work fine, but incremental backups immediately after full backup
generate errors.

Full backup command line:

duplicity full --no-encryption --s3-use-multiprocessing
--s3-multipart-chunk-size 10 --num-retries 5 --exclude "**typo3temp/**"
--exclude "**_temp/**" --exclude "**_processed_/**" /var/www
"s3://s3.eu-central-1.amazonaws.com/backup-bucket-name/www"

Incremental backup command line:

duplicity incremental --no-encryption --s3-use-multiprocessing
--s3-multipart-chunk-size 10 --num-retries 5 --exclude "**typo3temp/**"
--exclude "**_temp/**" --exclude "**_processed_/**" /var/www
"s3://s3.eu-central-1.amazonaws.com/backup-bucket-name/www"

Output on erratic incremental backup:

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Mar 19 02:15:10 2017
Giving up after 1 attempts. OSError: [Errno 2] No such file or directory:
'/root/.cache/duplicity/652b29dc986d2c5850695bd80bf5261a/duplicity-inc.
20170319T001510Z.to.20170321T001455Z.manifest'

After backup, in /root/.cache/duplicity/652b29dc986d2c5850695bd80bf5261a/ there
are such files:

duplicity-full.20170319T001510Z.manifest
duplicity-new-signatures.20170319T001510Z.to.20170321T001455Z.sigtar.gz
duplicity-inc.20170319T001510Z.to.20170321T001455Z.manifest.part

In AWS bucket are from latest backup such files:

www/duplicity-inc.20170319T001510Z.to.20170321T001455Z.vol1.difftar.gz
www/duplicity-new-signatures.20170319T001510Z.to.20170321T001455Z.sigtar.gz

And manifest file is missing.

Seems that for some reason, manifest is created as .part file and upload
function does not use this file.

[ljesmin (lauri-jesmin)]

First 200 lines of output

[ljesmin (lauri-jesmin)]

Last 200 lines of output

[Kenneth Loafman (kenneth-loafman)]

Could you do an 'ls -l' on /root/.cache/duplicity/
652b29dc986d2c5850695bd80bf5261a/?

On Tue, Mar 21, 2017 at 5:25 PM, ljesmin <email address hidden> wrote:

> Last 200 lines of output
>
> ** Attachment added: "last200.txt"
> https://bugs.launchpad.net/duplicity/+bug/1674833/+
> attachment/4842002/+files/last200.txt
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1674833
>
> Title:
> error uploading manifest file on incremental backup
>
> Status in Duplicity:
> New
>
> Bug description:
> Have a problem with duplicity incremental backups to S3.
>
> I do backup from about 30 G of small files, with exclusion of some files
> by
> pattern to S3, eu-central-1 bucket.
>
> I use duplicity 0.7.11 from EPEL on CentOS 7.3. Python is 2.7.5.
>
> Full backups work fine, but incremental backups immediately after full
> backup
> generate errors.
>
> Full backup command line:
>
> duplicity full --no-encryption --s3-use-multiprocessing
> --s3-multipart-chunk-size 10 --num-retries 5 --exclude "**typo3temp/**"
> --exclude "**_temp/**" --exclude "**_processed_/**" /var/www
> "s3://s3.eu-central-1.amazonaws.com/backup-bucket-name/www"
>
> Incremental backup command line:
>
> duplicity incremental --no-encryption --s3-use-multiprocessing
> --s3-multipart-chunk-size 10 --num-retries 5 --exclude "**typo3temp/**"
> --exclude "**_temp/**" --exclude "**_processed_/**" /var/www
> "s3://s3.eu-central-1.amazonaws.com/backup-bucket-name/www"
>
> Output on erratic incremental backup:
>
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Sun Mar 19 02:15:10 2017
> Giving up after 1 attempts. OSError: [Errno 2] No such file or directory:
> '/root/.cache/duplicity/652b29dc986d2c5850695bd80bf5261a/duplicity-inc.
> 20170319T001510Z.to.20170321T001455Z.manifest'
>
> After backup, in /root/.cache/duplicity/652b29dc986d2c5850695bd80bf5261a/
> there
> are such files:
>
> duplicity-full.20170319T001510Z.manifest
> duplicity-new-signatures.20170319T001510Z.to.20170321T001455Z.sigtar.gz
> duplicity-inc.20170319T001510Z.to.20170321T001455Z.manifest.part
>
> In AWS bucket are from latest backup such files:
>
> www/duplicity-inc.20170319T001510Z.to.20170321T001455Z.vol1.difftar.gz
> www/duplicity-new-signatures.20170319T001510Z.to.
> 20170321T001455Z.sigtar.gz
>
> And manifest file is missing.
>
> Seems that for some reason, manifest is created as .part file and upload
> function does not use this file.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1674833/+subscriptions
>

[ljesmin (lauri-jesmin)]

Upgraded it to 0.7.12, but it did not fix error, last backup is done with version 0.7.12 (.manifest.part is left behind):

drwxr-xr-x. 2 root root 4096 22. märts 02:25 .
drwxr-xr-x. 5 root root 4096 20. märts 23:59 ..
-rw-r--r--. 1 root root 23925657 8. märts 21:20 duplicity-full.20170307T173156Z.manifest
-rw-r--r--. 1 root root 24376459 8. märts 21:20 duplicity-full.20170308T161708Z.manifest
-rw-------. 1 root root 24376160 12. märts 03:51 duplicity-full.20170312T001508Z.manifest
-rw-------. 1 root root 24557672 19. märts 03:51 duplicity-full.20170319T001510Z.manifest
-rw-r--r--. 1 root root 375650731 8. märts 21:19 duplicity-full-signatures.20170307T173156Z.sigtar.gz
-rw-r--r--. 1 root root 383838030 8. märts 21:20 duplicity-full-signatures.20170308T161708Z.sigtar.gz
-rw-------. 1 root root 383887812 12. märts 03:51 duplicity-full-signatures.20170312T001508Z.sigtar.gz
-rw-------. 1 root root 384573437 19. märts 03:51 duplicity-full-signatures.20170319T001510Z.sigtar.gz
-rw-r--r--. 1 root root 1134 10. märts 00:11 duplicity-inc.20170308T161708Z.to.20170309T215557Z.manifest
-rw-------. 1 root root 171 10. märts 02:26 duplicity-inc.20170309T215557Z.to.20170310T001528Z.manifest
-rw-------. 1 root root 818 11. märts 02:26 duplicity-inc.20170310T001528Z.to.20170311T001554Z.manifest
-rw-------. 1 root root 5425019 13. märts 23:30 duplicity-inc.20170312T001508Z.to.20170313T212005Z.manifest
-rw-------. 1 root root 675 14. märts 02:24 duplicity-inc.20170313T212005Z.to.20170314T001450Z.manifest
-rw-------. 1 root root 24876 15. märts 02:28 duplicity-inc.20170314T001450Z.to.20170315T001504Z.manifest
-rw-------. 1 root root 44210 16. märts 02:24 duplicity-inc.20170315T001504Z.to.20170316T001514Z.manifest
-rw-------. 1 root root 32855 17. märts 02:24 duplicity-inc.20170316T001514Z.to.20170317T001449Z.manifest
-rw-------. 1 root root 4985 18. märts 02:25 duplicity-inc.20170317T001449Z.to.20170318T001452Z.manifest
-rw-------. 1 root root 961525 21. märts 23:58 duplicity-inc.20170319T001510Z.to.20170321T214444Z.manifest
-rw-------. 1 root root 245 22. märts 00:45 duplicity-inc.20170321T214444Z.to.20170321T223228Z.manifest
-rw-------. 1 root root 249 22. märts 02:25 duplicity-inc.20170321T223228Z.to.20170322T001524Z.manifest.part
-rw-r--r--. 1 root root 1630 10. märts 00:11 duplicity-new-signatures.20170308T161708Z.to.20170309T215557Z.sigtar.gz
-rw-------. 1 root root 45 10. märts 02:26 duplicity-new-signatures.20170309T215557Z.to.20170310T001528Z.sigtar.gz
-rw-------. 1 root root 55294 11. märts 02:26 duplicity-new-signatures.20170310T001528Z.to.20170311T001554Z.sigtar.gz
-rw-------. 1 root root 9107227 13. märts 23:30 duplicity-new-signatures.20170312T001508Z.to.20170313T212005Z.sigtar.gz
-rw-------. 1 root root 907 14. märts 02:24 duplicity-new-signatures.20170313T212005Z.to.20170314T001450Z.sigtar.gz
-rw-------. 1 root root 112011 15. märts 02:28 duplicity-new-signatures.20170314T001450Z.to.20170315T001504Z.sigtar.gz
-rw-------. 1 root root 301068 16. märts 02:24 duplicity-new-signatures.20170315T001504Z.to.20170316T001514Z.sigtar.gz
-rw-------. 1 ro...

[Kenneth Loafman (kenneth-loafman)]

Try removing the .part file. That should solve the problem.

...Ken

On Wed, Mar 22, 2017 at 2:22 PM, ljesmin <email address hidden> wrote:

> Upgraded it to 0.7.12, but it did not fix error, last backup is done with
> version 0.7.12 (.manifest.part is left behind):
>
> drwxr-xr-x. 2 root root 4096 22. märts 02:25 .
> drwxr-xr-x. 5 root root 4096 20. märts 23:59 ..
> -rw-r--r--. 1 root root 23925657 8. märts 21:20 duplicity-full.
> 20170307T173156Z.manifest
> -rw-r--r--. 1 root root 24376459 8. märts 21:20 duplicity-full.
> 20170308T161708Z.manifest
> -rw-------. 1 root root 24376160 12. märts 03:51 duplicity-full.
> 20170312T001508Z.manifest
> -rw-------. 1 root root 24557672 19. märts 03:51 duplicity-full.
> 20170319T001510Z.manifest
> -rw-r--r--. 1 root root 375650731 8. märts 21:19
> duplicity-full-signatures.20170307T173156Z.sigtar.gz
> -rw-r--r--. 1 root root 383838030 8. märts 21:20
> duplicity-full-signatures.20170308T161708Z.sigtar.gz
> -rw-------. 1 root root 383887812 12. märts 03:51
> duplicity-full-signatures.20170312T001508Z.sigtar.gz
> -rw-------. 1 root root 384573437 19. märts 03:51
> duplicity-full-signatures.20170319T001510Z.sigtar.gz
> -rw-r--r--. 1 root root 1134 10. märts 00:11 duplicity-inc.
> 20170308T161708Z.to.20170309T215557Z.manifest
> -rw-------. 1 root root 171 10. märts 02:26 duplicity-inc.
> 20170309T215557Z.to.20170310T001528Z.manifest
> -rw-------. 1 root root 818 11. märts 02:26 duplicity-inc.
> 20170310T001528Z.to.20170311T001554Z.manifest
> -rw-------. 1 root root 5425019 13. märts 23:30 duplicity-inc.
> 20170312T001508Z.to.20170313T212005Z.manifest
> -rw-------. 1 root root 675 14. märts 02:24 duplicity-inc.
> 20170313T212005Z.to.20170314T001450Z.manifest
> -rw-------. 1 root root 24876 15. märts 02:28 duplicity-inc.
> 20170314T001450Z.to.20170315T001504Z.manifest
> -rw-------. 1 root root 44210 16. märts 02:24 duplicity-inc.
> 20170315T001504Z.to.20170316T001514Z.manifest
> -rw-------. 1 root root 32855 17. märts 02:24 duplicity-inc.
> 20170316T001514Z.to.20170317T001449Z.manifest
> -rw-------. 1 root root 4985 18. märts 02:25 duplicity-inc.
> 20170317T001449Z.to.20170318T001452Z.manifest
> -rw-------. 1 root root 961525 21. märts 23:58 duplicity-inc.
> 20170319T001510Z.to.20170321T214444Z.manifest
> -rw-------. 1 root root 245 22. märts 00:45 duplicity-inc.
> 20170321T214444Z.to.20170321T223228Z.manifest
> -rw-------. 1 root root 249 22. märts 02:25 duplicity-inc.
> 20170321T223228Z.to.20170322T001524Z.manifest.part
> -rw-r--r--. 1 root root 1630 10. märts 00:11 duplicity-new-signatures.
> 20170308T161708Z.to.20170309T215557Z.sigtar.gz
> -rw-------. 1 root root 45 10. märts 02:26 duplicity-new-signatures.
> 20170309T215557Z.to.20170310T001528Z.sigtar.gz
> -rw-------. 1 root root 55294 11. märts 02:26 duplicity-new-signatures.
> 20170310T001528Z.to.20170311T001554Z.sigtar.gz
> -rw-------. 1 root root 9107227 13. märts 23:30 duplicity-new-signatures.
> 20170312T001508Z.to.20170313T212005Z.sigtar.gz
> -rw-------. 1 root root 907 14. märts 02:24 duplicity-new-signatures.
> 20170313T212005Z.to.20170314T00...

[ljesmin (lauri-jesmin)]

Thanks, but it does not really solve my problem.

I do know, that when i remove this .part file and also latest backup files from S3, the backup will continue.

Other solution is to rename manifest file removing .part from filename and upload it to S3 - and then will even latest backup work.

However, my problem is that this error appears after every full backup.

So currently, after every full backup i have to clean up and then sometimes second incremental works, sometimes not.

[Kenneth Loafman (kenneth-loafman)]

What boto version are you using?

On Mon, Mar 27, 2017 at 2:42 PM, ljesmin <email address hidden> wrote:

> Thanks, but it does not really solve my problem.
>
> I do know, that when i remove this .part file and also latest backup
> files from S3, the backup will continue.
>
> Other solution is to rename manifest file removing .part from filename
> and upload it to S3 - and then will even latest backup work.
>
> However, my problem is that this error appears after every full backup.
>
> So currently, after every full backup i have to clean up and then
> sometimes second incremental works, sometimes not.
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1674833
>
> Title:
> error uploading manifest file on incremental backup
>
> Status in Duplicity:
> New
>
> Bug description:
> Have a problem with duplicity incremental backups to S3.
>
> I do backup from about 30 G of small files, with exclusion of some files
> by
> pattern to S3, eu-central-1 bucket.
>
> I use duplicity 0.7.11 from EPEL on CentOS 7.3. Python is 2.7.5.
>
> Full backups work fine, but incremental backups immediately after full
> backup
> generate errors.
>
> Full backup command line:
>
> duplicity full --no-encryption --s3-use-multiprocessing
> --s3-multipart-chunk-size 10 --num-retries 5 --exclude "**typo3temp/**"
> --exclude "**_temp/**" --exclude "**_processed_/**" /var/www
> "s3://s3.eu-central-1.amazonaws.com/backup-bucket-name/www"
>
> Incremental backup command line:
>
> duplicity incremental --no-encryption --s3-use-multiprocessing
> --s3-multipart-chunk-size 10 --num-retries 5 --exclude "**typo3temp/**"
> --exclude "**_temp/**" --exclude "**_processed_/**" /var/www
> "s3://s3.eu-central-1.amazonaws.com/backup-bucket-name/www"
>
> Output on erratic incremental backup:
>
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Sun Mar 19 02:15:10 2017
> Giving up after 1 attempts. OSError: [Errno 2] No such file or directory:
> '/root/.cache/duplicity/652b29dc986d2c5850695bd80bf5261a/duplicity-inc.
> 20170319T001510Z.to.20170321T001455Z.manifest'
>
> After backup, in /root/.cache/duplicity/652b29dc986d2c5850695bd80bf5261a/
> there
> are such files:
>
> duplicity-full.20170319T001510Z.manifest
> duplicity-new-signatures.20170319T001510Z.to.20170321T001455Z.sigtar.gz
> duplicity-inc.20170319T001510Z.to.20170321T001455Z.manifest.part
>
> In AWS bucket are from latest backup such files:
>
> www/duplicity-inc.20170319T001510Z.to.20170321T001455Z.vol1.difftar.gz
> www/duplicity-new-signatures.20170319T001510Z.to.
> 20170321T001455Z.sigtar.gz
>
> And manifest file is missing.
>
> Seems that for some reason, manifest is created as .part file and upload
> function does not use this file.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1674833/+subscriptions
>

[ljesmin (lauri-jesmin)]

My Boto is from EPEL, versions are:

python2-botocore-1.5.18-1.el7.noarch
python2-boto-2.44.0-1.el7.noarch