Bugfix for S3 multipart upload on 0.6.16

[Henrique Carvalho Alves]

I'm not sure if the last patch I sent was tested, but a bug slipped into which would make uploads fail. I just noticed after I changed from my branch to use the released package. I'm attaching the correct version of the patch, with this bug fixed.

I suggest a new release ASAP as multipart upload is currently broken.

[derp herp (junkmail-trash)]

Could you submit a patch vs 0.6.16?

[Henrique Carvalho Alves (hcarvalhoalves)]

I'm attaching a new patch against 0.6.16 for fixing 3 things:

- Multiprocessing support on BSD systems
- Including FileChunkIO in the source tree
- Bugfix on chunk upload code

It will fix all the bugs reported on bug 676109

[Kenneth Loafman (kenneth-loafman)]

I just pushed this to the trunk. Please test.

[Henrique Carvalho Alves (hcarvalhoalves)]

I can upload correctly, but restoring actually gives me an error.

Traceback:

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Mon Nov 7 15:50:12 2011
Chain end time: Mon Nov 7 15:50:12 2011
Number of contained backup sets: 1
Total number of contained volumes: 0
Tipo de conjunto de cópia de segurança Time: Número de volumes:
-------------------------
Also found 0 backup sets not part of any chain,
and 5 incomplete backup sets.
These may be deleted by running duplicity with the "cleanup" command.
Removing still remembered temporary file /var/folders/k1/fctx1v3x5xbgkcslg16kl5tm0000gn/T/duplicity-jfP864-tempdir/mkstemp-WqXgD5-1
Traceback (most recent call last):
  File "./duplicity-bin", line 1347, in <module>
    with_tempdir(main)
  File "./duplicity-bin", line 1330, in with_tempdir
    fn()
  File "./duplicity-bin", line 1264, in main
    restore(col_stats)
  File "./duplicity-bin", line 591, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 522, in Write_ROPaths
    for ropath in rop_iter:
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 494, in integrate_patch_iters
    for patch_seq in collated:
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 379, in yield_tuples
    setrorps( overflow, elems )
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 368, in setrorps
    elems[i] = iter_list[i].next()
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 112, in difftar2path_iter
    tarinfo_list = [tar_iter.next()]
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 329, in next
    self.set_tarfile()
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 323, in set_tarfile
    self.current_fp = self.fileobj_iter.next()
  File "./duplicity-bin", line 627, in get_fileobj_iter
    backup_set.volume_name_dict[vol_num],
KeyError: 1

[Henrique Carvalho Alves (hcarvalhoalves)]

The restore error might actually be bug 885670, so I suppose the upload operation is working as intended given Duplicity no more fails at checksum (giving the "the file was corrupted during upload").

[Mike Rich (l-launchpad-origin)]

This also fixes the problem with the patch in bug 871875, specifically, it's this chunk:
@@ -388,7 +396,7 @@ def multipart_upload_worker(scheme, url, bucket_name, multipart_id, filename,

             for mp in bucket.get_all_multipart_uploads():
                 if mp.id == multipart_id:
- with FileChunkIO(filename, 'r', offset=offset, bytes=bytes) as fd:
+ with FileChunkIO(filename, 'r', offset=offset * bytes, bytes=bytes) as fd:
                         mp.upload_part_from_file(fd, offset + 1, cb=_upload_callback)
                     break
         except Exception, e:

So 871875 can be closed after this is pushed

[Mike Rich (l-launchpad-origin)]

Also, FileChunkIO requires 2.6+, which means duplicity will require 2.6+ now - the "Requirements" may need to be updated on the main page (http://duplicity.nongnu.org/), and possibly the Changelog?