Bugfix for S3 multipart upload on 0.6.16

Bug #881070 reported by Henrique Carvalho Alves on 2011-10-24
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Duplicity
High
Unassigned

Bug Description

I'm not sure if the last patch I sent was tested, but a bug slipped into which would make uploads fail. I just noticed after I changed from my branch to use the released package. I'm attaching the correct version of the patch, with this bug fixed.

I suggest a new release ASAP as multipart upload is currently broken.

derp herp (junkmail-trash) wrote :

Could you submit a patch vs 0.6.16?

Changed in duplicity:
status: New → Incomplete
importance: Undecided → High
assignee: nobody → Kenneth Loafman (kenneth-loafman)

I'm attaching a new patch against 0.6.16 for fixing 3 things:

- Multiprocessing support on BSD systems
- Including FileChunkIO in the source tree
- Bugfix on chunk upload code

It will fix all the bugs reported on bug 676109

I just pushed this to the trunk. Please test.

Changed in duplicity:
status: Incomplete → Fix Committed
assignee: Kenneth Loafman (kenneth-loafman) → nobody
Changed in duplicity:
milestone: none → 0.7.00
milestone: 0.7.00 → none

I can upload correctly, but restoring actually gives me an error.

Traceback:

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Mon Nov 7 15:50:12 2011
Chain end time: Mon Nov 7 15:50:12 2011
Number of contained backup sets: 1
Total number of contained volumes: 0
Tipo de conjunto de cópia de segurança Time: Número de volumes:
-------------------------
Also found 0 backup sets not part of any chain,
and 5 incomplete backup sets.
These may be deleted by running duplicity with the "cleanup" command.
Removing still remembered temporary file /var/folders/k1/fctx1v3x5xbgkcslg16kl5tm0000gn/T/duplicity-jfP864-tempdir/mkstemp-WqXgD5-1
Traceback (most recent call last):
  File "./duplicity-bin", line 1347, in <module>
    with_tempdir(main)
  File "./duplicity-bin", line 1330, in with_tempdir
    fn()
  File "./duplicity-bin", line 1264, in main
    restore(col_stats)
  File "./duplicity-bin", line 591, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 522, in Write_ROPaths
    for ropath in rop_iter:
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 494, in integrate_patch_iters
    for patch_seq in collated:
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 379, in yield_tuples
    setrorps( overflow, elems )
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 368, in setrorps
    elems[i] = iter_list[i].next()
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 112, in difftar2path_iter
    tarinfo_list = [tar_iter.next()]
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 329, in next
    self.set_tarfile()
  File "/Users/hcalves/Projetos/duplicity/duplicity/patchdir.py", line 323, in set_tarfile
    self.current_fp = self.fileobj_iter.next()
  File "./duplicity-bin", line 627, in get_fileobj_iter
    backup_set.volume_name_dict[vol_num],
KeyError: 1

The restore error might actually be bug 885670, so I suppose the upload operation is working as intended given Duplicity no more fails at checksum (giving the "the file was corrupted during upload").

Changed in duplicity:
milestone: none → 0.6.17
Mike Rich (l-launchpad-origin) wrote :

This also fixes the problem with the patch in bug 871875, specifically, it's this chunk:
@@ -388,7 +396,7 @@ def multipart_upload_worker(scheme, url, bucket_name, multipart_id, filename,

             for mp in bucket.get_all_multipart_uploads():
                 if mp.id == multipart_id:
- with FileChunkIO(filename, 'r', offset=offset, bytes=bytes) as fd:
+ with FileChunkIO(filename, 'r', offset=offset * bytes, bytes=bytes) as fd:
                         mp.upload_part_from_file(fd, offset + 1, cb=_upload_callback)
                     break
         except Exception, e:

So 871875 can be closed after this is pushed

Mike Rich (l-launchpad-origin) wrote :

Also, FileChunkIO requires 2.6+, which means duplicity will require 2.6+ now - the "Requirements" may need to be updated on the main page (http://duplicity.nongnu.org/), and possibly the Changelog?

Changed in duplicity:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers