Segment upload failures should abort other segment jobs

Bug #1527017 reported by Tim Burke
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
python-swiftclient
Confirmed
Medium
Unassigned

Bug Description

Currently, swiftclient will refuse to upload a large object manifest if any segments failed to upload. This is good, but it will still try to upload every segment, even in the face of failure:

$ swift upload -S 1M --use-slo foo large.object
large.object segment 1
large.object segment 7
large.object segment 6
large.object segment 8
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000004', ...) failure and no ability to reset contents for reupload.
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000003', ...) failure and no ability to reset contents for reupload.
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000009', ...) failure and no ability to reset contents for reupload.
large.object segment 2
large.object segment 0
large.object segment 5
large.object segment 10
large.object segment 12
large.object segment 11
large.object segment 13
large.object segment 14
large.object segment 18
large.object segment 19
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000015', ...) failure and no ability to reset contents for reupload.
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000017', ...) failure and no ability to reset contents for reupload.
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000016', ...) failure and no ability to reset contents for reupload.
large.object segment 20
large.object segment 21
large.object segment 22
large.object segment 24
large.object segment 25
large.object segment 26
large.object segment 27
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000023', ...) failure and no ability to reset contents for reupload.
large.object segment 35
large.object segment 33
large.object segment 34
large.object segment 36
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000028', ...) failure and no ability to reset contents for reupload.
large.object segment 37
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000029', ...) failure and no ability to reset contents for reupload.
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000030', ...) failure and no ability to reset contents for reupload.
large.object segment 39
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000031', ...) failure and no ability to reset contents for reupload.
large.object segment 38
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000032', ...) failure and no ability to reset contents for reupload.
large.object segment 42
large.object segment 41
large.object segment 43
large.object segment 46
large.object segment 48
large.object segment 47
large.object segment 49
large.object segment 50
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000040', ...) failure and no ability to reset contents for reupload.
large.object segment 52
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000044', ...) failure and no ability to reset contents for reupload.
large.object segment 53
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000045', ...) failure and no ability to reset contents for reupload.
large.object segment 54
large.object segment 55
large.object segment 56
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000051', ...) failure and no ability to reset contents for reupload.
large.object segment 58
large.object segment 57
large.object segment 61
large.object segment 63
large.object segment 64
large.object segment 68
large.object segment 69
large.object segment 70
large.object segment 71
large.object segment 72
large.object segment 74
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000059', ...) failure and no ability to reset contents for reupload.
large.object segment 75
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000060', ...) failure and no ability to reset contents for reupload.
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000062', ...) failure and no ability to reset contents for reupload.
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000065', ...) failure and no ability to reset contents for reupload.
large.object segment 77
large.object segment 78
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000066', ...) failure and no ability to reset contents for reupload.
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000067', ...) failure and no ability to reset contents for reupload.
large.object segment 79
large.object segment 80
large.object segment 81
large.object segment 83
large.object segment 85
large.object segment 84
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000073', ...) failure and no ability to reset contents for reupload.
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000076', ...) failure and no ability to reset contents for reupload.
put_object(u'foo_segments', u'large.object/slo/1449867353.000000/90044871/1048576/00000082', ...) failure and no ability to reset contents for reupload.
Aborting manifest creation because not all segments could be uploaded. foo/large.object

So now my upload failed *and* I wasted a bunch of time and bandwidth. We should abort the rest of this large-object upload on the first segment failure.

Tags: dlo slo
Revision history for this message
Joel Wright (joel-wright-k) wrote :

We should be able to fix this by setting 'fail_fast' to True in the options dict passed in to the SwiftService instance.

Revision history for this message
Joel Wright (joel-wright-k) wrote :

The big question is where to add the 'fail_fast' setting - global, for the entire list of objects to upload, or just for large object uploads...?

Changed in python-swiftclient:
status: New → Confirmed
Changed in python-swiftclient:
assignee: nobody → Joel Wright (joel-wright-k)
importance: Undecided → Medium
Changed in python-swiftclient:
assignee: Joel Wright (joel-wright-k) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.