md5sum mismatch

[fractil]

affected version: s3cmd version 1.1.0-beta3

a md5 mismatch occurs when performing a PUT. Possibly because it is performing a multipart PUT?

# s3cmd --config=/opt/compliance/.s3cfg put WIRELINE-TO-WIRELESS-NORANGE_073015.txt s3://tcn-compliance/tcpa/WIRELINE-TO-WIRELESS-NORANGE_073015.txt -> s3://tcn-compliance/tcpa/WIRELINE-TO-WIRELESS-NORANGE_073015.txt [part 1 of 4, 15MB]
 15728640 of 15728640 100% in 2s 6.02 MB/s done
WIRELINE-TO-WIRELESS-NORANGE_073015.txt -> s3://tcn-compliance/tcpa/WIRELINE-TO-WIRELESS-NORANGE_073015.txt [part 2 of 4, 15MB]
 15728640 of 15728640 100% in 2s 6.03 MB/s done
WIRELINE-TO-WIRELESS-NORANGE_073015.txt -> s3://tcn-compliance/tcpa/WIRELINE-TO-WIRELESS-NORANGE_073015.txt [part 3 of 4, 15MB]
 15728640 of 15728640 100% in 2s 5.68 MB/s done
WIRELINE-TO-WIRELESS-NORANGE_073015.txt -> s3://tcn-compliance/tcpa/WIRELINE-TO-WIRELESS-NORANGE_073015.txt [part 4 of 4, 7MB]
 7586259 of 7586259 100% in 1s 3.97 MB/s done

# s3cmd --config=/opt/compliance/.s3cfg info s3://tcn-compliance/tcpa/WIRELINE-TO-WIRELESS-NORANGE_073015.txt
s3://tcn-compliance/tcpa/WIRELINE-TO-WIRELESS-NORANGE_073015.txt (object):
   File size: 54772179
   Last mod: Thu, 30 Jul 2015 19:49:19 GMT
   MIME type: text/plain; charset=us-ascii
   MD5 sum: b043e8ba574568f4f5dbfcc073c3c782-4
   ACL: admin: FULL_CONTROL

# md5sum /opt/compliance/server/tcpa/files/WIRELINE-TO-WIRELESS-NORANGE_073015.txt | grep -o "[0-9a-f]\{32\}"
b9ca30c4fdfdc7a25ca041c0208428f0

when a GET is performed s3cmd also reports the discrepancy:

# s3cmd --config=/opt/compliance/.s3cfg get s3://tcn-compliance/tcpa/WIRELINE-TO-WIRELESS-NORANGE_073015.txt
s3://tcn-compliance/tcpa/WIRELINE-TO-WIRELESS-NORANGE_073015.txt -> ./WIRELINE-TO-WIRELESS-NORANGE_073015.txt [1 of 1]
 54772179 of 54772179 100% in 3s 13.21 MB/s done
WARNING: MD5 signatures do not match: computed=b9ca30c4fdfdc7a25ca041c0208428f0, received="b043e8ba574568f4f5dbfcc073c3c782-4"

the original file and the version placed and retrieved from the S3 bucket are identical, but the md5sum mismatches are affecting automated GET/PUT processes.

This does not occur using version 1.0.0-1 (Precise), or the current s3cmd stable 1.0.0-4

[Moses Moore (moses-ubuntu)]

I believe the Etag is expected to have an md5sum of the payload, but in the case of multipart uploads, Amazon stores the md5sum of the most recent piece uploaded, with "-" and the piece number appended. You saw that your upload was in four pieces, and "-4" was appended to what was supposed to be the MD5 checksum.

The problem is still present in s3cmd 1.5.3, and I suspect it is a mistake made by Amazon, not by the s3cmd developers.

One suggested workaround was to use 's3cmd mv realname baloney ; s3cmd mv baloney realname ' to force Amazon S3 to recompute the md5sum... but IIRC this doesn't really move (or re-key) the file so much as download, reupload, then delete the original, so if the file is larger than multipart-chunk-size then it will reupload in pieces, recreating the problem.

[Gianfranco Costamagna (costamagnagianfranco)]

Looks a really old and probably already fixed bug. Closing