| 2018-12-29 10:01:44 |
Bhaskar Singhal |
description |
I am trying to upload an object (100+MB) to swift3/swift using the AWS .Net SDK multi-segment upload ( https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileDotNet.html ).
But the upload keeps failing with 'Amazon.Runtime.AmazonClientException' Expected hash not equal to calculated hash.
This is happening because the ETag calculated in obj/server is passed on the entire chunk, but in this case, the chunk has the chunk-signature etc. We need to get the decoded content and then calculate ETag.
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
Various headers set in the request:
{'HTTP_AUTHORIZATION': 'AWS4-HMAC-SHA256 Credential=4a203e8a63cc4fdfa4bbd3ea6c208bac/20181228/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=e7d32398ec602bd2fbfbbcd613ba26835d2036f6215862255a04ee91dadb242e', 'SCRIPT_NAME': '', 'keystone.token_auth': <keystonemiddleware.auth_token._user_plugin.UserAuthPlugin object at 0x7f00e68d5b90>, 'webob.adhoc_attrs': {'response': <_AuthTokenResponse at 0x7f00e68d5f50 200 OK>}, 'REQUEST_METHOD': 'PUT', 'HTTP_X_AMZ_DATE': '20181228T181903Z', 'PATH_INFO': '/bucket1/BBBunny123', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': 'partNumber=4&uploadId=YzQ0Yjk2MzMtMzdhYy00NDkyLWEwZjItZjBhYjA0M2FjYjMx', 'REMOTE_ADDR': '150.50.2.89', 'CONTENT_LENGTH': '5246566', 'HTTP_X_AMZ_DECODED_CONTENT_LENGTH': '5242880', 'HTTP_USER_AGENT': 'aws-sdk-dotnet-45/3.3.8.2 aws-sdk-dotnet-core/3.3.17.0 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_Windows_NT_6.2.9200.0 ClientAsync TransferManager/MultipartUploadCommand', 'eventlet.posthooks': [], 'RAW_PATH_INFO': '/bucket1/BBBunny123', 'REMOTE_PORT': '58993', 'eventlet.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_X_IDENTITY_STATUS': 'Invalid', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '80', 'HTTP_X_AMZ_CONTENT_SHA256': 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'wsgi.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_HOST': 'vsa-00000ad7-zadara-dev2.zadarazios.com', 'wsgi.multithread': True, 'HTTP_EXPECT': '100-continue', 'wsgi.version': (1, 0), 'SERVER_NAME': '150.50.2.132', 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <swift.common.utils.LoggerFileObject object at 0x7f00df2665d0>, 'wsgi.multiprocess': False, 'swift.trans_id': 'tx8f4b1c9093d1422389664-005c266917', 'CONTENT_TYPE': 'text/plain', 'swift.cache': <swift.common.memcached.MemcacheRing object at 0x7f00df2667d0>} |
I am trying to upload an object (100+MB) to swift3/swift using the AWS .Net SDK multi-segment upload ( https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileDotNet.html ).
But the upload keeps failing with 'Amazon.Runtime.AmazonClientException' Expected hash not equal to calculated hash.
This is happening because the ETag calculated in obj/server is passed on the entire chunk, but in this case, the chunk has the chunk-signature etc. We need to get the decoded content and then calculate ETag.
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
Various headers set in the request:
{'HTTP_AUTHORIZATION': 'AWS4-HMAC-SHA256 Credential=4a203e8a63cc4fdfa4bbd3ea6c208bac/20181228/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=e7d32398ec602bd2fbfbbcd613ba26835d2036f6215862255a04ee91dadb242e', 'SCRIPT_NAME': '', 'keystone.token_auth': <keystonemiddleware.auth_token._user_plugin.UserAuthPlugin object at 0x7f00e68d5b90>, 'webob.adhoc_attrs': {'response': <_AuthTokenResponse at 0x7f00e68d5f50 200 OK>}, 'REQUEST_METHOD': 'PUT', 'HTTP_X_AMZ_DATE': '20181228T181903Z', 'PATH_INFO': '/bucket1/BBBunny123', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': 'partNumber=4&uploadId=YzQ0Yjk2MzMtMzdhYy00NDkyLWEwZjItZjBhYjA0M2FjYjMx', 'REMOTE_ADDR': '150.50.2.89', 'CONTENT_LENGTH': '5246566', 'HTTP_X_AMZ_DECODED_CONTENT_LENGTH': '5242880', 'HTTP_USER_AGENT': 'aws-sdk-dotnet-45/3.3.8.2 aws-sdk-dotnet-core/3.3.17.0 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_Windows_NT_6.2.9200.0 ClientAsync TransferManager/MultipartUploadCommand', 'eventlet.posthooks': [], 'RAW_PATH_INFO': '/bucket1/BBBunny123', 'REMOTE_PORT': '58993', 'eventlet.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_X_IDENTITY_STATUS': 'Invalid', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '80', 'HTTP_X_AMZ_CONTENT_SHA256': 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'wsgi.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_HOST': 'hostname.xxx.xxx.xxx', 'wsgi.multithread': True, 'HTTP_EXPECT': '100-continue', 'wsgi.version': (1, 0), 'SERVER_NAME': '150.50.2.132', 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <swift.common.utils.LoggerFileObject object at 0x7f00df2665d0>, 'wsgi.multiprocess': False, 'swift.trans_id': 'tx8f4b1c9093d1422389664-005c266917', 'CONTENT_TYPE': 'text/plain', 'swift.cache': <swift.common.memcached.MemcacheRing object at 0x7f00df2667d0>}
Thanks,
Bhaskar |
|
| 2018-12-29 10:05:48 |
Bhaskar Singhal |
description |
I am trying to upload an object (100+MB) to swift3/swift using the AWS .Net SDK multi-segment upload ( https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileDotNet.html ).
But the upload keeps failing with 'Amazon.Runtime.AmazonClientException' Expected hash not equal to calculated hash.
This is happening because the ETag calculated in obj/server is passed on the entire chunk, but in this case, the chunk has the chunk-signature etc. We need to get the decoded content and then calculate ETag.
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
Various headers set in the request:
{'HTTP_AUTHORIZATION': 'AWS4-HMAC-SHA256 Credential=4a203e8a63cc4fdfa4bbd3ea6c208bac/20181228/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=e7d32398ec602bd2fbfbbcd613ba26835d2036f6215862255a04ee91dadb242e', 'SCRIPT_NAME': '', 'keystone.token_auth': <keystonemiddleware.auth_token._user_plugin.UserAuthPlugin object at 0x7f00e68d5b90>, 'webob.adhoc_attrs': {'response': <_AuthTokenResponse at 0x7f00e68d5f50 200 OK>}, 'REQUEST_METHOD': 'PUT', 'HTTP_X_AMZ_DATE': '20181228T181903Z', 'PATH_INFO': '/bucket1/BBBunny123', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': 'partNumber=4&uploadId=YzQ0Yjk2MzMtMzdhYy00NDkyLWEwZjItZjBhYjA0M2FjYjMx', 'REMOTE_ADDR': '150.50.2.89', 'CONTENT_LENGTH': '5246566', 'HTTP_X_AMZ_DECODED_CONTENT_LENGTH': '5242880', 'HTTP_USER_AGENT': 'aws-sdk-dotnet-45/3.3.8.2 aws-sdk-dotnet-core/3.3.17.0 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_Windows_NT_6.2.9200.0 ClientAsync TransferManager/MultipartUploadCommand', 'eventlet.posthooks': [], 'RAW_PATH_INFO': '/bucket1/BBBunny123', 'REMOTE_PORT': '58993', 'eventlet.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_X_IDENTITY_STATUS': 'Invalid', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '80', 'HTTP_X_AMZ_CONTENT_SHA256': 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'wsgi.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_HOST': 'hostname.xxx.xxx.xxx', 'wsgi.multithread': True, 'HTTP_EXPECT': '100-continue', 'wsgi.version': (1, 0), 'SERVER_NAME': '150.50.2.132', 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <swift.common.utils.LoggerFileObject object at 0x7f00df2665d0>, 'wsgi.multiprocess': False, 'swift.trans_id': 'tx8f4b1c9093d1422389664-005c266917', 'CONTENT_TYPE': 'text/plain', 'swift.cache': <swift.common.memcached.MemcacheRing object at 0x7f00df2667d0>}
Thanks,
Bhaskar |
I am trying to upload an object (100+MB) to swift3/swift using the AWS .Net SDK multi-segment upload ( https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileDotNet.html ).
But the upload keeps failing with 'Amazon.Runtime.AmazonClientException' Expected hash not equal to calculated hash.
This is happening because the ETag calculated in obj/server is on the entire chunk, but in this case, the chunk has the chunk-signature etc. We need to decoded the chunk and then calculate ETag.
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
for chunk in iter(timeout_reader, ''):
start_time = time.time()
if start_time > upload_expiration:
self.logger.increment('PUT.timeouts')
return HTTPRequestTimeout(request=request)
etag.update(chunk) --> this chunk has signature, etc
upload_size = writer.write(chunk)
elapsed_time += time.time() - start_time
Various headers set in the request:
{'HTTP_AUTHORIZATION': 'AWS4-HMAC-SHA256 Credential=4a203e8a63cc4fdfa4bbd3ea6c208bac/20181228/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=e7d32398ec602bd2fbfbbcd613ba26835d2036f6215862255a04ee91dadb242e', 'SCRIPT_NAME': '', 'keystone.token_auth': <keystonemiddleware.auth_token._user_plugin.UserAuthPlugin object at 0x7f00e68d5b90>, 'webob.adhoc_attrs': {'response': <_AuthTokenResponse at 0x7f00e68d5f50 200 OK>}, 'REQUEST_METHOD': 'PUT', 'HTTP_X_AMZ_DATE': '20181228T181903Z', 'PATH_INFO': '/bucket1/BBBunny123', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': 'partNumber=4&uploadId=YzQ0Yjk2MzMtMzdhYy00NDkyLWEwZjItZjBhYjA0M2FjYjMx', 'REMOTE_ADDR': '150.50.2.89', 'CONTENT_LENGTH': '5246566', 'HTTP_X_AMZ_DECODED_CONTENT_LENGTH': '5242880', 'HTTP_USER_AGENT': 'aws-sdk-dotnet-45/3.3.8.2 aws-sdk-dotnet-core/3.3.17.0 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_Windows_NT_6.2.9200.0 ClientAsync TransferManager/MultipartUploadCommand', 'eventlet.posthooks': [], 'RAW_PATH_INFO': '/bucket1/BBBunny123', 'REMOTE_PORT': '58993', 'eventlet.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_X_IDENTITY_STATUS': 'Invalid', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '80', 'HTTP_X_AMZ_CONTENT_SHA256': 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'wsgi.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_HOST': 'hostname.xxx.xxx.xxx', 'wsgi.multithread': True, 'HTTP_EXPECT': '100-continue', 'wsgi.version': (1, 0), 'SERVER_NAME': '150.50.2.132', 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <swift.common.utils.LoggerFileObject object at 0x7f00df2665d0>, 'wsgi.multiprocess': False, 'swift.trans_id': 'tx8f4b1c9093d1422389664-005c266917', 'CONTENT_TYPE': 'text/plain', 'swift.cache': <swift.common.memcached.MemcacheRing object at 0x7f00df2667d0>}
Thanks,
Bhaskar |
|
