Support partNumber query parameter on object GET

Bug #1735284 reported by Tim Burke
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Confirmed
Medium
Unassigned
Swift3
Invalid
Undecided
Unassigned

Bug Description

There's an (undocumented) feature to request individual parts from multipart uploads -- requests look something like

> GET /<bucket>/<multipart-key>?partNumber=2 HTTP/1.1
> Host: s3.amazonaws.com
> User-Agent: curl/7.54.0
> Date: Wed, 29 Nov 2017 20:26:21 +0000
> Authorization: AWS ...
>
< HTTP/1.1 206 Partial Content
< x-amz-id-2: ...
< x-amz-request-id: ...
< Date: Wed, 29 Nov 2017 20:26:22 GMT
< Last-Modified: Sat, 03 Jun 2017 00:40:30 GMT
< ETag: "30ee66ffc6e6d27e3ebb1da8e0d2cb72-3"
< x-amz-mp-parts-count: 3
< Accept-Ranges: bytes
< Content-Range: bytes 200000000-399999999/585437638
< Content-Type: binary/octet-stream
< Content-Length: 200000000
< Server: AmazonS3
<

This even works with regular objects (note the lack of x-amz-mp-parts-count header):

> GET /<bucket>/<regular-key>?partNumber=1 HTTP/1.1
> Host: s3-us-west-2.amazonaws.com
> User-Agent: curl/7.54.0
> Date: Wed, 29 Nov 2017 20:28:28 +0000
> Authorization: AWS ...
>
< HTTP/1.1 206 Partial Content
< x-amz-id-2: ...
< x-amz-request-id: ...
< Date: Wed, 29 Nov 2017 20:28:29 GMT
< Last-Modified: Mon, 24 Aug 2015 17:30:47 GMT
< ETag: "60b725f10c9c85c70d97880dfe8191b3"
< Accept-Ranges: bytes
< Content-Range: bytes 0-1/2
< Content-Type: binary/octet-stream
< Content-Length: 2
< Server: AmazonS3
<

Asking for a part that doesn't exist (sensibly) 416s:

< HTTP/1.1 416 Requested Range Not Satisfiable
< x-amz-request-id: ...
< x-amz-id-2: ...
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Wed, 29 Nov 2017 20:28:36 GMT
< Server: AmazonS3
<
<?xml version="1.0" encoding="UTF-8"?>
* Connection #0 to host s3-us-west-2.amazonaws.com left intact
<Error><Code>InvalidPartNumber</Code><Message>The requested partnumber is not satisfiable</Message><PartNumberRequested>2</PartNumberRequested><ActualPartCount>1</ActualPartCount><RequestId>...</RequestId><HostId>...</HostId></Error>

Note that the 1<=partNumber<=1000 restriction is imposed *first*, so you could also get back a 400.

Tags: s3api slo
Revision history for this message
clayg (clay-gerrard) wrote :

So the feature seems legit/supported:

https://github.com/aws/aws-sdk-java/issues/1172

The response there suggested just using normal Range requests, but it's nice for parallel download to be able to make requests out directly to the specific segments/disks without overlap. Swift clients can do this today because the manifest and segments themselves are available using:

?multipart-manifest=get

I don't think the API can avoid reading the manifest to find the segment name, but it's an internal request so cheap enough. I think that's a reasonable approach for s3api/swift3 to support the API - and it'd be nice to do if there's some sdk's that will let applications take advantage of it easily.

Tim Burke (1-tim-z)
tags: added: s3api
tags: added: slo
Revision history for this message
Tim Burke (1-tim-z) wrote :

:-(

Apparently we not only don't support this currently, we 405 requests that try to use it, since we use the presence of the query param to push us to the PartController, which has no GET or HEAD methods. See https://github.com/openstack/swift/blob/2.20.0/swift/common/middleware/s3api/s3request.py#L971-L972

Changed in swift:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Tim Burke (1-tim-z) wrote :

Apparently I can't mark this "Won't Fix" for swift3, but any fix will appear in s3api, and won't be backported.

Changed in swift3:
status: New → Invalid
Revision history for this message
Tim Burke (1-tim-z) wrote :

Oh hey, they started documenting it!

> Part number of the object being read. This is a positive integer
> between 1 and 10,000. Effectively performs a 'ranged' GET request
> for the part specified. Useful for downloading just a part of an
> object.

Revision history for this message
Edward Konetzko (konetzed) wrote :

Any idea if there will be progress on getting this added to Swift?

Revision history for this message
Tim Burke (1-tim-z) wrote :

Yes, I'd say Swift is definitely interested in being compatible here, it's mainly a matter of havine time to implement it. There have been some other S3-related features (like versioning) that saw more interest so got implemented first, as well as other Swift-related bugs and features.

That said, if you're interested, I think it'll be a reasonably small amount of work. (Famous last words...) Given the way Swift implements MPUs, we probably want to add a similar feature to SLO then have s3api handle the translation between S3 and Swift APIs. I'd start looking around https://github.com/openstack/swift/blob/2.25.0/swift/common/middleware/slo.py#L981-L982 for where to start changing SLO; s3api might need to do some query-parameter plumbing around https://github.com/openstack/swift/blob/2.25.0/swift/common/middleware/s3api/controllers/obj.py#L91 and some response header translation around https://github.com/openstack/swift/blob/2.25.0/swift/common/middleware/s3api/s3response.py#L114-L123

If you need help getting a dev environment up or figuring out what needs to change where, you can find a bunch of us devs in IRC on freenode in #openstack-swift. It's also cool if you don't want to do it, of course -- just might take longer to implement ;-)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.