For the Object Storage API, further describe subdir marker and delimiter

[Anne Gentle]

Reported via doc comment:

When a request uses `format` and `path` together, the server returns a `subdir` object for each directory-like entry.

Consider this setup:

a/foo.txt
a/b/bar.txt

Now a request is issued with these params: ?prefix=a/&format;=json

The resulting JSON data will look like this:

[
  {"name": "a/foo.txt", ...}
  {"subdir": "a/b/"}
]

The subdir can then be used to construct a new prefix to simulate a directory listing of `a/b/`.
The `delimiter` param does not seem to have any effect when combined with `path`.

Say I have a list of objects like this (no directory marker files):

a/b/c/foo.txt
a/bar.txt

To be really directory-like, I should be able to view the "contents" of "a/b". Setting path=a/b&delimiter;=/ does not seem to do this (unless I use directory markers).

[Timur Alperovich (timur-alperovich)]

You probably should use the prefix parameter, rather than path. Setting prefix and delimiter resolves this for me: prefix "a/b/" and delimiter "/". This will return all objects that start with a/b/ and truncate to "/".

The documentation does mention that setting path is equivalent to setting prefix to "path" + "/" and delimiter to "/" (http://developer.openstack.org/api-ref-objectstorage-v1.html#showContainerDetails).

Note, however, that you may run into a relate issue: https://bugs.launchpad.net/swift/+bug/1475018
Essentially, if you create keys that start with "/" and want to list the contents under "/", it will not honor the delimiter setting and will list all keys in the container.

[Anne Gentle (annegentle)]

From the swift bug referenced above:
Setting just the delimiter (without a prefix) results in behavior that is different from setting both parameters.

Example to illustrate the problem:
1. Create the following objects in a container: /dir, /dir/subdir/foo, dir, dir/subdir/foo
2. Issue a query that sets the delimiter to "/" and prefix to "/"
    Expected result: subdir: "/dir/", blobs: "/dir"
    Actual result: subdir: "/dir/", blobs: "/dir" -- this is correct
3. Issue a query that sets the prefix to "" and delimiter to "/"
    Expected result: subdir: ["/", "dir/"]
    Actual result: subdir: ["dir/"], blobs: ["/dir", "/dir/subdir/foo", "dir"]

This is really puzzling, as for keys that start with the delimiter, _all_ keys are listed, even if they include multiple occurrences of the delimiter. This behavior seems wrong to me.

This is also interesting given the documentation for the path option: "Equivalent to setting delimiter to / and prefix to the path with a / at the end." However, this is not correct. Setting path to "" should then be the same as setting delimiter to "/" and prefix to "/", however, the actual results from setting path to "" are: "dir".
---

To add a note about this behavior, add it to here:
https://github.com/openstack/api-site/blob/5867cd2337f7591de41a2abb09264aebc4c0fac5/api-ref/src/wadls/object-api/src/common.ent#L634

"If you do not set a prefix and set the delimiter to "/" you may get unexpected results where all the objects are returned instead of only those with the delimiter set."

[Anne Gentle (annegentle)]

Suzanna, I'm un-assigning you but feel free to pick this up at another time!

[Suzanna Schulze (suzannaschulze)]

Hi Anne,

So sorry I have not fixed the bug yet...even though it's been on my list of things to do.

Thanks very much for your patience! I still would like to work in OpenStack in the future.

When you say '...feel free to pick this up at another time', do you mean this particular bug (I see it's now assigned to Laura Clymer), or would I just get a different bug to work?

Suzanna

> Date: Wed, 17 Feb 2016 16:20:13 +0000
> From: <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1268739] Re: For the Object Storage API, further describe subdir marker and delimiter
>
> Suzanna, I'm un-assigning you but feel free to pick this up at another
> time!
>
> ** Changed in: openstack-api-site
> Assignee: Suzanna Schulze (suzannaschulze) => (unassigned)
>
> ** Changed in: openstack-api-site
> Assignee: (unassigned) => Laura Clymer (laura-clymer)
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1268739
>
> Title:
> For the Object Storage API, further describe subdir marker and
> delimiter
>
> Status in openstack-api-site:
> Triaged
>
> Bug description:
> Reported via doc comment:
>
> When a request uses `format` and `path` together, the server returns a
> `subdir` object for each directory-like entry.
>
> Consider this setup:
>
> a/foo.txt
> a/b/bar.txt
>
> Now a request is issued with these params: ?prefix=a/&format;=json
>
> The resulting JSON data will look like this:
>
> [
> {"name": "a/foo.txt", ...}
> {"subdir": "a/b/"}
> ]
>
> The subdir can then be used to construct a new prefix to simulate a directory listing of `a/b/`.
> The `delimiter` param does not seem to have any effect when combined with `path`.
>
> Say I have a list of objects like this (no directory marker files):
>
> a/b/c/foo.txt
> a/bar.txt
>
> To be really directory-like, I should be able to view the "contents"
> of "a/b". Setting path=a/b&delimiter;=/ does not seem to do this
> (unless I use directory markers).
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openstack-api-site/+bug/1268739/+subscriptions

[Anne Gentle (annegentle)]

Hi Suzanna!

No worries on not fixing this particular bug, the great thing about open source is there's always another one to work on. :) What I meant is feel free to pick up another bug another time, sorry for not being more clear. Do an Advanced Search for another Triaged bug and see what you find, okay?

Oh, and this is a good learning lesson, since we didn't put "Closes-bug: 1268739" in the commit message in the beginning, the patchset isn't showing up here in the comments. It's https://review.openstack.org/#/c/281470/ and Laura, you can add the text that will automate updating this bug. I'll walk you through that tomorrow if that works.

Thanks to both of you!
Anne

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/281470
Committed: https://git.openstack.org/cgit/openstack/api-site/commit/?id=12c31ade79768c69e7f74b58cdd5b6efa5ad957f
Submitter: Jenkins
Branch: master

commit 12c31ade79768c69e7f74b58cdd5b6efa5ad957f
Author: Laura Clymer <email address hidden>
Date: Wed Feb 17 13:13:49 2016 -0600

    Adds additional text about delimiters for object storage api

    Closes-bug: 1268739
    Change-Id: If82d386cca0dfcee2721b37c64ed1f7806af7126