Walrus List bucket is not sorted as per S3 API guidelines

[Rajesh]

Walrus works based on Amazon S3 API. In that case, Amazon cloud returns List keys(objects) response in sorted order whereas Walrus cloud doesn't. This creates lot of problems when the cloud has more than 1000 objects.

If a cloud bucket contains more than 1000 objects, List object query will return only 1000 objects max in a single response. In order to query the rest of the keys we need to set the marker tag to the last key of the first response and query again. Since the first response itself is not in sorted order, there is no guarantee that the last key is actually in the correct position. Thus we may loose some keys. This scenario works perfect with Amazon cloud buckets. I'll try to explain this scenario with an example,

Consider following keys are present in a bucket (Just for example and not real time scenario)

1. m
2. n
3. g
4. a
5. d
6. b
7. k
8. c

and consider Amazon restricts only 5 keys maximum(not 1000) while listing. In this case, if I query for the first time, It is returning like,

1. m
2. n
3. g
4. a
5. d

which is not in sorted order, hence if I use the key 'd' as the marker and query for the next result, after that it is sorted and the result is given like,

6. g
7. m
8. n
9. k

Where 'b' and 'c' are missing. And also the previous keys are duplicated.

This issue can be reproduced very easily. I'm using eucalyptus-walrus-2.0.3. This issue happens even in CloudBerry explorer.

Hope this information helps.

Please see the Amazon S3 guidelines here: http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?RESTBucketGET.html

I have also posted this issue in Amazon S3 forum, please see the link
https://forums.aws.amazon.com/thread.jspa?threadID=80955

I tested this with Fedora but i believe this bug will also be in Ubuntu. I don't find any separate site for posting bugs in eucalyptus hence posting here.

Regards,
Rajesh A.

[Garrett Holmstrom (gholms)]

Relevant code snippet from WalrusManager.listBucket:

    if (marker != null || objectInfos.size() < maxKeys)
        Collections.sort(objectInfos);

I suspect that the "<" was meant to be ">", but Walrus should probably sort it unconditionally.

[Garrett Holmstrom (gholms)]

At last, this is fixed in mainline!

https://eucalyptus.atlassian.net/browse/EUCA-1243

Look for it in the next release.

[Andy Grimm (agrimm)]

This issue is now being tracked upstream at http://eucalyptus.atlassian.net/browse/EUCA-2786

Please watch that issue for further updates.