grizzly: swift endpoint listing support

Bug #1095496 reported by Tom Fifield
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openstack-api-site
Fix Released
Medium
Anne Gentle
openstack-manuals
Fix Released
Medium
Diane Fleming

Bug Description

If https://review.openstack.org/#/c/17791/ is merged, new functionality will be added to swift: listing endpoints

https://blueprints.launchpad.net/swift/+spec/list-endpoints

This blueprint implements the ability to list endpoints for an object, account or container, to make it possible to integrate swift with software that relies on data locality information to avoid network overhead.

In particular, it was implemented for usage with Hadoop - see https://issues.apache.org/jira/browse/HADOOP-8545

A piece of middleware is added which answers requests of the form: "/object_endpoint/{account}/{container}/{object}" with a list of endpoints of the form: "http://{server}:{port}/{device}/{partition}/{account}/{container}/{object}" and similarly for accounts and containers.

The service is intended for consumption by third-party services living inside the cluster (as the endpoints make sense only inside the cluster behind the firewall); potentially written in a different language. This is why it's provided as a REST API and not just a Python API: to avoid requiring clients to write their own ring parsers in their languages, and to avoid the necessity to distribute the ring file to clients and keep it up-to-date.

Tags: grizzly swift
Tom Fifield (fifieldt)
Changed in openstack-manuals:
milestone: none → grizzly
tags: added: swift
Revision history for this message
Tom Fifield (fifieldt) wrote :

patch abandoned

Revision history for this message
Tom Fifield (fifieldt) wrote :
Revision history for this message
Tom Fifield (fifieldt) wrote :

review in progress

Revision history for this message
Tom Fifield (fifieldt) wrote :

Review still in progress - this one is likely to make it in to Grizzly.

Tom Fifield (fifieldt)
Changed in openstack-manuals:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Stephen Gordon (sgordon) wrote :

Patch was merged on March 7. Tom is there anything left to do here?

Stephen Gordon (sgordon)
Changed in openstack-manuals:
status: Confirmed → Fix Committed
Revision history for this message
Tom Fifield (fifieldt) wrote :

Hi, yes - the code patch was merged - sorry for the ambiguous comment - but the doc patch has not yet been :)

Changed in openstack-manuals:
status: Fix Committed → Confirmed
Tom Fifield (fifieldt)
Changed in openstack-manuals:
milestone: grizzly → havana
Tom Fifield (fifieldt)
Changed in openstack-api-site:
status: New → Confirmed
importance: Undecided → Medium
milestone: none → havana
Tom Fifield (fifieldt)
Changed in openstack-manuals:
assignee: nobody → Tom Fifield (fifieldt)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-manuals (master)

Fix proposed to branch: master
Review: https://review.openstack.org/65897

Changed in openstack-manuals:
status: Confirmed → In Progress
Changed in openstack-manuals:
assignee: Tom Fifield (fifieldt) → Diane Fleming (diane-fleming)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-manuals (master)

Reviewed: https://review.openstack.org/65897
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=ab58e88c229332c2c24d6af3e4c237793207ec93
Submitter: Jenkins
Branch: master

commit ab58e88c229332c2c24d6af3e4c237793207ec93
Author: Tom Fifield <email address hidden>
Date: Fri Jan 10 16:46:10 2014 +0800

    Add List Endpoints to object storage confref

    This feature was implemented in Havana. Adding a short section
    explaining it and the single configuration option it references/

    Change-Id: I5a2a393a77741ce5964736f1608cd0d4bbd84847
    Closes-Bug: 1095496

Changed in openstack-manuals:
status: In Progress → Fix Released
Tom Fifield (fifieldt)
Changed in openstack-api-site:
milestone: havana → none
tags: added: grizzly
Anne Gentle (annegentle)
Changed in openstack-api-site:
assignee: nobody → Anne Gentle (annegentle)
Revision history for this message
Anne Gentle (annegentle) wrote :

example request and response JSON

curl -s http://192.168.22.100/endpoints/v2/AUTH_dev | python -m json.tool
{
    "endpoints": [
        "http://192.168.23.50:6002/d2/617/AUTH_dev",
        "http://192.168.23.46:6002/d8/617/AUTH_dev",
        "http://192.168.23.41:6002/d11/617/AUTH_dev"
    ],
    "headers": {}
}

curl -s http://192.168.22.100/endpoints/v2/AUTH_dev/non_ec_container | python -m json.tool
{
    "endpoints": [
        "http://192.168.23.42:6001/d14/931/AUTH_dev/non_ec_container",
        "http://192.168.23.41:6001/d10/931/AUTH_dev/non_ec_container",
        "http://192.168.23.50:6001/d0/931/AUTH_dev/non_ec_container"
    ],
    "headers": {}
}

curl -s http://192.168.22.100/endpoints/v2/AUTH_dev/non_ec_container/obj | python -m json.tool
{
    "endpoints": [
        "http://192.168.23.45:6007/d4/588/AUTH_dev/non_ec_container/obj",
        "http://192.168.23.42:6008/d13/588/AUTH_dev/non_ec_container/obj",
        "http://192.168.23.46:6007/d7/588/AUTH_dev/non_ec_container/obj"
    ],
    "headers": {
        "X-Backend-Storage-Policy-Index": "0"
    }
}

curl -s http://192.168.22.100/endpoints/v2/AUTH_dev/EC_cont1 | python -m json.tool
{
    "endpoints": [
        "http://192.168.23.50:6001/d1/322/AUTH_dev/EC_cont1",
        "http://192.168.23.46:6001/d8/322/AUTH_dev/EC_cont1",
        "http://192.168.23.42:6001/d13/322/AUTH_dev/EC_cont1"
    ],
    "headers": {}
}

curl -s http://192.168.22.100/endpoints/v2/AUTH_dev/EC_cont1/obj | python -m json.tool
{
    "endpoints": [
        "http://192.168.23.46:6008/d8/583/AUTH_dev/EC_cont1/obj",
        "http://192.168.23.50:6008/d2/583/AUTH_dev/EC_cont1/obj",
        "http://192.168.23.41:6009/d11/583/AUTH_dev/EC_cont1/obj",
        "http://192.168.23.45:6006/d3/583/AUTH_dev/EC_cont1/obj",
        "http://192.168.23.42:6009/d14/583/AUTH_dev/EC_cont1/obj",
        "http://192.168.23.45:6008/d5/583/AUTH_dev/EC_cont1/obj",
        "http://192.168.23.50:6007/d1/583/AUTH_dev/EC_cont1/obj",
        "http://192.168.23.42:6007/d12/583/AUTH_dev/EC_cont1/obj",
        "http://192.168.23.41:6007/d9/583/AUTH_dev/EC_cont1/obj",
        "http://192.168.23.46:6007/d7/583/AUTH_dev/EC_cont1/obj",
        "http://192.168.23.45:6007/d4/583/AUTH_dev/EC_cont1/obj",
        "http://192.168.23.42:6008/d13/583/AUTH_dev/EC_cont1/obj",
        "http://192.168.23.46:6006/d6/583/AUTH_dev/EC_cont1/obj",
        "http://192.168.23.41:6008/d10/583/AUTH_dev/EC_cont1/obj"
    ],
    "headers": {
        "X-Backend-Storage-Policy-Index": "2"
    }
}

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to api-site (master)

Fix proposed to branch: master
Review: https://review.openstack.org/183529

Changed in openstack-api-site:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to api-site (master)

Reviewed: https://review.openstack.org/183529
Committed: https://git.openstack.org/cgit/openstack/api-site/commit/?id=cf5c7bf909f8ee57552c9028e47215e81f5b12d4
Submitter: Jenkins
Branch: master

commit cf5c7bf909f8ee57552c9028e47215e81f5b12d4
Author: Anne Gentle <email address hidden>
Date: Fri May 15 08:51:00 2015 -0500

    Adds reference info for /endpoints resource for Object Storage

    - This is more-or-less an internal-only API but needs doc.

    Change-Id: I7ebeecfc9ffd3eee08c4d9a11ad0487346636603
    Closes-bug:1095496

Changed in openstack-api-site:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-manuals 15.0.0

This issue was fixed in the openstack/openstack-manuals 15.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.