glance fails to upload to S3 store

Bug #997658 reported by Andrew Glen-Young
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Fix Released
High
Brian Waldon

Bug Description

When I upload an image to glance it fails to upload (or even connect) to an S3 store. I am using Swift to provide S3 storage, however Glance fails to even attempt to connect and so the actual storage provider is not a factor.

The host that glance seems to attempt to connect to is: ${s3_store_bucket}.${s3_store_host} instead of ${s3_store_host}

By adding a bogus entry for this "host" to my /etc/hosts file I can at least get glance to attempt to connect.

glance-api.conf:

default_store = s3
s3_store_host = 127.0.0.1
s3_store_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
s3_store_secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
s3_store_bucket = s3bucket
s3_store_create_bucket_on_put = True

/var/log/glance/api.log

2012-05-10 13:59:36 32732 DEBUG [glance.api.middleware.version_negotiation] Processing request: POST /v1/images Accept:
2012-05-10 13:59:36 32732 DEBUG [glance.api.middleware.version_negotiation] Matched versioned URI. Version: 1.0
2012-05-10 13:59:36 32732 DEBUG [keystone.middleware.auth_token] Authenticating user token
2012-05-10 13:59:36 32732 DEBUG [keystone.middleware.auth_token] Removing headers from request environment: X-Identity-Status,X-Tenant-Id,X-Tenant-Name,X-User-Id,X-User-Name,X-Roles,X-User,X-Tenant,X-Role
2012-05-10 13:59:37 32732 DEBUG [routes.middleware] Matched POST /images
2012-05-10 13:59:37 32732 DEBUG [routes.middleware] Route path: '/images', defaults: {'action': u'create', 'controller': <glance.common.wsgi.Resource object at 0x9df6fac>}
2012-05-10 13:59:37 32732 DEBUG [routes.middleware] Match dict: {'action': u'create', 'controller': <glance.common.wsgi.Resource object at 0x9df6fac>}
2012-05-10 13:59:37 32732 DEBUG [glance.registry] Adding image metadata...
2012-05-10 13:59:37 32732 DEBUG [glance.api.v1.images] Setting image abff1de6-d5e3-4350-b19e-e68149eba6a3 to status 'saving'
2012-05-10 13:59:37 32732 DEBUG [glance.registry] Updating image metadata for image abff1de6-d5e3-4350-19e-e68149eba6a3...
2012-05-10 13:59:37 32732 DEBUG [glance.api.v1.images] Uploading image data for image abff1de6-d5e3-4350-19e-e68149eba6a3 to s3 store
2012-05-10 13:59:37 32732 DEBUG [boto] path=/
2012-05-10 13:59:37 32732 DEBUG [boto] auth_path=/s3bucket/
2012-05-10 13:59:37 32732 DEBUG [boto] path=/?&max-keys=0
2012-05-10 13:59:37 32732 DEBUG [boto] auth_path=/s3bucket/?&max-keys=0
2012-05-10 13:59:37 32732 DEBUG [boto] Method: GET
2012-05-10 13:59:37 32732 DEBUG [boto] Path: /?&max-keys=0
2012-05-10 13:59:37 32732 DEBUG [boto] Data:
2012-05-10 13:59:37 32732 DEBUG [boto] Headers: {}
2012-05-10 13:59:37 32732 DEBUG [boto] Host: s3bucket.127.0.0.1
2012-05-10 13:59:37 32732 DEBUG [boto] establishing HTTP connection: kwargs={}
2012-05-10 13:59:37 32732 DEBUG [boto] Token: None
2012-05-10 13:59:37 32732 DEBUG [boto] StringToSign:
GET

Thu, 10 May 2012 13:59:37 GMT
/s3bucket/

[...]
2012-05-10 14:00:02 32732 DEBUG [boto] encountered gaierror exception, reconnecting
2012-05-10 14:00:02 32732 DEBUG [boto] establishing HTTP connection: kwargs={}
2012-05-10 14:00:36 32732 ERROR [glance.api.v1.images] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/glance/api/v1/images.py", line 393, in _upload
    image_size)
  File "/usr/lib/python2.7/dist-packages/glance/store/s3.py", line 334, in add
    create_bucket_if_missing(self.bucket, s3_conn, self.conf)
  File "/usr/lib/python2.7/dist-packages/glance/store/s3.py", line 472, in create_bucket_if_missing
    s3_conn.get_bucket(bucket)
  File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 370, in get_bucket
    bucket.get_all_keys(headers, maxkeys=0)
  File "/usr/lib/python2.7/dist-packages/boto/s3/bucket.py", line 358, in get_all_keys
    '', headers, **params)
  File "/usr/lib/python2.7/dist-packages/boto/s3/bucket.py", line 315, in _get_all
    query_args=s)
  File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 449, in make_request
    override_num_retries=override_num_retries)
  File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 829, in make_request
    return self._mexe(http_request, sender, override_num_retries)
  File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 794, in _mexe
    raise e
gaierror: [Errno -2] Name or service not known

Server and package information:

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS"

# dpkg-query --show glance* python-boto*
glance 2012.1-0ubuntu2
glance-api 2012.1-0ubuntu2
glance-client 2012.1-0ubuntu2
glance-common 2012.1-0ubuntu2
glance-registry 2012.1-0ubuntu2
python-boto 2.2.2-0ubuntu2

Tags: canonistack
tags: added: canonistack
Revision history for this message
Brian Waldon (bcwaldon) wrote :

This definitely works with Amazon S3. Are we depending on unofficial behavior?

Changed in glance:
status: New → Incomplete
Revision history for this message
Brian Waldon (bcwaldon) wrote :

Bump!

Revision history for this message
Brian Waldon (bcwaldon) wrote :

Actually, it's boto that's pulling the bucket into the subdomain. Here's the log of info being passed to boto. 's3_host' is what is given to boto as its 'host' kwarg:

2012-09-10 20:11:20 DEBUG glance.store.s3 [290a49f4-0fae-4711-96c6-0ecf311c73a9 None None] Adding image object to S3 using (s3_host=s3.amazonaws.com, access_key=<REDACTED>, bucket=glance.72726, key=<REDACTED>) from (pid=2872) add /Users/bcwaldon/src/glance/glance/store/s3.py:362

Changed in glance:
status: Incomplete → Invalid
Revision history for this message
Brian Waldon (bcwaldon) wrote :

I'll talk with John offline

Revision history for this message
Brian Waldon (bcwaldon) wrote :

So it's (somewhat) documented that the swift3 (swift's s3 middleware) requires the ordinary calling format (non-subdomain format). I think we can provide a new s3 configuration option that gets passed in to boto for this.

Changed in glance:
status: Invalid → In Progress
importance: Undecided → High
assignee: nobody → Brian Waldon (bcwaldon)
milestone: none → folsom-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/12760

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/12760
Committed: http://github.com/openstack/glance/commit/4de8670bcbb4e977f1d550bcf470926bbd77803f
Submitter: Jenkins
Branch: master

commit 4de8670bcbb4e977f1d550bcf470926bbd77803f
Author: Brian Waldon <email address hidden>
Date: Mon Sep 10 21:21:51 2012 -0700

    Add s3_store_bucket_url_format config option

    Swift's implementation of the S3 interface doesn't support subdomain
    access to containers. It requires that the bucket be prepended to the
    request path. The option 's3_store_bucket_url_format' can be set to
    either 'path' or 'subdomain' (default) to control how boto forms
    the bucket url.

    Fixes bug 997658

    Change-Id: Ia6e1e7356eef7ac2267f7738e2f4a7c70dc12eeb

Changed in glance:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in glance:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in glance:
milestone: folsom-rc1 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.