stack.sh fails with Swift-proxy failure and ECONNREFUSED excepction in g-api

Bug #1193112 reported by Scott DAngelo on 2013-06-20
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
High
Jamie Lennox
OpenStack Object Storage (swift)
Undecided
Michael Kerrin
devstack
Undecided
Unassigned

Bug Description

Devstack repo was cloned on 6/19/13 on a new VM.

stack.sh fails with:
2013-06-20 15:52:51 Request returned failure status.
2013-06-20 15:52:51 500 Internal Server Error
2013-06-20 15:52:51 Failed to upload image
2013-06-20 15:52:51 (HTTP 500)
2013-06-20 15:52:51 ++ failed

Swift-proxy failed with:
scottda@june19devstack:~/devstack$ cd /opt/stack/swift && /opt/stack/swift/bin/swift-proxy-server /etc/swift/proxy-server.conf -v || touch "/opt/stack/status/stack/s-proxy.failure"
proxy-server Starting keystone auth_token middleware
proxy-server Using /var/cache/swift as cache directory for signing certificate
proxy-server signing_dir mode is 0755 instead of 0700
proxy-server Starting the S3 Token Authentication component
Traceback (most recent call last):
  File "/opt/stack/swift/bin/swift-proxy-server", line 22, in <module>
    run_wsgi(conf_file, 'proxy-server', default_port=8080, **options)
  File "/opt/stack/swift/swift/common/wsgi.py", line 249, in run_wsgi
    loadapp(conf_path, global_conf={'log_name': log_name})
  File "/opt/stack/swift/swift/common/wsgi.py", line 100, in wrapper
    return f(conf_uri, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
    return loadobj(APP, uri, name=name, **kw)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
    return context.create()
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
    return self.object_type.invoke(self)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 207, in invoke
    app = filter(app)
  File "/opt/stack/keystone/keystone/middleware/s3_token.py", line 215, in auth_filter
    return S3Token(app, conf)
  File "/opt/stack/keystone/keystone/middleware/s3_token.py", line 65, in __init__
    self.http_client_class = environment.httplib.HTTPConnection
AttributeError: 'NoneType' object has no attribute 'HTTPConnection'

G-api.log:
2013-06-20 15:52:51.635 7682 ERROR glance.api.v1.upload_utils [910a1b19-21d0-4986-bb73-8af880607700 180b7a81c47e428694f8038b3da203df e89788e278a74e46b1a2169ed0ed057d] Failed to upload image
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/api/v1/upload_utils.py", line 85, in upload_data_to_store
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils image_meta['size'])
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/store/swift.py", line 329, in add
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self._create_container_if_missing(location.container, connection)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/store/swift.py", line 489, in _create_container_if_missing
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils connection.head_container(container)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/python-swiftclient/swiftclient/client.py", line 1095, in head_container
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils return self._retry(None, head_container, container)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/python-swiftclient/swiftclient/client.py", line 1048, in _retry
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils rv = func(self.url, self.token, *args, **kwargs)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/python-swiftclient/swiftclient/client.py", line 567, in head_container
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils conn.request(method, path, '', req_headers)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/python-swiftclient/swiftclient/client.py", line 193, in request_escaped
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils func(method, url, body=body, headers=headers or {})
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 962, in request
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self._send_request(method, url, body, headers)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 996, in _send_request
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self.endheaders(body)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 958, in endheaders
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self._send_output(message_body)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 818, in _send_output
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self.send(msg)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 780, in send
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self.connect()
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 761, in connect
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self.timeout, self.source_address)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/eventlet/green/socket.py", line 59, in create_connection
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils raise error, msg
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils error: [Errno 111] ECONNREFUSED
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils

Fix proposed to branch: master
Review: https://review.openstack.org/34207

Changed in swift:
assignee: nobody → Michael Kerrin (michael-kerrin-w)
status: New → In Progress
Changed in swift:
status: In Progress → Confirmed
Pete Zaitcev (zaitcev) wrote :

The regressing review:
 https://review.openstack.org/29050

Commit 3afd9791ef3e2472987cdabb54ef5f27a062469c
by Jamie Lennox
in Keystone

Jamie Lennox (jamielennox) wrote :

The change that is causing the breakage is designed to be a completely keystone local change.

The issue is that devstack is installing the s3_token middleware from keystone server, which is using the new environment module. This is generally wrong and as mentioned there are a number of ways this could be fixed, however i will put a review to revert the change in keystone and file a new bug where this can be discussed.

Changed in keystone:
assignee: nobody → Jamie Lennox (jamielennox)
status: New → Confirmed

Fix proposed to branch: master
Review: https://review.openstack.org/34484

Changed in keystone:
status: Confirmed → In Progress
Dolph Mathews (dolph) on 2013-06-26
Changed in keystone:
importance: Undecided → High

Reviewed: https://review.openstack.org/34484
Committed: http://github.com/openstack/keystone/commit/3bc661a8a79c2d86c83b8a74184bb8dc62b2aa40
Submitter: Jenkins
Branch: master

commit 3bc661a8a79c2d86c83b8a74184bb8dc62b2aa40
Author: Jamie Lennox <email address hidden>
Date: Wed Jun 26 09:45:58 2013 +1000

    Revert environment module usage in middleware.

    Devstack is pulling s3_token into the swift pipeline and so depending on
    keystone.environment is breaking devstack installs.

    Fixes bug 1193112

    Change-Id: Ifd89e542f79a2bee00113e7df676d30da0f05e59

Changed in keystone:
status: In Progress → Fix Committed
Jamie Lennox (jamielennox) wrote :

Tracking of a longer term solution has moved to: https://bugs.launchpad.net/devstack/+bug/1194688

Chmouel Boudjnah (chmouel) wrote :

FYI: Devstack should enabling s3_token only when the swift3 service is enabled which is not by default with 'swift' in ENABLED_SERVICES

Thierry Carrez (ttx) on 2013-07-17
Changed in keystone:
milestone: none → havana-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-10-17
Changed in keystone:
milestone: havana-2 → 2013.2
Dean Troyer (dtroyer) on 2013-12-12
Changed in devstack:
status: New → Won't Fix
Tim Burke (1-tim-z) wrote :

This seems to have been resolved sometime in the last few years.

Changed in swift:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers