stack.sh fails with Swift-proxy failure and ECONNREFUSED excepction in g-api

[Scott DAngelo]

Devstack repo was cloned on 6/19/13 on a new VM.

stack.sh fails with:
2013-06-20 15:52:51 Request returned failure status.
2013-06-20 15:52:51 500 Internal Server Error
2013-06-20 15:52:51 Failed to upload image
2013-06-20 15:52:51 (HTTP 500)
2013-06-20 15:52:51 ++ failed

Swift-proxy failed with:
scottda@june19devstack:~/devstack$ cd /opt/stack/swift && /opt/stack/swift/bin/swift-proxy-server /etc/swift/proxy-server.conf -v || touch "/opt/stack/status/stack/s-proxy.failure"
proxy-server Starting keystone auth_token middleware
proxy-server Using /var/cache/swift as cache directory for signing certificate
proxy-server signing_dir mode is 0755 instead of 0700
proxy-server Starting the S3 Token Authentication component
Traceback (most recent call last):
  File "/opt/stack/swift/bin/swift-proxy-server", line 22, in <module>
    run_wsgi(conf_file, 'proxy-server', default_port=8080, **options)
  File "/opt/stack/swift/swift/common/wsgi.py", line 249, in run_wsgi
    loadapp(conf_path, global_conf={'log_name': log_name})
  File "/opt/stack/swift/swift/common/wsgi.py", line 100, in wrapper
    return f(conf_uri, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
    return loadobj(APP, uri, name=name, **kw)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
    return context.create()
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
    return self.object_type.invoke(self)
  File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 207, in invoke
    app = filter(app)
  File "/opt/stack/keystone/keystone/middleware/s3_token.py", line 215, in auth_filter
    return S3Token(app, conf)
  File "/opt/stack/keystone/keystone/middleware/s3_token.py", line 65, in __init__
    self.http_client_class = environment.httplib.HTTPConnection
AttributeError: 'NoneType' object has no attribute 'HTTPConnection'

G-api.log:
2013-06-20 15:52:51.635 7682 ERROR glance.api.v1.upload_utils [910a1b19-21d0-4986-bb73-8af880607700 180b7a81c47e428694f8038b3da203df e89788e278a74e46b1a2169ed0ed057d] Failed to upload image
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/api/v1/upload_utils.py", line 85, in upload_data_to_store
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils image_meta['size'])
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/store/swift.py", line 329, in add
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self._create_container_if_missing(location.container, connection)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/store/swift.py", line 489, in _create_container_if_missing
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils connection.head_container(container)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/python-swiftclient/swiftclient/client.py", line 1095, in head_container
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils return self._retry(None, head_container, container)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/python-swiftclient/swiftclient/client.py", line 1048, in _retry
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils rv = func(self.url, self.token, *args, **kwargs)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/python-swiftclient/swiftclient/client.py", line 567, in head_container
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils conn.request(method, path, '', req_headers)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/opt/stack/python-swiftclient/swiftclient/client.py", line 193, in request_escaped
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils func(method, url, body=body, headers=headers or {})
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 962, in request
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self._send_request(method, url, body, headers)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 996, in _send_request
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self.endheaders(body)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 958, in endheaders
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self._send_output(message_body)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 818, in _send_output
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self.send(msg)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 780, in send
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self.connect()
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 761, in connect
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils self.timeout, self.source_address)
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/eventlet/green/socket.py", line 59, in create_connection
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils raise error, msg
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils error: [Errno 111] ECONNREFUSED
2013-06-20 15:52:51.635 7682 TRACE glance.api.v1.upload_utils

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/34207

[Pete Zaitcev (zaitcev)]

The regressing review:
 https://review.openstack.org/29050

Commit 3afd9791ef3e2472987cdabb54ef5f27a062469c
by Jamie Lennox
in Keystone

[Jamie Lennox (jamielennox)]

The change that is causing the breakage is designed to be a completely keystone local change.

The issue is that devstack is installing the s3_token middleware from keystone server, which is using the new environment module. This is generally wrong and as mentioned there are a number of ways this could be fixed, however i will put a review to revert the change in keystone and file a new bug where this can be discussed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/34484

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/34484
Committed: http://github.com/openstack/keystone/commit/3bc661a8a79c2d86c83b8a74184bb8dc62b2aa40
Submitter: Jenkins
Branch: master

commit 3bc661a8a79c2d86c83b8a74184bb8dc62b2aa40
Author: Jamie Lennox <email address hidden>
Date: Wed Jun 26 09:45:58 2013 +1000

    Revert environment module usage in middleware.

    Devstack is pulling s3_token into the swift pipeline and so depending on
    keystone.environment is breaking devstack installs.

    Fixes bug 1193112

    Change-Id: Ifd89e542f79a2bee00113e7df676d30da0f05e59

[Jamie Lennox (jamielennox)]

Tracking of a longer term solution has moved to: https://bugs.launchpad.net/devstack/+bug/1194688

[Chmouel Boudjnah (chmouel)]

FYI: Devstack should enabling s3_token only when the swift3 service is enabled which is not by default with 'swift' in ENABLED_SERVICES

[Tim Burke (1-tim-z)]

This seems to have been resolved sometime in the last few years.