swift proxy fails to start, get_logger gets connection refused

[Brant Knudson]

This happened during a tempest run for review [1] . Tempest fails quickly because it fails to upload the images to glance. console.html[2] shows that glance gets 400 Bad Request:

...
2013-12-19 19:01:58 ++ glance --os-auth-token ... --os-image-url http://127.0.0.1:9292 image-create --name cirros-0.3.1-x86_64-uec-kernel --is-public True --container-format aki --disk-format aki
2013-12-19 19:01:58 Request returned failure status.
2013-12-19 19:01:58 400 Bad Request
2013-12-19 19:01:58 Client disconnected before sending all data to backend
2013-12-19 19:01:58 (HTTP 400)
...

glance-api[3] shows that glance got ECONNREFUSED from swift:

...
2013-12-19 19:01:58.213 4371 ERROR swiftclient [-] [Errno 111] ECONNREFUSED
2013-12-19 19:01:58.213 4371 TRACE swiftclient Traceback (most recent call last):
2013-12-19 19:01:58.213 4371 TRACE swiftclient File "/opt/stack/new/python-swiftclient/swiftclient/client.py", line 1131, in _retry
2013-12-19 19:01:58.213 4371 TRACE swiftclient rv = func(self.url, self.token, *args, **kwargs)
2013-12-19 19:01:58.213 4371 TRACE swiftclient File "/opt/stack/new/python-swiftclient/swiftclient/client.py", line 581, in head_container
2013-12-19 19:01:58.213 4371 TRACE swiftclient conn.request(method, path, '', req_headers)
2013-12-19 19:01:58.213 4371 TRACE swiftclient File "/opt/stack/new/python-swiftclient/swiftclient/client.py", line 178, in request_escaped
2013-12-19 19:01:58.213 4371 TRACE swiftclient func(method, url, body=body, headers=headers or {})
2013-12-19 19:01:58.213 4371 TRACE swiftclient File "/usr/lib/python2.7/httplib.py", line 958, in request
2013-12-19 19:01:58.213 4371 TRACE swiftclient self._send_request(method, url, body, headers)
2013-12-19 19:01:58.213 4371 TRACE swiftclient File "/usr/lib/python2.7/httplib.py", line 992, in _send_request
2013-12-19 19:01:58.213 4371 TRACE swiftclient self.endheaders(body)
2013-12-19 19:01:58.213 4371 TRACE swiftclient File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
2013-12-19 19:01:58.213 4371 TRACE swiftclient self._send_output(message_body)
2013-12-19 19:01:58.213 4371 TRACE swiftclient File "/usr/lib/python2.7/httplib.py", line 814, in _send_output
2013-12-19 19:01:58.213 4371 TRACE swiftclient self.send(msg)
2013-12-19 19:01:58.213 4371 TRACE swiftclient File "/usr/lib/python2.7/httplib.py", line 776, in send
2013-12-19 19:01:58.213 4371 TRACE swiftclient self.connect()
2013-12-19 19:01:58.213 4371 TRACE swiftclient File "/usr/lib/python2.7/httplib.py", line 757, in connect
2013-12-19 19:01:58.213 4371 TRACE swiftclient self.timeout, self.source_address)
2013-12-19 19:01:58.213 4371 TRACE swiftclient File "/usr/local/lib/python2.7/dist-packages/eventlet/green/socket.py", line 59, in create_connection
2013-12-19 19:01:58.213 4371 TRACE swiftclient raise error, msg
2013-12-19 19:01:58.213 4371 TRACE swiftclient error: [Errno 111] ECONNREFUSED
2013-12-19 19:01:58.213 4371 TRACE swiftclient
...

swift-proxy log shows that it didn't start:

+ ln -sf /opt/stack/new/screen-logs/screen-s-proxy.2013-12-19-185502.log /opt/stack/new/screen-logs/screen-s-proxy.log
+ export PYTHONUNBUFFERED=1
+ PYTHONUNBUFFERED=1
+ exec /bin/bash -c 'cd /opt/stack/new/swift && /opt/stack/new/swift/bin/swift-proxy-server /etc/swift/proxy-server.conf -v'
Traceback (most recent call last):
  File "/opt/stack/new/swift/bin/swift-proxy-server", line 23, in <module>
    sys.exit(run_wsgi(conf_file, 'proxy-server', default_port=8080, **options))
  File "/opt/stack/new/swift/swift/common/wsgi.py", line 249, in run_wsgi
    _initrp(conf_path, app_section, *args, **kwargs)
  File "/opt/stack/new/swift/swift/common/wsgi.py", line 345, in _initrp
    log_route='wsgi')
  File "/opt/stack/new/swift/swift/common/utils.py", line 1004, in get_logger
    raise e
socket.error: [Errno 111] Connection refused

Looks like it failed to connect to a log server?

[1] https://review.openstack.org/#/c/62068/5
[2] http://logs.openstack.org/68/62068/5/check/gate-tempest-dsvm-neutron-large-ops/2946b85/console.html
[3] http://logs.openstack.org/68/62068/5/check/gate-tempest-dsvm-neutron-large-ops/2946b85/logs/screen-g-api.txt.gz
[4] http://logs.openstack.org/68/62068/5/check/gate-tempest-dsvm-neutron-large-ops/2946b85/logs/screen-s-proxy.txt.gz