Uploads to a swift container failing with connection reset by peer or broken pipe

[David Moreau Simard]

Using the following provided by Ubuntu's Cloud Archive:
- Swift (1.10.0)
- python-swiftclient (1.6.0)
- python-django-horizon (2013.2-0ubuntu1~cloud0)
- Django 1.5.4

How to reproduce:
- Create a container "test"
- Try to upload a file to the test container

Horizon is able to create and delete containers.
It is also able to delete objects but you are unable to upload files without a connection reset by peer (if the endpoint is SSL) or a broken pipe error (if the endpoint is HTTP).

Using python-swiftclient on the same machine where horizon is installed will upload the file successfully regardless if the endpoint is SSL or not.

I have tried to understand why python-swiftclient is able to upload a file while horizon cannot - without success, hence this bug report.

Environment information:
- nginx in front of proxy nodes. If the endpoint is HTTPS, nginx handles the SSL termination.
- swift endpoint is https://swift.example.org:443 or http://swift.example.org:80 which corresponds to the nginx server
- keystone PKI authentication
- horizon (HTTP, no SSL) on a standalone server
- swift-proxy on a standalone server
- swift-container, swift-object and swift-account running on the same standalone server

Full stack trace of an upload to a swift container through Horizon with the swift endpoint in SSL (https termination done by nginx):
===
2014-02-25 19:17:44,266 30361 DEBUG openstack_dashboard.api.swift Swift connection created using token "69adf78ca3971562af3e7dcee5060ab5" and url "https://swift.example.org:443/v1/AUTH_c8bc4d5c4d694995882d3a43318fd6be"
2014-02-25 19:18:15,355 30361 ERROR swiftclient [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1110, in _retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 890, in put_object
    conn.send('%x\r\n%s\r\n' % (len(chunk), chunk))
  File "/usr/lib/python2.7/httplib.py", line 790, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/ssl.py", line 229, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python2.7/ssl.py", line 198, in send
    v = self._sslobj.write(data)
error: [Errno 104] Connection reset by peer
2014-02-25 19:18:15,357 30361 ERROR django.request Internal Server Error: /horizon/project/containers/test/upload
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 86, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 86, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/generic/edit.py", line 165, in post
    return self.form_valid(form)
  File "/usr/lib/python2.7/dist-packages/horizon/forms/views.py", line 94, in form_valid
    exceptions.handle(self.request)
  File "/usr/lib/python2.7/dist-packages/horizon/forms/views.py", line 91, in form_valid
    handled = form.handle(self.request, form.cleaned_data)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/containers/forms.py", line 95, in handle
    exceptions.handle(request, _("Unable to upload object."))
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/containers/forms.py", line 91, in handle
    object_file)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/swift.py", line 243, in swift_upload_object
    headers=headers)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1233, in put_object
    response_dict=response_dict)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1110, in _retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 890, in put_object
    conn.send('%x\r\n%s\r\n' % (len(chunk), chunk))
  File "/usr/lib/python2.7/httplib.py", line 790, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/ssl.py", line 229, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python2.7/ssl.py", line 198, in send
    v = self._sslobj.write(data)
error: [Errno 104] Connection reset by peer
===

Full stack trace of an upload to a swift container through Horizon with the swift endpoint is NOT SSL (http proxy by nginx):
===
2014-02-25 19:45:38,873 30362 DEBUG openstack_dashboard.api.swift Swift connection created using token "69adf78ca3971562af3e7dcee5060ab5" and url "https://swift.example.org:443/v1/AUTH_c8bc4d5c4d694995882d3a43318fd6be"
2014-02-25 19:46:09,968 30362 ERROR swiftclient [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1110, in _retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 890, in put_object
    conn.send('%x\r\n%s\r\n' % (len(chunk), chunk))
  File "/usr/lib/python2.7/httplib.py", line 790, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/ssl.py", line 229, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python2.7/ssl.py", line 198, in send
    v = self._sslobj.write(data)
error: [Errno 32] Broken pipe
2014-02-25 19:46:09,969 30362 ERROR django.request Internal Server Error: /horizon/project/containers/test/upload
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 86, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 86, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/generic/edit.py", line 165, in post
    return self.form_valid(form)
  File "/usr/lib/python2.7/dist-packages/horizon/forms/views.py", line 94, in form_valid
    exceptions.handle(self.request)
  File "/usr/lib/python2.7/dist-packages/horizon/forms/views.py", line 91, in form_valid
    handled = form.handle(self.request, form.cleaned_data)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/containers/forms.py", line 95, in handle
    exceptions.handle(request, _("Unable to upload object."))
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/containers/forms.py", line 91, in handle
    object_file)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/swift.py", line 243, in swift_upload_object
    headers=headers)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1233, in put_object
    response_dict=response_dict)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1110, in _retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 890, in put_object
    conn.send('%x\r\n%s\r\n' % (len(chunk), chunk))
  File "/usr/lib/python2.7/httplib.py", line 790, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/ssl.py", line 229, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python2.7/ssl.py", line 198, in send
    v = self._sslobj.write(data)
error: [Errno 32] Broken pipe
===

[David Moreau Simard (dmsimard)]

test suite without virtualenv (with development dependencies installed):
http://paste.openstack.org/show/69545/

test suite with virtualenv:
http://paste.openstack.org/show/69546/

Noticing the large differences in the package versions, I did a "pip install -r requirements.txt" off of the stable/havana repository, resulting in a new error - 411 length required:
http://paste.openstack.org/show/69547/

Uploads done directly by CLI through python-swiftclient still work...

[David Moreau Simard (dmsimard)]

Interestingly enough - now that I have installed the requirements from stable/havana, tests pass without any failures:
http://paste.openstack.org/show/69548/

[David Moreau Simard (dmsimard)]

I added ubuntu cloud archive as it could be related.

[David Moreau Simard (dmsimard)]

I have some progress...

Might in fact be related to nginx for some reasons yet unknown.

If the endpoint is the swift-proxy node directly, I am able to upload files through horizon successfully.
Still doesn't explain why swiftclient works regardless of endpoint configuration...

[Kieran Spear (kspear)]

What do the nginx logs say? I suspect you're hitting a request length limit (either time or size). The connections in the description both fail after 30 seconds.

[David Moreau Simard (dmsimard)]

Thanks for your input, Kieran, I unfortunately haven't had the chance to provide the logs yet, sorry.
From memory, there are no nginx errors as is (aside from the 411 length required when I was using non-standard package and module versions).
Also, please note that I am not trying to upload a large file. The test file I was using in the logged cases above was less than 2MB.

The issue could very much be nginx but I was also wondering if it could be related to Swift CORS support within Horizon.
I tried asking #openstack-horizon and #openstack-swift but could not get a definitive answer.

Should I also investigate in that direction ?

[Kieran Spear (kspear)]

The 411 error may actually be instructive. I would try applying this fix and see if it fixes things:

https://review.openstack.org/#/c/37046/2/openstack_dashboard/api/swift.py
https://bugs.launchpad.net/horizon/+bug/1200534

Otherwise, might be a network issue. I would try using swiftclient directly from the Horizon server to check that.

Alternatively, we replaced nginx with Pound on our swift cluster a while ago and it has worked well for us.

[Launchpad Janitor (janitor)]

[Expired for ubuntu-cloud-archive because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for OpenStack Dashboard (Horizon) because there has been no activity for 60 days.]