swiftclient returns response headers without 'Content-Length' param, thus causing upload object to fail

[Arun Mani]

When trying to upload the data in swift, the _upload_object_job() method fails to upload the swift object due to the 'Content-Length' param not getting returned in the response headers. I'm not sure what causes the response headers to NOT have this param as part of the response, while trying to get the object info. Here's the stack trace:

2016-09-07 10:59:20.870 35831 DEBUG swiftclient [-] RESP STATUS: 200 OK http_log /usr/lib/python2.7/site-packages/swiftclient/client.py:165
2016-09-07 10:59:20.870 35831 DEBUG swiftclient [-] RESP HEADERS: {u'X-Object-Meta-Mtime': u'1473223600.000000', u'Accept-Ranges': u'bytes', u'Server': u'Apache', u'Last-Modified': u'Wed, 07 Sep 2016 04:46:59 GMT', u'Etag': u'd41d8cd98f00b204e9800998ecf8427e', u'X-Timestamp': u'1473223618.21456', u'X-Trans-Id': u'tx7c8fef78c8a64767876fc-0057d0396b', u'Date': u'Wed, 07 Sep 2016 15:59:39 GMT', u'Content-Type': u'application/octet-stream'} http_log /usr/lib/python2.7/site-packages/swiftclient/client.py:166
2016-09-07 10:59:20.872 35831 ERROR swiftclient.service [-] int() argument must be a string or a number, not 'NoneType'
2016-09-07 10:59:20.872 35831 ERROR swiftclient.service Traceback (most recent call last):
2016-09-07 10:59:20.872 35831 ERROR swiftclient.service File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 1800, in _upload_object_job
2016-09-07 10:59:20.872 35831 ERROR swiftclient.service cl = int(headers.get('content-length'))
2016-09-07 10:59:20.872 35831 ERROR swiftclient.service TypeError: int() argument must be a string or a number, not 'NoneType'
2016-09-07 10:59:20.872 35831 ERROR swiftclient.service
2016-09-07 10:59:20.877 35831 DEBUG nova_powervm.virt.powervm.nvram.swift [req-e37462cc-5cdb-4610-9338-182cbfbc1ac2 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 fab78243e0b44130984bc8d64cbb06a6 - - -] {u'traceback': 'Traceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 1800, in _upload_object_job\n cl = int(headers.get(\'content-length\'))\nTypeError: int() argument must be a string or a number, not \'NoneType\'\n', u'container': u'powervm_nvram', u'success': False, u'object': u'd929dfea-c061-43e3-bcd3-cf0de4c5f862_slot_map', u'headers': {u'x-object-meta-mtime': u'1473263961.000000'}, u'error': TypeError("int() argument must be a string or a number, not 'NoneType'",), u'action': u'upload_object', u'path': None, u'error_timestamp': 1473263960.872064} _run_operation /usr/lib/python2.7/site-packages/nova_powervm/virt/powervm/nvram/swift.py:86

Hence, if 'Content-Length' isn't present in the response headers the code should handle it more gracefully.

[Arun Mani (arun-mani)]

The possible reason for this failure could be due to when upload operation is attempted to upload an empty file it results in the TypeError. Found a similar bug, https://bugs.launchpad.net/python-swiftclient/+bug/1586690. Don't think it's fixed yet. Hence, proposing a new fix

[Arun Mani (arun-mani)]

review link, https://review.openstack.org/#/c/392420

[Tim Burke (1-tim-z)]

So this is Apache, reverse-proxying to Swift? And it seems to be dropping the Content-Length header that Swift normally provides. FWIW, I found some ancient reference to a problem like this [1] but surely by now that's been fixed? From their own docs [2], it looks like

> Only the [Location, Content-Location and URI headers] will be rewritten. Apache httpd will not rewrite other response headers

So... seems like either a bug or some sort of misconfiguration in Apache?

[1] http://grokbase.com/t/apache/dev/027tka3pcv/mod-proxy-drops-the-content-length-header
[2] https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxypassreverse

[OpenStack Infra (hudson-openstack)]

Change abandoned by Arun Mani (<email address hidden>) on branch: master
Review: https://review.openstack.org/392420