swiftclient is hogging all the memory

[Derek Higgins]

devtest is currently failing when uploading the user image to the overcloud

# load-image ../user.qcow2
Extracting kernel + ramdisk from /root/tripleo/user.qcow2 to tmp-vmlinuz and tmp-initrd in /tmp/image.3UHRb4S8
Error communicating with http://192.0.2.5:9292 [Errno 104] Connection reset by peer

the glance-api is being killed by the OOM

Out of memory: Kill process 24085 (glance-api) score 183 or sacrifice child
Killed process 24085 (glance-api) total-vm:627936kB, anon-rss:373752kB, file-rss:1148kB

After doing a git bisect of glance back a few weeks there doesn't seem to be any problem there, changing glance to be filesystem backed makes the problem go away.

git bisect of python-swiftclient identifies a commit that seems to be causing the extra memory usage
https://review.openstack.org/#/c/69187/

[Derek Higgins (derekh)]

The image being uploaded is 397M
-rw-r--r--. 1 root root 397M Feb 14 12:35 ../user.qcow2

[Alan Pevec (apevec)]

> git bisect of python-swiftclient identifies a commit that seems to be causing the extra memory usage
> https://review.openstack.org/#/c/69187/

Specifically:
933 + conn.putrequest(path, headers=headers, files={"file": contents})

Quotes from #openstack-swift for posterity:
" the 'files' requests parameter was a bad move as it causes the file to be read in memory before being sent"
" if you use 'files={"file": contents}' as arg for requests it iterates over all files, building the data in memory"

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/73585
Committed: https://git.openstack.org/cgit/openstack/python-swiftclient/commit/?id=380e83087447b724458ba16e11f527babb39dd01
Submitter: Jenkins
Branch: master

commit 380e83087447b724458ba16e11f527babb39dd01
Author: Tristan Cacqueray <email address hidden>
Date: Fri Feb 14 12:52:26 2014 +0100

    Remove multipart/form-data file upload

    The requests 'files' parameter adds this 'Content-Type: multipart/form-data'
    HTTP header and the whole multipart body data get stored with the object.
    This also create a memory hog issue because files are loaded in memory before
    being actually sent. This patch removes this behavior and restores what was
    done before, ie: direct uploading.

    This patches also fixes an issue in requests, when used with glance
    CooperativeReader it mis-calculates content-length leading to chunked encoding
    for raw upload.

    Change-Id: Ie5b0a1078bedd33f09c6157f48b5f88116c589fa
    Closes-Bug: #1280072
    Closes-Bug: #1280275