Swift: errors during large file upload. Need to change 'node_timeout' parameter

[Timur Nurlygayanov]

Steps To Reproduce:
1. Deploy OpenStack with Swift data storage for Glance images.
2. Upload large image (1-2Gb) to Glance several times.

Observed Result:
Errors during image upload. If we will upload small files, all will be ok.
During the large file upload we can see 100% disk usage and in the result swift can't send reply in-time.

How To Fix:
Need to add the following parameter to /etc/swift/proxy-server.conf

[DEFAULT]
...
node_timeout = 90

--------------------------------

New Steps To Reproduce:
1. Create Glance image with 5 Gb size:
fallocate -l 5G image.img
2. Upload it to the glance:
glance image-create --name "Test1" --is-public True --disk-format raw --container-format bare --file image.img
3. Create Glance image with 70 Gb size:
fallocate -l 70G image2.img
4. Upload it to the glance:
glance image-create --name "Test2" --is-public True --disk-format raw --container-format bare --file image2.img --progress

Observed Result:
We can successfully upload image with 5Gb size but can't upload image with 70 Gb size, we can see the following errors:
...
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 153, in _chunked_read
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store raise IOError("unexpected end of file while parsing chunked data")
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store IOError: unexpected end of file while parsing chunked data
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store
2015-05-29T11:01:12.775270+00:00 info: 2015-05-29 11:01:12.773 22501 INFO urllib3.connectionpool [-] Starting new HTTP connection (2): 192.168.0.1
2015-05-29T11:01:13.305009+00:00 info: 2015-05-29 11:01:13.302 22501 INFO glance.wsgi.server [-] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 459, in handle_one_response
    write(''.join(towrite))
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 390, in write
    _writelines(towrite)
  File "/usr/lib/python2.7/socket.py", line 334, in writelines
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 357, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 340, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 32] Broken pipe

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/155487

[Ryan Moe (rmoe)]

Medium priority bugs are not backported to stable releases.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/155487
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=860671365eb5370fe03b9204ee94ba166a350a60
Submitter: Jenkins
Branch: master

commit 860671365eb5370fe03b9204ee94ba166a350a60
Author: Alexander Nevenchannyy <email address hidden>
Date: Tue Mar 3 15:16:49 2015 +0300

    Add node_timeout parameter to proxy-server.conf

    Add node_timeout parameter to proxy-server.conf configuration file.
    Set default node_timeout to 90 sec.
    This resolve problem with uploading big files.
    Change-Id: I9ef2923651554369f72c7912e99f868e18829fde
    Related-Bug: #1422533
    Closes-Bug: #1375596

[Alexander Nevenchannyy (anevenchannyy)]

Verified on MOS 6.1 ISO #429
Steps to Verify:
root@node-1:~# grep node_timeout /etc/swift/proxy-server.conf
node_timeout = 90

[Timur Nurlygayanov (tnurlygayanov)]

Verified on MOS 6.1 ISO #455

Steps To Verify:
1. Create Glance image with 5 Gb size:
fallocate -l 5G image.img
2. Upload it to the glance:
glance image-create --name "Test1" --is-public True --disk-format raw --container-format bare --file image.img
3. Create Glance image with 70 Gb size:
fallocate -l 70G image2.img
4. Upload it to the glance:
glance image-create --name "Test2" --is-public True --disk-format raw --container-format bare --file image2.img --progress

Observed Result:
We can successfully upload image with 5Gb size but can't upload image with 70 Gb size, we can see the following errors:

2015-05-29T11:01:09.987535+00:00 info: 2015-05-29 11:01:09.986 22501 INFO glance.wsgi.server [-] 192.168.0.6 - - [29/May/2015 11:01:09] "OPTIONS / HTTP/1.0" 300 697 0.000806
2015-05-29T11:01:12.744975+00:00 err: 2015-05-29 11:01:12.723 22501 ERROR glance_store._drivers.swift.store [-] Error during chunked upload to backend, deleting stale chunks
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store Traceback (most recent call last):
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store File "/usr/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py", line 513, in add
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store content_length=content_length)
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1369, in put_object
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store response_dict=response_dict)
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1243, in _retry
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store rv = func(self.url, self.token, *args, **kwargs)
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 997, in put_object
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store conn.putrequest(path, headers=headers, data=data)
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 227, in putrequest
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store return self.request('PUT', full_path, data, headers, files)
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 217, in request
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store files=files, **self.requests_args)
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 185, in _request
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store return self.request_session.request(*arg, **kwarg)
2015-05-29 11:01:12.723 22501 TRACE glance_store._drivers.swift.store File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
2015-05-29 11:01:12.723 22501 TRACE glance_s...

[Timur Nurlygayanov (tnurlygayanov)]

it is not the issue with expiration token, because it takes 15 minutes.

[Alexander Tivelkov (ativelkov)]

What are the values of swift_store_large_object_size and swift_store_large_object_chunk_size in glance-api.conf?

[Timur Nurlygayanov (tnurlygayanov)]

@ativelkov:

root@node-3:~# cat /etc/glance/glance-api.conf | grep swift_store_large_object
swift_store_large_object_size = 5120
swift_store_large_object_chunk_size = 200

[Alexander Tivelkov (ativelkov)]

Can we reduce the swift_store_large_object_size to some lower value and reverify if the 5Gb images still are created properly?

[Timur Nurlygayanov (tnurlygayanov)]

@ativelkov: it works fine with swift_store_large_object_size = 1024 and 5Gb image

[Mike Fedosin (mfedosin)]

Folks, sorry for a late response... It's a known issue in Juno, that has been fixed in Kilo. If you you use fallocate to create your file it's possible that your file may contain EOF. It leads to the fact that glance client can't upload it end exits with broken pipe error.

Solution: don't use fallocate to create big files.

For more information look at https://review.openstack.org/#/c/142952/

[Timur Nurlygayanov (tnurlygayanov)]

ok, good to know about this known issue.

[Timur Nurlygayanov (tnurlygayanov)]

So, the initial bug was successfully fixed, new staps are not valid, status of the issues changed to Fix Released.

[Alexander Bozhenko (alexbozhenko)]

@anevenchannyy: I checked fuel 6.1 and found that node_timeout=90 appears in the default section.
But description here doesn't mention node_timeout as option for DEFAULT section.
http://docs.openstack.org/kilo/config-reference/content/proxy-server-configuration.html

Also, sample configuration file here:
http://docs.openstack.org/kilo/config-reference/content/proxy-server-conf.html
doesn't show node_timeout in DEFAULT section as well.

Do you know how does it work? Does DEFAULT overrides other config sections?
According to what I understood, node_timeout shouldn't be present in DEFAULT.

[Tuan (tuanla)]

You can add in local.conf to pass the bug

SWIFT_LOOPBACK_DISK_SIZE=100G
[[post-config|${SWIFT_CONF_DIR}/swift.conf]]
[swift-constraints]
max_file_size = 21474836482

From <https://ask.openstack.org/en/question/66825/error-during-chunked-upload-cant-create-large-glance-images/>