Copy middleware leak socket when copying object bigger than max_file_size

Bug #1698207 reported by Romain LE DISEZ on 2017-06-15
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Undecided
Unassigned

Bug Description

How to reproduce:
1. Upload an object bigger than max_file_size by segmenting it
2. Try to copy it

You'll get a "413 Request Entity Too Large" response. The connections the proxy established to the object servers won't ever get closed.

Changed in swift:
status: New → Confirmed
Alistair Coles (alistair-coles) wrote :
Download full text (4.1 KiB)

On my SAIO I set max_file_size = 52428800 in swift.conf

$ swift-init main restart
$ swift upload c1 100MiB_file --segment-size 10485760
$ swift copy c1 100MiB_file
Object COPY failed: http://localhost:8080/v1/AUTH_test/c1/100MiB_file 413 Request Entity Too Large [first 60 chars of response] <html><h1>Request Entity Too Large</h1><p>The body of your r

...while watching the connections with

$ watch -n 0.5 'sudo netstat -ape --numeric-ports --tcp|grep python|grep swift'

and on master see object server connections pile up

```

Every 0.5s: sudo netstat -ape --numeric-ports --tcp|grep python|grep swift Fri Jun 16 11:04:38 2017

tcp 0 0 127.0.0.1:6040 0.0.0.0:* LISTEN swift 1169921 21598/python
tcp 0 0 127.0.0.1:6041 0.0.0.0:* LISTEN swift 1169385 21590/python
tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN swift 1169914 21595/python
tcp 0 0 127.0.0.1:6042 0.0.0.0:* LISTEN swift 1169364 21594/python
tcp 0 0 127.0.0.1:6011 0.0.0.0:* LISTEN swift 1169487 21587/python
tcp 0 0 127.0.0.1:6012 0.0.0.0:* LISTEN swift 1169374 21591/python
tcp 0 0 127.0.0.1:6020 0.0.0.0:* LISTEN swift 1169836 21596/python
tcp 0 0 127.0.0.1:6021 0.0.0.0:* LISTEN swift 1169677 21588/python
tcp 0 0 127.0.0.1:6022 0.0.0.0:* LISTEN swift 1169390 21592/python
tcp 0 0 127.0.0.1:6030 0.0.0.0:* LISTEN swift 1169906 21597/python
tcp 0 0 127.0.0.1:6031 0.0.0.0:* LISTEN swift 1169486 21589/python
tcp 0 0 127.0.0.1:6032 0.0.0.0:* LISTEN swift 1169663 21593/python
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN swift 1169246 21586/python
tcp 0 3770878 127.0.0.1:6010 127.0.0.1:53942 ESTABLISHED swift 1196190 21814/python
tcp 0 3568541 127.0.0.1:6030 127.0.0.1:36786 ESTABLISHED swift 1196594 21805/python
tcp 0 0 127.0.0.1:57404 127.0.0.1:11211 ESTABLISHED swift 1170263 21713/python
tcp 940182 0 127.0.0.1:36600 127.0.0.1:6030 ESTABLISHED swift 1170774 21713/python
tcp 941228 0 127.0.0.1:38846 127.0.0.1:6040 ESTABLISHED swift 1194938 21713/python
tcp 0 4037162 127.0.0.1:6030 127.0.0.1:36600 ESTABLISHED swift 1170775 21805/python
tcp 0 4103157 127.0.0.1:6010 127.0.0.1:53906 ESTABLISHED swift 1195324 21814/python
tcp 939670 0 127.0.0.1:53906 127.0.0.1:6010 ESTABLISHED swift 1195323 21713/python
tcp 0 3460231 127.0.0.1:6010 127.0.0.1:53926 ESTABLISHED sw...

Read more...

Alistair Coles (alistair-coles) wrote :

Also provoked same symptons by restricting conatiner listing limit in swift.conf (same object segments as comment #1)

#max_file_size = 52428800
container_listing_limit = 5

Reviewed: https://review.openstack.org/474767
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=a7c8becd4ed41dbf1c88d36b8601d965d6e7cb75
Submitter: Jenkins
Branch: master

commit a7c8becd4ed41dbf1c88d36b8601d965d6e7cb75
Author: Romain LE DISEZ <email address hidden>
Date: Thu Jun 15 22:09:45 2017 +0200

    Fix a socket leak in copy middleware

    When the "copy" middleware tries to copy a segmented object which is
    bigger than max_file_size, it immediatly returns "413 Request Entity Too
    Large". But at that point, connections have already been established by
    the proxy server to the object servers. These connections must be closed
    before returning.

    Closes-Bug: #1698207
    Change-Id: I430c48c4a81e8392fa271160bcbc1817ef0a88f7

Changed in swift:
status: Confirmed → Fix Released

This issue was fixed in the openstack/swift 2.15.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers