Comment 12 for bug 1493303

Revision history for this message
Andreas Andersen (yoshiyaka) wrote : Re: Swift proxy memory leak on unfinished read

The problem is not that the machine runs out of memory, the problem is that the kernel runs out of buffer space for the TCP stack. This renders the machine unable to do any form of network communication, including SSH. It will however not get the proxy process killed, as there's still plenty of memory left on the machine (just not for TCP). This effectively renders the machine completely unresponsive until it's rebooted, or the proxy process is restarted.

dmesg gets filled up with messages like these:
[ 276.495010] TCP: out of memory -- consider tuning tcp_mem
[ 276.852703] TCP: out of memory -- consider tuning tcp_mem
[ 277.996776] TCP: out of memory -- consider tuning tcp_mem
[ 278.111035] TCP: out of memory -- consider tuning tcp_mem
[ 279.840715] TCP: out of memory -- consider tuning tcp_mem
[ 280.421439] TCP: out of memory -- consider tuning tcp_mem
[ 281.435703] TCP: out of memory -- consider tuning tcp_mem
[ 283.579847] TCP: out of memory -- consider tuning tcp_mem
[ 283.616647] TCP: out of memory -- consider tuning tcp_mem
[ 283.652761] TCP: out of memory -- consider tuning tcp_mem

These symptons are easily reproduced with the script I posted above, just run it a few times, and the machine will become completely unresponsive. The following line should do the trick:
$ vagrant ssh -c 'for i in {1..100}; do /vagrant/leakconnections.sh; done'