version 1.43 sometimes doesn't close connections

[Malcolm Tredinnick]

Recently upgrade from 1.34 (standard with Debian Etch) to 1.43 on a reasonably heavily-loaded production machine and started seeing the problem reported here: http://code.djangoproject.com/ticket/5133

Basically, the number of open connections went up drastically mostly because they weren't being closed. Not really sure what is going on there. The threadlocal change that was introduced in 1.36 does seem to play a role, since backing that out "fixes" things for this problem, but it obviously doesn't help the whatever problem caused the introduction of threadlocals in the first place.

For now, we've applied a patch on the Django side of things to force the close after each HTTP request, but I wanted to make sure that this was known upstream and I couldn't see any reports of it outside of Django circles. Django's caching code really is doing nothing more than opening a connection to the server and passing directly through to memcache's get() and set() methods.

[Sean Reifschneider (jafo)]

This is the correct action, if you do not wish to leave the connection to the memcached server open you need to close it when you are done. Conversely, if you wish to leave the connection to the memcache servers open after a request, so that a future connection can re-use them, then do not close it when you are done.

Automatically closing and re-opening the connections isn't something the memcache module can decide on it's own, the applications that make use of it need to use their knowledge to optimize this.

For Django, probably the ideal situation would be to use a pool of open memcached connections to gain the benefits of not having to open new connections, while not requiring $MEMCACHED_SERVER_COUNT*$WEB_SERVER_COUNT number of open connections.

Sean