Memory doesn't free after cursor close

[Creotiv]

Python version: Python 2.6.6
MySQL Connector/Python version (and/or revision): python -c "import mysql.connector as db; print db.__version__"
MySQL version: 5.1.49-1ubuntu8.1-log
Platform: Linux 2.6.38-10-generic-pae #46-Ubuntu SMP Tue Jun 28 16:54:49 UTC 2011 i686 i686 i386 GNU/Linux

I think cursor object doesn't free memory on closing.
I use gevent + myconnpy + dbutils(pooling). I create about 100 connections to db on application init, and starting backup some user information from DB server, but after backup is finished there app doesn't free memory.
Do you free memory on cursor closing? Or only on connection close?

[Creotiv (creotiv)]

Here http://bazaar.launchpad.net/~geertjmvdk/myconnpy/main/view/head:/python2/mysql/connector/cursor.py
you add __del__ method for Cursor object, thats mean that all objects attributes you must delete manually due to GC working bad with objects that have __del__ methods ovveriden. On close method you don't delete self._results. I think this may cause bug with memory.

I will try to test this, and answer here tomorrow.

[Warwick Prince (warwickp)]

How did you go with your testing? I have encountered a memory issue myself, and this could make sense in my context as well.

[Dillon Sadofsky (dillonsadofsky)]

I would like to add to this bug, as I believe I experienced the issue and found a workaround. Firstly, as context, I have a python app that reads in MySQL blobs based on modified hashes and replicates them to another db. So, that means I have loops that work with very large result sets. I quickly noticed that python.exe could start using up gigs of space, despite that I do things in small chunks.

What I noticed is that the MySQLConnection has a list of all cursors that have ever been created through it. This container keeps cursors from every being cleaned up (due to reference counting). There are functions for removing cursors from the list, but it didn't appear to be happening automatically.

I tried several things, including wrapping the cursor in my own class that held a weakref to the real cursor. I have a custom subclass for the MySQLConnection, and it contains a function that accepts a query to be run and returns a result set. I was able to solve my memory issues by adding this line to the bottom of my execute function:
del self.cursors[:]

This cleans out the MySQLConnection cursor list. I have no idea why the connection holds onto these cursors. Because it does, their shared data never gets cleaned up (I believe MySQLConnection::close empties this list). IMO, if this list were not there, the result sets would get cleaned up automatically when the last instance of the caller using it goes out of scope. Maybe I'm wrong about that.

[Geert JM Vanderkelen (geertjmvdk)]

Something seems indeed not quite right. The following code mightn show some issues, but I'm not sure if this is the same as discussed in this bug.

import time
import mysql.connector

def bug812970():
    print('Script starts. Waiting 5 secs..')
    time.sleep(5)
    cnx = mysql.connector.connect(user='root',database='test',buffered=False)

    i = 0
    while i < 100:
        cur = cnx.cursor()
        cur.execute("SELECT REPEAT('a',5000000)")
        rows = cur.fetchall()
        cur.close()
        i += 1
    rows = None

    print('Cursor execution done. Waiting 10 secs..')
    time.sleep(10)
    cnx.close()
    print('Connection closed. Waiting 10 secs..')
    time.sleep(10)

[Geert JM Vanderkelen (geertjmvdk)]

Fix for Bug #812970 meant quite a bit of code/design changes and the results are good.

This bug was moved to http://bugs.mysql.com/bug.php?id=64394

Please add comments or subscribe using the above URL.