PCF leaks memory

[Dariusz Suchojad]

Hi,

I'm having a problem using pymqi (1.0) to write some scripts that retrieve some information from MQ servers. Not sure 100% if related to a memory leak or a mistake of mine, but when I execute the following code it seems that some memory is not released:

This is an extract of the code, to retrieve the information from channels.

...
while True:
  op=mq.cd()
  ref = qm.connectTCPClient("QM_NAME", op, "CHANNEL", "SERVER(PORT)")
  pcf = mq.PCFExecute(qm)

  # Next one is the line wich gets memory: if commented, memory doesn't grow
   channels = pcf.MQCMD_INQUIRE_CHANNEL_STATUS({CMQCFC.MQCACH_CHANNEL_NAME : "*"})

   # Process information, name, status, ...

  del channels
  del pcf
  del ref
  del op
....

Of course this is not exactly the code I'm using on the scripts, probably I don't even need the deletes, but my code also involves an infinite loop for monitoring some stuff (queues depth and more). When executing the previous code I get the python.exe process memory growing until reaches aprox 2GB and then crashes.

Am I missing something ?

Other information is: reproduced with both pymqi 0.5d and 1.0, in a Windows 2003 system as MQ client. Targets systems are MQ 6 and 7 in different platforms (Win, Linux, iSeries). pymqi compiled with MQ client version 6.0.27 and MinGW

Any ideas ? Thank you for your time.

regards,
Joan

[Dariusz Suchojad (dsuch)]

Hi Joan,

as you can see, I have converted the https://answers.launchpad.net/pymqi/+question/97293 question into a bug report. I've run the code under a debug build of Python 2.6.2 and sys.gettotalrefcount() was showing the reference count increasing after each invocation of PCF so it seems like a bug.

I have also created a bzr branch for working on the issue, you can get it by issuing the following command:

$ bzr branch lp:~dsuch/pymqi/lp-507617

Although right now it's only a clean copy of trunk, that's where I'll be pushing any code changes to.

I'm not the PCF/MQAI bits author and it's actually the part I'm the least familiar with, however I'll try to resolve it as soon as I can. Will post new comments here as I'll be progressing.

Cheers!

[Dariusz Suchojad (dsuch)]

Hello,

I think I have chased down all the missing calls to Py_XDECREF and all should work flawlessly now. The memory usage stays constant and on my system it's below 6MB, which I believe is quite okay :-)

Can you, please, check out the code I have just pushed to the branch and let me know how it works for you?

Thanks!

[Joan (jblanch)]

good news !!! I'll test it asap, I'll be travelling till Wednesday, but try to do connect remotelly and send you feedback.

Thanks again!

[Joan (jblanch)]

Dariusz, YOU ARE THE MAN !

That version solves the memory problems. I've just set all the scripts running again, and I'll keep checking their behaviour.

When running the infinite loop with the PCF call for a short period with same iterations, the results where:

PyMQI 1.0: 25MB (and growing)
with FIX: 7 MB (and stable)

so it's ok,

Do you plan to update the official release of pymqi ? I mean, do you recommend me to use the version I compiled from bzr branch, or I'll better wait till you pack it, before I install it in several systems?

thank you very much,

Joan

[Dariusz Suchojad (dsuch)]

You're welcome, glad we got it working now! :-)

> Do you plan to update the official release of pymqi ?

Yes, I believe it warrants a 1.0.1 bugfix release. I hope I can get it ready in the next couple of days. In the meantime, you can safely use the branch code, the 1.0.1 code will only differ by the version tag (if you check pymqi.__version__, you'll see it's been set to "1.0-lp507617" whereas the released code will have a "1.0.1" tag). I won't introduce any other changes so it's actually your call, can you wait a couple of days more?

[Dariusz Suchojad (dsuch)]

Hi,

I have just release PyMQI 1.0.1, you can get it from here https://launchpad.net/pymqi/+download. It contains the fix for the bug you had encountered.

Thanks for your help!