Python gzip lib says "MemoryError: Can't allocate memory for compression object"

[Ray C Horn]

Traceback:

  File "Z:\python projects\@lib\lib\pyax\connection.py", line 614, in query
    query_result = self._callApex(self.svc.query, query_string)

  File "Z:\python projects\@lib\lib\pyax\connection.py", line 158, in _callApex
    apex_result = apex_method(*args, **kw)

  File "Z:\python projects\@lib\lib\pyax\beatbox.py", line 56, in query
    return QueryRequest(self.context, self.sessionId, soql).post()

  File "Z:\python projects\@lib\lib\pyax\beatbox.py", line 315, in post
    data=self.makeEnvelope(),

  File "Z:\python projects\@lib\lib\pyax\beatbox.py", line 273, in makeEnvelope
    ("s", _envNs), ("p", _partnerNs), ("o", _sobjectNs))

  File "Z:\python projects\@lib\lib\pyax\beatbox.py", line 243, in __init__
    XmlWriter.__init__(self, context.gzip_request)

  File "Z:\python projects\@lib\lib\pyax\beatbox.py", line 173, in __init__
    self.__gzip = gzip.GzipFile(mode='wb', fileobj=self.__buf)

  File "C:\Python25\lib\gzip.py", line 120, in __init__
    0)

MemoryError: Can't allocate memory for compression object

[Kevin Shuk (surf)]

I'm making the assumption that you're query is resulting in a very large data set - many fields, some quite large and perhaps many records.

Run the query, only instead of returning fields, return count() to see how many rows you're expecting back.
How many fields are you requesting in your query and how big are they?
Do you have a UNIX instance on which you can try this query in pyax? This may be exacerbated by differences in Windows' memory management.

The most immediate thing you can try is to reduce the number of fields you're returning - exclude any that are not necessary (especially large text fields).

Also, if the query wants to return more than 200 rows, we can try reducing the query batch size to the minimum of 200 rows at a time. This will require releasing some code that's checked in, but not yet on the release branch.

[Ray C Horn (mrpython94596)]

Nope, rather small data set is being queried.

The problem is related to a "bug" in which gzip fails when it is used by a
large number of os threads - in my case it fails around 400 threads all
hitting pyax and then pyax hits gzip.

This is less a pyax bug than a Python bug but since pyax uses gzip this is a
pyax problem too.

I am guessing Stackless Python would not have this problem because I am
guessing gzip is hitting a C stack limit rather than a Python limit since I
have 1 GB RAM available when gzip complains about the lack of RAM.

This apparent bug does not have a fix other than to reduce the number of
threads being used however you might want to code a MemoryError trap to
notice when this condition happens and there are a bunch of threads being
used.

It took me a while to isolate the root cause however in the meantime I
reported the bug as it appeared to be related to pyax then later learned
this is a pyax second-hand bug via Python's gzip module.

--------------------------------------------------
From: "Kevin Shuk" <email address hidden>
Sent: Friday, August 08, 2008 1:31 PM
To: <email address hidden>
Subject: [Bug 256141] Re: Python gzip lib says "MemoryError: Can't
allocatememory for compression object"

> I'm making the assumption that you're query is resulting in a very large
> data set - many fields, some quite large and perhaps many records.
>
> Run the query, only instead of returning fields, return count() to see how
> many rows you're expecting back.
> How many fields are you requesting in your query and how big are they?
> Do you have a UNIX instance on which you can try this query in pyax? This
> may be exacerbated by differences in Windows' memory management.
>
> The most immediate thing you can try is to reduce the number of fields
> you're returning - exclude any that are not necessary (especially large
> text fields).
>
> Also, if the query wants to return more than 200 rows, we can try
> reducing the query batch size to the minimum of 200 rows at a time. This
> will require releasing some code that's checked in, but not yet on the
> release branch.
>
> ** Changed in: pyax
> Status: New => Triaged
>
> --
> Python gzip lib says "MemoryError: Can't allocate memory for compression
> object"
> https://bugs.launchpad.net/bugs/256141
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Salesforce.com Apex API library for Python: Triaged
>
> Bug description:
> Traceback:
>
> File "Z:\python projects\@lib\lib\pyax\connection.py", line 614, in query
> query_result = self._callApex(self.svc.query, query_string)
>
> File "Z:\python projects\@lib\lib\pyax\connection.py", line 158, in
> _callApex
> apex_result = apex_method(*args, **kw)
>
> File "Z:\python projects\@lib\lib\pyax\beatbox.py", line 56, in query
> return QueryRequest(self.context, self.sessionId, soql).post()
>
> File "Z:\python projects\@lib\lib\pyax\beatbox.py", line 315, in post
> data=self.makeEnvelope(),
>
> File "Z:\python projects\@lib\lib\pyax\beatbox.py", line 273, in
> makeEnvelope
> ("s", _envNs), ("p", _partnerNs), ("o", _sobjectNs))
>
> File "Z:\python projects\@l...