ZODB load state error

[Bug Importer]

I just found this in my Zope log file. After this error the Zope system stopped. No wonder, but certainly not because of physical memory error. What could be the problem?

2004-01-25T02:55:21 ERROR(200) ZODB Couldn't load state for 0000000000003ef2
Traceback (most recent call last):
  File "/opt/freeware/zope/lib/python/ZODB/Connection.py", line 564, in setstate
    self._set_ghost_state(obj, p)
  File "/opt/freeware/zope/lib/python/ZODB/Connection.py", line 603, in _set_ghost_state
    state = unpickler.load()
MemoryError

[Andreas Jung (ajung)]

Status: Pending => Rejected

Please use the Zope or Zodb-Dev mailing list for asking questions.

-aj

[Tim Peters (tim-one)]

Status: Rejected => Accepted

 Supporters added: jeremy, tim_one

Accepting this, and assigned to Jeremy and me.

I've heard of this happening before, and I want a Collector issue open on it so we can collect clues in one place.

Thyb, is this reproducible?

Chances are "something is corrupted", but without a reproducible guess it's hard to guess what or how or why.

[thyb (thyb)]

Hi,

I can't tell if it's reproducible or not. This was the first time I ran into this problem. My Zope system was running for a week then crashed. So I guess I have to wait another week for the same error to occure if reproduceable.

How do I get more information on what has happened? How should I configure logging? Debugging?

Otherwise I get the same error message like the one in 'Entry #1', which you said doesn't help much.

Thx for the assistance

[Tim Peters (tim-one)]

This is such a low-level "impossible" error that I doubt any logging is going to help. There are no other reports of it, either -- it *could* be due to things like a flaky memory chip, a bad network device, a bad spot on a disk, or a stray gamma ray flipping a bit.

I may (or may not) have seen a similar thing when running test suites intensively overnight, but about 6 months ago: there was one instance of a mysterious MemoryError in the same place. I've never seen that again.

But because I did see something similar once, I want to keep the Collector report open to gather clues in one place. If it's actually a bug in ZODB code, it's extremely shy, and diagnosing and fixing something like that will most likely require finding someone for whom it happens often, and who can reduce it to a self-contained stress test. I've tried (and failed) to provoke it on purpose.

[thyb (thyb)]

I vote for gamma ray =] that bastard...

[thyb (thyb)]

ZODB load state error has occured again. But something new has happened before the error. What could this be?

2004-01-31T11:59:53 ERROR(200) ZServer uncaptured python exception, closing chan
nel <ZServer.HTTPServer.zhttp_channel connected 193.225.218.194:61340 at 0x252cb
9cc channel#: 17802 requests:> (socket.error:(32, 'Broken pipe') [/usr/local/lib
/python2.3/asynchat.py|initiate_send|218] [/opt/freeware/zope/lib/python/ZServer
/medusa/http_server.py|send|417] [/usr/local/lib/python2.3/asyncore.py|send|334]
)

Furthermore, the number of httpd server processes was 70. Which is too high for a rarely used web server. What happens here? Too many connections? The apache httpd and zope make loops?

Up till now I ran zope with 'runzope' script. Now I'll try zopectl.

Fastcgi is not configured. That shouldn't be a problem. Right?

In apache I use RewriteRule.

RewriteEngine On
RewriteRule ^/(.*) http://hubertus.kgk.bmf.hu:8080/KGK/VirtualHostBase/http/www.
kgk.bmf.hu:80/VirtualHostRoot/$1 [L,P]

[Vahur Rebas (berlingo666)]

I get these errors once a week.
Python 2.3.3, Zope 2.7.0 on FreeBSD
With Zope 2.6.1 I only once got MemoryError

about 20 of these
------
2004-03-11T15:44:26 ERROR(200) ZODB Couldn't load state for 000000000017ad16
Traceback (most recent call last):
  File "/usr/local/www/zope/Zope.B/lib/python/ZODB/Connection.py", line 564, in setstate
    self._set_ghost_state(obj, p)
  File "/usr/local/www/zope/Zope.B/lib/python/ZODB/Connection.py", line 603, in _set_ghost_state
    state = unpickler.load()
MemoryError
------

following about thousand of these errors in my
event.log

------
2004-03-11T15:46:20 ERROR(200) ZODB Couldn't load state for 000000000017ad16
Traceback (most recent call last):
  File "/usr/local/www/zope/Zope.B/lib/python/ZODB/Connection.py", line 561, in setstate
    p, serial = self._storage.load(oid, self._version)
  File "/usr/local/www/zope/Zope.B/lib/python/ZODB/FileStorage.py", line 687, in load
    return self._load(oid, version, self._index, self._file)
  File "/usr/local/www/zope/Zope.B/lib/python/ZODB/FileStorage.py", line 678, in _load
    return read(u64(plen)), serial
MemoryError
------

and zope restarts itself with last message

------
2004-03-11T18:41:11 ERROR(200) ZServer uncaptured python exception, closing channel <select-trigger (pipe) at
 852be0c> (exceptions.MemoryError: [/usr/local/lib/python2.3/asyncore.py|read|69] [/usr/local/lib/python2.3/a
syncore.py|handle_read_event|390] [/usr/local/www/zope/Zope.B/lib/python/ZServer/medusa/thread/select_trigger
.py|handle_read|77] [/usr/local/lib/python2.3/asyncore.py|recv|342] [/usr/local/lib/python2.3/asyncore.py|rec
v|523])
------

[Tim Peters (tim-one)]

berlingo666: You said you're running on FreeBSD. That OS is notorious for giving threads tiny stacks, and it wouldn't be surprising if Zope 2.7 needs more stack space than older Zopes. Most OSes give threads plenty of stack space, so people on other platforms wouldn't notice this. The usual symptom of running out of stack space is seemingly random segfaults, but seemingly random MemoryErrors could also result.

I've never run on FreeBSD myself, so can't tell you how to test this hypothesis. Searching the Zope mailing list archives might.

[Vahur Rebas (berlingo666)]

I recompiled my python with 2MB of stack size but problems are still here. I think there is something wrong with something else. Zope used about 600M of memory, there was still over 100M of free memory and these MemoryErrors started to appear. All errors, people have posted here appeared.

[Vasudeva Server (vasudevaserver)]

We are witnessing exactly the same problem.

We are running Zope 2.7.3, running on Python 2.3.4 compiled with large_stack, on a FreeBSD 4.10.

The error is:
MemoryError: (Also, an error occurred while attempting to render the standard error message.)

It started happening on a Plone page, when we moved to Zope 2.7.3.

It happens right now about once a day.

[Tim Peters (tim-one)]

Sorry, I don't know anything about FreeBSD. How large a thread stack does "large_stack" give? FYI, most OS/crt combos give a thread at least a MB of stack space.

[ken manheimer (ken-manheimer)]

Changes: submitter name, submitter email, edited transcript, new comment

removing original submitter from the issue, at his request.

[Juan Jose Amor Iglesias (jjamor)]

We got these errors using Plone 3.1 and Zope 2.10.6-final. The Plone finally crashed:

2009-05-13T13:32:19 ERROR ZODB.Connection Couldn't load state for 0x03db0a
Traceback (most recent call last):
  File "/usr/local/Plone-3.1/zinstance/parts/zope2/lib/python/ZODB/Connection.py", line 761, in setstate
    self._setstate(obj)
  File "/usr/local/Plone-3.1/zinstance/parts/zope2/lib/python/ZODB/Connection.py", line 819, in _setstate
    self._reader.setGhostState(obj, p)
  File "/usr/local/Plone-3.1/zinstance/parts/zope2/lib/python/ZODB/serialize.py", line 604, in setGhostState
    state = self.getState(pickle)
  File "/usr/local/Plone-3.1/zinstance/parts/zope2/lib/python/ZODB/serialize.py", line 597, in getState
    return unpickler.load()
  File "/usr/local/Plone-3.1/zinstance/parts/zope2/lib/python/ZODB/serialize.py", line 471, in _persistent_load
    return self.load_oid(reference)
  File "/usr/local/Plone-3.1/zinstance/parts/zope2/lib/python/ZODB/serialize.py", line 537, in load_oid
    return self._conn.get(oid)
  File "/usr/local/Plone-3.1/zinstance/parts/zope2/lib/python/ZODB/Connection.py", line 213, in get
    p, serial = self._storage.load(oid, self._version)
  File "/usr/local/Plone-3.1/zinstance/parts/zope2/lib/python/ZODB/FileStorage/FileStorage.py", line 554, in load
    data = self._file.read(h.plen)
MemoryError
------

also a lots of errors like this:

2009-05-13T13:45:11 ERROR Zope.SiteErrorLog http://nnn/nnn/nnn/sendto_form
Traceback (innermost last):
  Module ZPublisher.Publish, line 122, in publish
  Module ZServer.HTTPResponse, line 262, in setBody
  Module ZPublisher.HTTPResponse, line 324, in setBody
  Module ZPublisher.HTTPResponse, line 476, in _encode_unicode
MemoryError

Is this bug still unsolved? We have this Plone in a production site since more than 1 year and this is the first time it crashes with these errors...

[Hanno Schlichting (hannosch)]

The last comment reopening this ticket isn't helpful. If you have too little memory, then you get MemoryErrors. This is known to happen before you totally exhaust the last bits.

[mallow (princemallow)]

I've started to have this issue as well.

I get a MemoryError after a few minutes of uptime.

Watching 'Top' however the process doesn't seem to show the process missing any memory. It stays around 4-16 percent CPU and 2-10.3 percent memory. I don't think it should be throwing a MemoryError here.