[patch] fix completely broken analyze.py

[Laurence Rowe]

The code for handling BTree and missing imports with analyze.py is completely broken from at least ZODB 3.7 onwards, only a small percentage of records seemed to be processed. This patch changes get_type to use a custom unpickler.

[Jim Fulton (jim-zope)]

This broke because there wasn't a test.

Can you supply a test so that it doesn't break again?

[Laurence Rowe (lrowe)]

svn diff with test

[Christian Theune (ctheune)]

Patch and test is there, just needs to be reviewed/integrated.

[Laurence Rowe (lrowe)]

Though I'm still confidant that this patch makes things more robust, I think the tests may look a little strange because I didn't really understand at the time that ZODB could store the class and state of an object sequentially in a pickle stream rather than as a pickled tuple of (class, state).

[Jim Fulton (jim-zope)]

On Thu, Sep 23, 2010 at 7:20 PM, Laurence Rowe <email address hidden> wrote:
> Though I'm still confidant that this patch makes things more robust, I
> think the tests may look a little strange because I didn't really
> understand at the time that ZODB could store the class and state of an
> object sequentially in a pickle stream rather than as a pickled tuple of
> (class, state).

Thanks. And thanks for the patch and test.

Unfortunately, I think this needs more work, or maybe we need a different
approach. I'm sorry to be a pain in the ass. :)

I don't know what analyze is supposed to do and I'm not sure what your test
if attempting to test. Like many of the ZODB scripts, I suspect that analyze
was a one-off script that got shat into the scripts directory.

I think we should step back a bit and ask ourselves what problems we're trying
to solve and whether scripts like this are really all that helpful.

I often want to analyze databases but the analysis varies wildly depending on
the questions I want to ask. Canned scripts like this are almost never
satisfying,
leaving aside the fact that they are rarely will documented or tested.

I suspect user's goals would be better served by a documented high-level
toolkit for writing scripts like this build around the ZODB storage
iteration APIs.

I'd prefer to set this aside for now and see if I can come up with a
better solution.

Jim

--
Jim Fulton

[Laurence Rowe (lrowe)]

It's certainly not a very informative name, though I have found it useful. (I've written up one use of it before here: http://plone.org/documentation/kb/debug-zodb-bloat). A documented toolkit for writing scripts like this would be nice, but I've found even these scripts helpful as a starting point for custom analyses. However, I suspect there is no need for them to be part of ZODB itself, they would probably be better moved to a separate zodbscripts distribution which is not tied to the ZODB release cycle. If I like I can volunteer to create such a distribution, though I can't promise to add any more tests.

The test was an attempt to mimic various types of pickles that may be found in a database. I'm sure it's incomplete, it was based more on what I observed in pdb while fixing the script.

Laurence