Convert blog listing to get_catalog_batch

[Paul Everitt]

Make sure the tests get refactored correctly.

[Carlos de la Guardia (cguardia)]

Added a proposed fix on branch get_catalog_batch.

[Paul Everitt (paul-agendaless)]

Shane has a neat way to look at object loads. Add a ?__debugload__ to the end of the URL and you'll get, in your karl.log file, a dump of how many of each kind of object was loaded during the request.

For a small blog, the container version loaded 165 objects:

165 ZODB objects loaded after traversal
      38 <class 'karl.content.models.blog.BlogEntry'>
      25 <type 'BTrees.OOBTree.OOBTree'>
...etc.

and the catalog version loaded 7,965 objects:

7976 ZODB objects loaded after traversal
    4634 <type 'BTrees.IFBTree.IFSet'>
    2867 <type 'BTrees.IFBTree.IFTreeSet'>
     228 <type 'BTrees.IOBTree.IOBucket'>
...etc.

For a big blog, the container version loaded:

1679 ZODB objects loaded after traversal
    1509 <class 'karl.content.models.blog.BlogEntry'>
      75 <type 'BTrees.OOBTree.OOBucket'>
      25 <type 'BTrees.OOBTree.OOBTree'>

...and the catalog version loaded:

8 ZODB objects loaded during traversal
       3 <type 'BTrees.OOBTree.OOBTree'>
       1 <class 'persistent.mapping.PersistentMapping'>
       1 <type 'BTrees.OOBTree.OOBucket'>
       1 <class 'karl.models.site.Site'>
       1 <class 'karl.models.community.CommunitiesFolder'>
       1 <class 'karl.models.community.Community'>
14161 ZODB objects loaded after traversal
    6230 <type 'BTrees.IFBTree.IFSet'>
    2867 <type 'BTrees.IFBTree.IFTreeSet'>
    2244 <type 'BTrees.IOBTree.IOBucket'>
    1509 <class 'karl.content.models.blog.BlogEntry'>
     968 <type 'BTrees.IIBTree.IIBucket'>
     113 <type 'BTrees.OIBTree.OIBucket'>

As the email from Shane notes, it is possible that I needed to mutate the catalog to get a more accurate reading. Carlos, perhaps we need to let you do the driving on karlstaging on some things, as I'm a bit of a bottleneck (and, you might want to experiment.)

One other thing to note...the blog view has a portlet which gets counts of earlier months. That probably defeats any attempt we have at only looking at 50 things.

[Paul Everitt (paul-agendaless)]

The only next step I can imagine is for Carlos to read and process Shane's email about the ?__debugload__ flag, plus the caveat about mutating the catalog, then trying some experiments on karlstaging. It should take an hour or less to prove/disprove that the catalog helps/hurts us on small/large blogs/folders.

Moving to January, less urgent.

[Carlos de la Guardia (cguardia)]

I made several tries with different parameters. The catalog batch loads way more objects on first load, and just a bit more objects than the container batch once the catalog is cached. It also is slower. I guess we'll need to find some other to optimize these views.

[Paul Everitt (paul-agendaless)]

Let's close this one as infeasible. Thanks for the investigation.