ZCatalog DateRangeIndex caching makes REQUEST large

[khink]

DateRangeIndex's _apply_index() [1] stores its result on the REQUEST [2] for re-use in the same REQUEST. This is also the case for the query for "active content" (effective date in the past), which is very common in Plone. For large databases, this will be a huge result.

It's hard to say if that is a bad thing per se, i think it should be benchmarked.

One thing is sure: It makes the DebugPanel a lot less usable for sure, because that prints REQUEST.other, resulting in a large HTML page (6Mb for about 450,000 record ids in the result cache ).

An alternative would be to cache the _apply_index() result someplace other than on the request.

This report follows from a discussion on plone-users [3].

This probably applies to other indexes as well.

[1] Products.PluginIndexes.DateRangeIndex.DateRangeIndex.DateRangeIndex._apply_index()
[2] REQUEST in capitals, to discern it from 'request' in the _apply_index() code, where it just means the catalog query
[3] http://plone.293351.n2.nabble.com/request-other-has-large-portal-catalog-X-key-td7566058.html

[Hanno Schlichting (hannosch)]

When we introduced this cache, it was based on benchmarking against small and large sites. As the request itself is short-lived the memory is just used for a short period of time.

This is the same sort of request cache that's often found in Plone via plone.memoize. The main difference is that plone.memoize makes the assumption that the request object is annotatable (in the zope.annotation sense) and thus stores its cached values in annotations. Zope itself doesn't make this assumption, so direct Zope code cannot use this approach. Storing cached values in request.other was therefor the least invasive approach.

Unless someone can produce benchmarks which show an adverse effect of this cache, I'd consider the current approach to be reasonable.

[Colin Watson (cjwatson)]

The zope2 project on Launchpad has been archived at the request of the Zope developers (see https://answers.launchpad.net/launchpad/+question/683589 and https://answers.launchpad.net/launchpad/+question/685285). If this bug is still relevant, please refile it at https://github.com/zopefoundation/zope2.