Large OFS.Image.File download trashes connectino cache

[Tres Seaver]

'OFS.Image.File' breaks up "large" files into a singly-linked list of 'Pdata'
chunks, each stored as its own Persistent object. This strategy keeps Zope
from loading the entire file into RAM when serviing it, which is a Good
Thing(TM).

However, it reads through those 'Pdata' objects from its connection one at a
time, and the connection *caches* them. This has two downsides:

  - Once served out the the client, the 'Pdata' objects remains in the cache,
    where they are not likely to be needed again (large file downloads are
    much rarer than catalog queries, for instance).

  - Worse, the 'Pdata' objects are "newer" than the objects already in the
    cache, and thus *displace* them when the cache uses LRU to shrink back to
    its target size. The application thus spends a lot of pain reloading its
    old "working set".

I have a branch in Subversion[1] which modifies 'OFS.Image' as follows:

  - It adds a class, 'FileIterator', which implements
    'ZPublisher.Iterators.IStreamIterator', using the data from an
    'OFS.Image.File'.

  - It changes 'OFS.Image.File.index_html', such that in the case
    where it would previously have used 'RESPOSNE.write' to "stream"
    data to the client, it now manufactures a transient 'ZODB.Connection'
    object, not tied to the main appserver thread's connection.
    It then fetches the "clone" of the file object from that connection,
    and returns a 'FileIterator' wrapped around the clone.

Under light testing, I can see that the number of reads apparent on the
'Control_Panel/Database' tab drops of sharply when running with my branch,
because the 'Pdata' objects read in the transient connection are not
affecting the long-lived connection objects.

Note that this patch does *not* deal with cache-thrasing at the level
of the disk-based ZEO Client Cache; it is purely relevant to the object
cache maintained in RAM.

[1] http://svn.zope.org/Sandbox/tseaver/no_pdata_in_cache_dammit/

[Tres Seaver (tseaver)]

A Zope2 which has ZODB >= 3.8.0 could use the blob storage feature provided in
that version to implement OFS.Image.File, which would remove the need for my patch
altogether.

[Philipp von Weitershausen (philikon)]

On 26 Oct 2007, at 00:05 , Tres Seaver wrote:
> A Zope2 which has ZODB >= 3.8.0 could use the blob storage feature
> provided in that version to implement OFS.Image.File, which would
> remove the need for my patch altogether.

It could, but that doesn't solve the problem for Zope <= 2.10, old
pickles and installations w/o a configured blob storage.

[Colin Watson (cjwatson)]

The zope2 project on Launchpad has been archived at the request of the Zope developers (see https://answers.launchpad.net/launchpad/+question/683589 and https://answers.launchpad.net/launchpad/+question/685285). If this bug is still relevant, please refile it at https://github.com/zopefoundation/zope2.