ZODB error when copy-pasting a blob file (zope.file)

Bug #240381 reported by Christophe Combelles on 2008-06-16
4
Affects Status Importance Assigned to Milestone
ZODB
Medium
Christian Theune
zope.copypastemove
High
Christian Theune

Bug Description

- I create a clean buildout with zopeproject (either as is, or pointing to the 3.4 KGS).
- I add zope.mimetype and zope.file in the setup.py and configure.zcml
- I replace the filestorage with a blobstorage in zope.conf:
  <blobstorage>
    blob-dir var/blob
    <filestorage>
      path var/Data.fs
    </filestorage>
  </blobstorage>

- I log into the zmi
- I add a zope.file object
- copy and paste it:

2008-06-16T14:28:13 ERROR SiteError http://127.0.0.1:8080/@@contents.html
Traceback (most recent call last):
  File "/home/ccomb/buildout-eggs/tmpcCNQrB/zope.publisher-3.5.2-py2.4.egg/zope/publisher/publish.py", line 138, in publish
  File "/home/ccomb/buildout-eggs/tmpg7_LmF/zope.app.publication-3.4.3-py2.4.egg/zope/app/publication/browser.py", line 78, in afterCall
  File "/home/ccomb/buildout-eggs/tmpg7_LmF/zope.app.publication-3.4.3-py2.4.egg/zope/app/publication/zopepublication.py", line 175, in afterCall
  File "/home/ccomb/buildout-eggs/tmpYbaQQJ/ZODB3-3.9.0_dev_r77011-py2.4-linux-i686.egg/transaction/_transaction.py", line 325, in commit
  File "/home/ccomb/buildout-eggs/tmpYbaQQJ/ZODB3-3.9.0_dev_r77011-py2.4-linux-i686.egg/transaction/_transaction.py", line 424, in _commitResources
  File "/home/ccomb/buildout-eggs/tmpYbaQQJ/ZODB3-3.9.0_dev_r77011-py2.4-linux-i686.egg/ZODB/Connection.py", line 538, in commit
  File "/home/ccomb/buildout-eggs/tmpYbaQQJ/ZODB3-3.9.0_dev_r77011-py2.4-linux-i686.egg/ZODB/Connection.py", line 583, in _commit
  File "/home/ccomb/buildout-eggs/tmpYbaQQJ/ZODB3-3.9.0_dev_r77011-py2.4-linux-i686.egg/ZODB/Connection.py", line 626, in _store_objects
  File "/home/ccomb/buildout-eggs/tmpYbaQQJ/ZODB3-3.9.0_dev_r77011-py2.4-linux-i686.egg/ZODB/blob.py", line 85, in _p_invalidate
TypeError: unsupported operand type(s) for +: 'NoneType' and 'NoneType'

Christophe Combelles (ccomb) wrote :

A first fix is :

In ZODB/blob.py:85, replace:
        for ref in self.readers+self.writers:
with
        for ref in (self.readers or []) + (self.writers or []):

But that's not enough, the copied file is empty.

Changed in zope3:
importance: Undecided → High
Christian Theune (ctheune) wrote :

Happens here as well.

Changed in zope3:
assignee: nobody → ct-gocept
status: New → Confirmed
Christian Theune (ctheune) wrote :

There are two actual problems on this:

1. zope.copypastemove pickles/unpickles the blob object without calling it's __setstate__ method. This is unexpected from the Blob class and I can't see immediately what happens in pickle/cPickle that prevents this.
2. The blob class uses its object id and a temporary filename to maintain its state. A simple pickle/unpickle/store routine will turn it into an empty blob. I don't see an obvious solution on this either. It sounds like the blob class would have to get a lot smarter to support this use case.

I'm unassigning myself for now.

Christophe Combelles (ccomb) wrote :

Where is the blobfile supposed to be copied?
Should it be copied during the pickling/unpickling of the blob object, by sort some of hook ?

Actually, the pickling/unpickling is initiated in zope.location.pickling.locationCopy,
and I've tried an small hack that writes the new blobfile at the end of this function.

By replacing:

    return unpickler.load()

with:

    copied = unpickler.load()
    if hasattr(copied, '_data') and type(copied._data) is Blob:
        copied._data._create_uncommitted_file()
        targetfile = open(copied._data._current_filename(), 'w')
        sourcefile = loc.openDetached()
        chunk = sourcefile.read(1000000)
        while chunk:
            targetfile.write(chunk)
            chunk = sourcefile.read(1000000)
        targetfile.close()
    return copied

It also need the fix mentionned in my previous comment (blob.py).

Jim Fulton (jim-zope) wrote :

What is the status of this? This looks like a limitation in the approach of Zope 3's copyy/paste approach. Basically, a blob is a case of an object that can't be copied through pickling alone. I suspect there's a feature request lurking here. And I have a vague idea that someone "fixed" this.

Changed in zodb:
importance: Undecided → Medium
status: New → Incomplete
Dan Korostelev (nadako) wrote :

Now, when zope.copy (formerly zc.copy) pluggable mechanism is integrated into zope.copypastemove, it can be easily fixed using copy hook adapters. I've done that for z3c.blobfile some time ago, please see http://svn.zope.org/z3c.blobfile/trunk/src/z3c/blobfile/copy.py?view=auto

So no further change is needed in ZODB?

Jim

On Sun, Aug 23, 2009 at 8:17 AM, Dan Korostelev<email address hidden> wrote:
> Now, when zope.copy (formerly zc.copy) pluggable mechanism is integrated
> into zope.copypastemove, it can be easily fixed using copy hook
> adapters. I've done that for z3c.blobfile some time ago, please see
> http://svn.zope.org/z3c.blobfile/trunk/src/z3c/blobfile/copy.py?view=auto
>
> --
> ZODB error when copy-pasting a blob file (zope.file)
> https://bugs.launchpad.net/bugs/240381
> You received this bug notification because you are a member of Zope 3
> Developers, which is subscribed to Zope 3.
>

--
Jim Fulton

Dan Korostelev (nadako) wrote :

I guess so.

Jim Fulton (jim-zope) wrote :

This seems to have been addressed at the zope level.

Changed in zodb:
status: Incomplete → Won't Fix
Christian Theune (ctheune) wrote :

Actually this is interesting on the ZODB level. With current discussion about high quality backup strategies on the ZODB development list and the pointer back to this bug I found a solution how this might be solved easily on the ZODB level.

The underlying issue here is that blobs use the OID as the pointer to which file belongs to it. When copying a blob object that piece of data goes away. IMHO the ZODB kinda nurishes the assumption that the content of an object is what is stored in the pickle, so as a consequence if I pickle/unpickle such an object its value should be the same.

Thus th idea of a solution: we could save the OID in the state of the blob object and if that is different from the _p_oid then we know we need to initialise the blob from that object's state, e.g. by providing a new hard link to that file.

Maybe we don't even have to store that marker in the object's value persistently, but I don't know, we'll see.

Changed in zodb:
assignee: nobody → Christian Theune (ct-gocept)
status: Won't Fix → Confirmed
Jim Fulton (jim-zope) wrote :

On Fri, Nov 20, 2009 at 2:49 PM, Christian Theune <email address hidden> wrote:
> Actually this is interesting on the ZODB level. With current discussion
> about high quality backup strategies on the ZODB development list and
> the pointer back to this bug I found a solution how this might be solved
> easily on the ZODB level.
>
> The underlying issue here is that blobs use the OID as the pointer to
> which file belongs to it.

All objects use the oid as a key to where the data are.

> When copying a blob object that piece of data
> goes away.

Also when copying a regular object.

> IMHO the ZODB kinda nurishes the assumption that the content
> of an object is what is stored in the pickle, so as a consequence if I
> pickle/unpickle such an object its value should be the same.

If you buy the premise, which I don't. Copying objects is always tricky
and somewhat application dependent.

> Thus th idea of a solution: we could save the OID in the state of the
> blob object and if that is different from the _p_oid then we know we
> need to initialise the blob from that object's state, e.g. by providing
> a new hard link to that file.
>
> Maybe we don't even have to store that marker in the object's value
> persistently, but I don't know, we'll see.

You haven't fleshed this out enough for me to judge this. You also
haven't really defined copying for that matter. There are many ways to
copy an object and it's not clear what's meant in the context of ZODB,
which is why I view this as an application issue, rather than a ZODB issue.

Perhaps you should come up with some definition of copying in the context of
ZODB and then we can decide if there is a compelling reason to
have such a feature in ZODB.

Given other priorities, I don't think we should try to deal with this.

Jim

--
Jim Fulton

affects: zope3 → zope.copypastemove
Jim Fulton (jim-zope) on 2010-04-22
Changed in zodb:
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers