ZODB: more control over object deactivations

[Dieter Maurer]

Several parts of Zope use volatile variables ("_v_")
which must not disappear during a transaction.
Most prominent examples are Zope's database adapters
that use a volatile attribute to hold the connection
to the database.

The current ZODB implementation can deactivate objects
at sub-transactions and transaction boundaries.
Deactivating objects with "_v_" attributes at subtransaction
boundaries, can seriously affect transaction consistency
(when a connection in a Zope database adapter is dropped).

This has beend discussed on zodb-dev in Dec. 2003. Most
people favoured a way to control object deactivation during
a transaction.

On Feb. 7th, 2004, I posted the attached patch to zodb-dev
with the following description. It allows to easily fix
problems with premature loss of "_v_" variables.

Quote from a "zodb-dev" post from Feb 7th, 2004, titled:
More "cPickleCache.incrgc" control
  (was: [Problem] "_v_" variables too volatile)

The attached patch against the Zope-Head CVS version
of 2004-02-07 provides more control as follows:

  * Persistent objects get a new "_p_sticky" attribute which
     can be read and written (from Python).

     It is a boolean, but currently implemented (in "C") as a byte
     (just because byte access is more efficient than bit access).

     The "Sticky" state could not be used for this purpose
     as it is reset at transaction boundaries while
     our "_p_sticky" must remain until object deactivation

  * "cPickleCache.incrgc" already had an ignored optional
     argument.
     This optional argument now gets the interpretation
     "mayFlushSticky" (and is no longer ignored).
     Its default is "True".

     If true, "incrgc" is allowed to deactivate objects
     with a true "_p_sticky". Otherwise, it must not deactivate
     such objects.

  * "ZODB.Connection" is changed such that it calls "cPickleCache.incrgc"
     with "mayFlushSticky" true only in
     "setDB", "sync" and at boundaries of main transactions (and
     not subtransactions).

     As subtransaction handling is a bit convoluted :-(
     "incrgc" may be called with "mayFlushSticky=False"
     even when a main transaction is aborted (as it is not
     always clear whether a main or a subtransaction is aborted).

  * "minimize" never flushes objects with "_p_sticky == True",
     as it is usually called asynchronously.

The patch does not automatically prevent "_v_" attribute flushing at
subtransaction boundaries. Only if "_p_sticky" is explicitely set to a true
value, a flush at subtransaction boundaries is prevented.

A "self->sticky = 1" in the "_v_" attribute handling
of "cPersistence:Per_setattro"
would make "_v_" protection automatic but
most discussion contributors said "explicit is better than implicit"
and saw use cases where "_v_" variables need no protection.

[Florent Guillaume (efge)]

Thanks Dieter.
I'd like to apply this patch quickly (I'd rather Tim did it though :))
Is it acceptable on the 2.7 branch ?
Would it need modifications on ZODB 3.3 for Zope 2.8 ?

[Dieter Maurer (d.maurer)]

The patch is already quite old.
Meanwhile, Tim took over ZODB maintenance any probably has cleaned up some parts. I expect that minor parts would need adjustment.

For Zope 3.3, I know from messages in zodb-dev, that some adjustments will be necessary as Tim removed e.g. "_ingrgc".

[Andreas Jung (ajung)]

Any volunteers to clean the patch for 2.8/2.7?

[Tres Seaver (tseaver)]

Status: Pending => Accepted

 Supporters added: tim_one

Tim, can you spare a cycle to look at the GC implications? The
rest of the patch I get, but I've never had the zen of playing
nice with Python's GC.

[Dieter Maurer (d.maurer)]

There is a problem with the patch, when the "_p_sticky"
is set in the class (rather than the instance).

In our current version, I copy the class attribute into the
instance attribute on instance activation provided the former
exists. But this is still not complete, as the instance might
have been newly added to the ZODB (and then become "active" without
going through an activation).

[Tim Peters (tim-one)]

Unassigned myself -- I haven't done ZODB work in nearly 3 years :-(

[Hanno Schlichting (hannosch)]

This is really just a ZODB issue.

[Kirill Smelkov (navytux)]

For the reference: wendelin.core also needs to preserve ._v_* attributes to not go away automatically during transaction. And to do so it adds LivePersistent class, which is just a Persistent with deactivated "_p_deactivate":

https://lab.nexedi.com/nexedi/wendelin.core/blob/a702d410/lib/zodb.py#L88-109
https://lab.nexedi.com/nexedi/wendelin.core/blob/a702d410/bigfile/file_zodb.py#L484-500
https://lab.nexedi.com/nexedi/wendelin.core/blob/a702d410/lib/tests/test_zodb.py#L53-183