Clock skew and packing

[Lalo Martins]

If you have transactions with a timestamp in the future (probably due to clock skew), they will never be packed away, at least not until you get to that date again.

If you compound that with an application-level bug that for some reason causes the database file to grow too fast, then you'll have a really huge and slow server and no easy way to fix it.

Proposed solution: on ZODB.fspack.GC.buildPackIndex, consider transactions that are in the future as infinitely old. Something like this:

[Tim Peters (tim-one)]

There's more about this patch on the zodb-dev list:

http://mail.zope.org/pipermail/zodb-dev/2005-September/009248.html

If the patch works, it might be nice to have an option to explicitly ask for it, but I wouldn't want to assume that "time in the future means infinitely old" by default.

In all, if I had time to look at this, I'd rather spend it on automating a way to force timestamps in badly skewed databases back to sane times.

[Thomas Güttler (hv-tbz-pariv)]

Uploaded: BaseStorage_reset_tids.patch

I had the problem, too. The clock was set to the year 2009 for very short, but now all new tids are from 2009.

Tim Peter suggested, that you can solve the problem by doing
a storage.copyTransactionsFrom by setting the tid to None.
See bottom of:
http://mail.zope.org/pipermail/zodb-dev/2005-September/009225.html

This is a small path to BaseStorage with gives the method copyTransactionsFrom a new keyword argument "reset_tids".

If you use the method once, all tids are reset to the current time.
This means, you don't need the patch zodb-pack-future.patch anymore. I prefere this, because you only need to repair the database once, while other patch need to installed for ever.

[Jim Fulton (jim-zope)]

A better patch would set tids > now to about now, making sure their order is preserved.

A better patch would also have a test. :)

Doing this as part of pack has the operational advantage that it doesn't require down time.