Should be easy to implement a transaction retry policy

[James Henstridge]

Similar to bug 94986 (adding the necessary features to implement database reconnection), Storm should make it easy to implement a transaction retry policy for certain classes of problems.

When this is fixed, it should be easy for an application to detect database errors that indicate serialisation problems that should be retried. For a Zope application, this would involve converting such errors to Retry exceptions in the publisher's handleException() method.

For Launchpad, we currently retry the following exceptions (this is Postgres with psycopg1):

 1. any psycopg.IntegrityError
 2. any psycopg.Error with a message that starts with one of:
    * ERROR: could not serialize access due to concurrent update
    * ERROR: deadlock detected

Ideally these exceptions should be marked such that they can easily be identified and handled appropriately.

[James Henstridge (jamesh)]

I've posted a psycopg2 patch to the mailing list at:
    http://thread.gmane.org/gmane.comp.python.db.psycopg.devel/4384

When it gets applied, the list of errors we want to retry will be limited to just psycopg2.IntegrityError and psycopg2.extensions.TransactionRollbackError.

[James Henstridge (jamesh)]

The psycopg2 patch has been merged as r923, so the next step is to decide how this should be handled in Storm.

[Free Ekanayaka (free.ekanayaka)]

Hi,

the following might be not entirely related to the bug, but it seems that the transaction module has support for something along these lines. One can write:

for attempt in transaction.attempts(attempts):
    with attempt as txn:
        # run application code here, like publish(request)

where the code of Attempt.__exit__ in the transaction module reads:

    def __exit__(self, t, v, tb):
        if v is None:
            self.manager.commit()
        else:
            retry = self.manager._retryable(t, v)
            self.manager.abort()
            return retry

in turn Manager._retryable will simply loop through the data managers and invoke their "should_retry" method, if available, a return True if any of them does:

    def _retryable(self, error_type, error):
        if issubclass(error_type, TransientError):
            return True

        for dm in self.get()._resources:
            should_retry = getattr(dm, 'should_retry', None)
            if (should_retry is not None) and should_retry(error):
                return True

So we could at least make retries very easy when using Storm with the transaction module, by providing a stock should_retry implementation, that could be possibly overridden by application code with some API.