Comment 1 for bug 86171

Revision history for this message
Stuart Bishop (stub) wrote : Re: Exception in Librarian caused outage

The trigger this time was a SERIALIZATION exception, which we can expect to happen regularly. I think the best fix for the trigger is to not run the librarian in the SERIALIZABLE transaction isolation, and fixing this is critical. After that, we can look at making the worker threads more robust in handling unexpected exceptions.

2007/02/18 19:14 GMT [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python2.4/threading.py", line 422, in run
            self.__target(*self.__args, **self.__kwargs)
          File "/srv/librarian.ubuntu.com/launchpad/lib/twisted/python/threadpool.py", line 148, in _worker
            context.call(ctx, function, *args, **kwargs)
          File "/srv/librarian.ubuntu.com/launchpad/lib/twisted/python/context.py", line 59, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/srv/librarian.ubuntu.com/launchpad/lib/twisted/python/context.py", line 37, in callWithContext
            return func(*args,**kw)
        --- <exception caught here> ---
          File "/srv/librarian.ubuntu.com/launchpad/lib/twisted/internet/threads.py", line 25, in _putResultInDeferred
            result = f(*args, **kwargs)
          File "/srv/librarian.ubuntu.com/launchpad/lib/canonical/librarian/web.py", line 88, in _getFileAlias
            commit()
          File "/srv/librarian.ubuntu.com/launchpad/lib/canonical/database/sqlbase.py", line 611, in commit
            ZopelessTransactionManager._installed.commit()
          File "/srv/librarian.ubuntu.com/launchpad/lib/canonical/database/sqlbase.py", line 339, in commit
            self.manager.get().commit()
          File "/srv/librarian.ubuntu.com/launchpad/lib/transaction/_transaction.py", line 380, in commit
            self._saveCommitishError() # This raises!
          File "/srv/librarian.ubuntu.com/launchpad/lib/transaction/_transaction.py", line 378, in commit
            self._commitResources()
          File "/srv/librarian.ubuntu.com/launchpad/lib/transaction/_transaction.py", line 436, in _commitResources
            rm.tpc_vote(self)
          File "/srv/librarian.ubuntu.com/launchpad/lib/transaction/_transaction.py", line 648, in tpc_vote
            self._datamanager.prepare(transaction)
          File "/srv/librarian.ubuntu.com/launchpad/lib/sqlos/transaction/__init__.py", line 159, in prepare
            obj.sync()
          File "/srv/librarian.ubuntu.com/launchpad/lib/sqlobject/main.py", line 704, in sync
            self.syncUpdate()
          File "/srv/librarian.ubuntu.com/launchpad/lib/sqlobject/main.py", line 723, in syncUpdate
            self._connection._SO_update(self, values)
          File "/srv/librarian.ubuntu.com/launchpad/lib/sqlobject/dbconnection.py", line 649, in _SO_update
            self.query("UPDATE %s SET %s WHERE %s = %s" %
          File "/srv/librarian.ubuntu.com/launchpad/lib/sqlobject/dbconnection.py", line 865, in query
            return self._dbConnection._query(self._connection, s)
          File "/srv/librarian.ubuntu.com/launchpad/lib/sqlobject/dbconnection.py", line 394, in _query
            self._executeRetry(conn, conn.cursor(), s)
          File "/srv/librarian.ubuntu.com/launchpad/lib/sqlobject/dbconnection.py", line 389, in _executeRetry
            return cursor.execute(query)
        psycopg.ProgrammingError: ERROR: could not serialise access due to concurrent update

        UPDATE LibraryFileAlias SET last_accessed = CURRENT_TIMESTAMP AT TIME ZONE 'UTC' WHERE id = 2673003