branch-rewrite.py does not reconnect after database outage

[Stuart Bishop]

branch-rewrite.py runs in autocommit mode and it appears the Storm reconnection logic isn't kicking in.

2011-08-29 13:11:20 ERROR Exception occurred:
Traceback (most recent call last):
  File "scripts/branch-rewrite.py", line 57, in main
    print rewriter.rewriteLine(line.strip())
  File "/home/stub/lp/disco/lib/lp/codehosting/rewrite.py", line 109, in rewriteLine
    resource_location)
  File "/home/stub/lp/disco/lib/lp/codehosting/rewrite.py", line 60, in _getBranchIdAndTrailingPath
    location, from_slave=True)
  File "/home/stub/lp/disco/lib/lp/code/model/branchlookup.py", line 336, in getIdAndTrailingPath
    return self._getIdAndTrailingPathByUniqueName(store, path)
  File "/home/stub/lp/disco/lib/lp/code/model/branchlookup.py", line 318, in _getIdAndTrailingPathByUniqueName
    Branch.unique_name.is_in(prefixes), Branch.private == False).one()
  File "/home/stub/lp/lp-sourcedeps/eggs/storm-0.18.0.99_lpwithnodatetime_r394-py2.6-linux-x86_64.egg/storm/store.py", line 1161, in one
    result = self._store._connection.execute(select)
  File "/home/stub/lp/lp-sourcedeps/eggs/storm-0.18.0.99_lpwithnodatetime_r394-py2.6-linux-x86_64.egg/storm/databases/postgres.py", line 249, in execute
    return Connection.execute(self, statement, params, noresult)
  File "/home/stub/lp/lp-sourcedeps/eggs/storm-0.18.0.99_lpwithnodatetime_r394-py2.6-linux-x86_64.egg/storm/database.py", line 227, in execute
    self._ensure_connected()
  File "/home/stub/lp/lp-sourcedeps/eggs/storm-0.18.0.99_lpwithnodatetime_r394-py2.6-linux-x86_64.egg/storm/database.py", line 344, in _ensure_connected
    raise DisconnectionError("Already disconnected")
DisconnectionError: Already disconnected
NULL

[Stuart Bishop (stub)]

Reconnection is triggered via zstorm, where store.rollback() is called on transaction.abort(). branch-rewrite.py never calls transaction.abort().

[Stuart Bishop (stub)]

Landing blocked on RT #47614 (updating launchpad-developer-dependencies in buildbot environments).

[William Grant (wgrant)]

Rolled back in r13865. pgbouncer lurks in /usr/sbin, which isn't in buildbot's PATH. Needs fixing in python-pgbouncer.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r13863 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13863>.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r13875 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13875>.