branch scanner wedging, still

[Steve McInerney]

had a wedged instance on the 5th of Feb on branch '~russel/groovy/trunk'.

unchecked the subscriptions, killed and restarted the scanner.

[Jonathan Lange (jml)]

Possibly fixed by fix to bug 280578.

[Steve McInerney (spm)]

From the branchscanner logs:
Had a few failures of this particular branch in the leadup to the wedging:

2009-02-05 07:19:32 INFO Scanning branch: ~russel/groovy/trunk
2009-02-05 07:19:32 INFO from lp-mirrored:///~russel/groovy/trunk
2009-02-05 07:19:32 INFO Retrieving ancestry from bzrlib.
2009-02-05 07:19:35 INFO Retrieving ancestry from database.
2009-02-05 07:19:35 INFO Planning changes.
2009-02-05 07:19:35 INFO Adding 0 new revisions.
2009-02-05 07:19:35 INFO Deleting 667 branchrevision records.
2009-02-05 07:19:38 INFO Inserting 671 branchrevision records.
2009-02-05 07:20:15 INFO creating lockfile
2009-02-05 07:20:24 INFO OOPS-1132SMS769: Failed to scan
2009-02-05 07:20:24 INFO Finished branch scanning
2009-02-05 07:21:11 INFO creating lockfile
2009-02-05 07:21:38 INFO Starting branch scanning
....

2009-02-05 07:24:26 INFO Scanning branch: ~russel/groovy/trunk
2009-02-05 07:24:26 INFO from lp-mirrored:///~russel/groovy/trunk
2009-02-05 07:24:26 INFO Retrieving ancestry from bzrlib.
2009-02-05 07:24:30 INFO Retrieving ancestry from database.
2009-02-05 07:24:30 INFO Planning changes.
2009-02-05 07:24:30 INFO Adding 0 new revisions.
2009-02-05 07:24:30 INFO Deleting 667 branchrevision records.
2009-02-05 07:24:33 INFO Inserting 671 branchrevision records.
2009-02-05 07:25:05 INFO OOPS-1132SMS778: Failed to scan
2009-02-05 07:25:05 INFO Finished branch scanning
2009-02-05 07:25:13 INFO creating lockfile
2009-02-05 07:25:45 INFO Starting branch scanning
....

But finally blew up totally and wedged:

2009-02-05 07:30:52 INFO Scanning branch: ~russel/groovy/trunk
2009-02-05 07:30:52 INFO from lp-mirrored:///~russel/groovy/trunk
2009-02-05 07:30:52 INFO Retrieving ancestry from bzrlib.
2009-02-05 07:30:59 INFO Retrieving ancestry from database.
2009-02-05 07:30:59 INFO Planning changes.
2009-02-05 07:31:00 INFO Adding 0 new revisions.
2009-02-05 07:31:00 INFO Deleting 667 branchrevision records.
2009-02-05 07:31:07 INFO Inserting 671 branchrevision records.
2009-02-05 07:31:12 INFO creating lockfile
2009-02-05 07:32:07 INFO creating lockfile
2009-02-05 07:33:08 INFO creating lockfile
2009-02-05 07:34:06 INFO creating lockfile
2009-02-05 07:35:13 INFO creating lockfile
2009-02-05 07:36:11 INFO creating lockfile
...

until manually killed, cleaned and restarted.

[Tim Penhey (thumper)]

There are two issues here:
 1) The scanner is getting wedged generating the body (not the diff) of the email, and Aaron is going to move this into the email job code.
 2) There is some bzrlib bug where it got wedged, so maybe now just the job will get wedged :-|