scan_branches uses too much memory, bogs down machine

[Martin Pool]

Following on from bug 690021, about scan_branches bogging down the machine:

Apparently scan_branches.py uses a lot of memory scanning some linux import branches. We should work out why and fix it. Log in https://pastebin.canonical.com/40916/

[John A Meinel (jameinel)]

Isn't this because scan_branches grabs the whole history of the branches, to figure out what is 'new' to be imported, rather than incrementally walking them to see what is missing? I don't fully understand what bzr needs to do here.

[Robert Collins (lifeless)]

@john I think the theory is that holding just the revids for all of e.g. linux shouldn't be a memory burden in and off itself, that something else is causing the issue (e.g. fragmentation, bzr caches ...)

[Martin Pool (mbp)]

This doesn't need a bzr specific task unless/until we determine the problem is actually internal to bzr. Diagnosis needs to start on the lp side.