We have fixed the code that does smart fetches to not fetch more than 5MB at a time, but IIRC we don't have any code that limits the local fetching.
So it seems we are trying to copy >2GB of data in one read + write, and that breaks fp.read().
As a side note, it appears that you are not using a shared repository, which means we have to copy the 20GB of history around. If you had done "bzr init-repo" at the beginning, we would only need to create new working-trees or very lightweight branches, etc.
Note that you only want to do this if you truly *don't* have a shared repository already, otherwise it deletes the existing shared repo.
Anyway, the simple fix is to pass "max_size=XXXX" to Transport._coalesce_offsets. At the moment we use "0" as the default value, which means unlimited. We could make 0 == 2GB, or more reasonably have a harder max-size of 10MB or so. 10MB should be a reasonable buffer size that doesn't hit all of available RAM, and also avoids enough round trips.
We have fixed the code that does smart fetches to not fetch more than 5MB at a time, but IIRC we don't have any code that limits the local fetching.
So it seems we are trying to copy >2GB of data in one read + write, and that breaks fp.read().
As a side note, it appears that you are not using a shared repository, which means we have to copy the 20GB of history around. If you had done "bzr init-repo" at the beginning, we would only need to create new working-trees or very lightweight branches, etc.
As a quick workaround, you could do:
bzr init-repo branch/ .bzr/repository .bzr /shared- storage
rm .bzr/repository -rf
mv existing_
touch .bzr/repository
Note that you only want to do this if you truly *don't* have a shared repository already, otherwise it deletes the existing shared repo.
Anyway, the simple fix is to pass "max_size=XXXX" to Transport. _coalesce_ offsets. At the moment we use "0" as the default value, which means unlimited. We could make 0 == 2GB, or more reasonably have a harder max-size of 10MB or so. 10MB should be a reasonable buffer size that doesn't hit all of available RAM, and also avoids enough round trips.
As an example, see the attached patch.