large bzr-svn imports fail with branch.fetch_tags enabled

[Michael Hudson-Doyle]

This seems new since the last rollout, see for example https://code.launchpad.net/~vcs-imports/gcc/trunk

Note that "Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure." is code for the import job being killed because there was no output on stdout for an hour.

Apparently it's something to do with tags.

[Jelmer Vernooij (jelmer)]

bzr now fetches not just tag metadata but all tagged revisions as well. This causes a significant increase in the number of revisions fetched, which causes this particular timeout.

[Matthias Klose (doko)]

blocks the packaging and upload of the gcc-4.6.1 release for oneiric (linaro needs to merge this first)

[Kate Stewart (kate.stewart)]

This is going to affect the ubuntu 11.10 (esp. with feature freeze) release - due to the dependency chain between the compiler and hence the arm linux builds.

[Michael Hudson-Doyle (mwhudson)]

I don't know how relevant the point is, but one way out of this would be more progress during the importing of the tagged revisions -- the jobs are only killed after an hour of no output at all.

Another option would be to raise the timeout in the system to allow the imports to pass. This will mean more impact from imports that get genuinely stuck of course, but it might be appropriate for a temporary hack. Do you know what we'd need to lift the timeout to?

[Jelmer Vernooij (jelmer)]

On 29/06/11 02:18, Michael Hudson-Doyle wrote:
> I don't know how relevant the point is, but one way out of this would be
> more progress during the importing of the tagged revisions -- the jobs
> are only killed after an hour of no output at all.
>
> Another option would be to raise the timeout in the system to allow the
> imports to pass. This will mean more impact from imports that get
> genuinely stuck of course, but it might be appropriate for a temporary
> hack. Do you know what we'd need to lift the timeout to?
Memory usage is the main problem as far as I can tell. The memory usage
is not just affecting the bzr-svn imports in question but also the other
imports on the same machine, as they force the machine into swap. I'm a
bit wary of making these imports time out later, the current timeout is
already quite high.

Cheers,

Jelmer

[Steve Langasek (vorlon)]

Oneiric unblocked by a by-hand import (thanks, Jelmer!); this should be enough to let Linaro move forward with 4.6.1 AIUI.

[John A Meinel (jameinel)]

Jelmer and I discussed a bit at the last sprint trying to help with the memory issue for bzr-svn. For tags, I guess it currently requests the full history of each tag into an iterator, expecting to stop it at some point (once it reaches 'trunk'). Instead, we could be presumptuous that tagged revisions won't exist far from trunk and only request short histories. And then if something hasn't converged yet, re-start the iterator.

As for the 'no output for 1hr', that sounds like just adding more "I've gotten some data" ticks would be reasonable.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r13660 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13660>.