Intermittent failures doing "bzr branch"

[John A Meinel]

We are seeing CI failures trying to branch one of our branches. The failure is intermittent, though since Saturday it has been failing at least as often as it succeeds.

Debugging with Colin seems to find at least one traceback with a known Twisted bug:
https://pastebin.canonical.com/p/kbf7m2H67m/

I was able to reproduce the failure fairly regularly with:
for i in `seq 100`; do rm -rf xx; time bzr branch 'lp:~canonical-juju-qa/update-juju-streams/trunk' xx; done

[John A Meinel (jameinel)]

See also this Canonical RT: https://portal.admin.canonical.com/C131990/

[Colin Watson (cjwatson)]

I believe this is fixed in the Twisted 20.3.0+lp4 backport that we're already using in turnip. Reusing that here would be my preference, but we're currently on 19.2.1 so we'll need to see if there are any issues with upgrading across that gap.

[John A Meinel (jameinel)]

As an update today talking to Colin, it seems there was a bit of fallout trying to change codehosting to 20.3. It still is the easiest way forward, so he'll look at updating the code for the Twisted changes, and it seems likely that there will be progress in another day or so.

[Colin Watson (cjwatson)]

The failures are actually related to bug 1937345, because we have two separate regressions we're trying to fix concurrently here :-(

[Colin Watson (cjwatson)]

20 consecutive successes after rolling out the Twisted upgrade, so I'm going to call this good now.

[John A Meinel (jameinel)]

I can confirm that our email spam from our CI bot stopped at UTC 10:30 and this does, indeed, appear fixed for us as well.

[Colin Watson (cjwatson)]

For the record, the upstream Twisted change was https://github.com/twisted/twisted/pull/1478.