PPA ssh reset trigger can hang the buildd-manager indefinitely

[James Troup]

1000 21864 5.8 7.0 532052 145280 ? Sl Jul22 253:45 /usr/bin/python2.4 /srv/launchpad.net/codelines/current//bin/twistd --pidfile /srv/launchpad.net/var/buildd-manager.pid --python /srv/launchpad.net/codelines/current//daemons/buildd-manager.tac --logfile /srv/launchpad.net/production-logs/buildd-manager.log
1000 16442 0.0 0.1 49820 2876 ? S 18:59 0:00 \_ ssh -i /home/lp_buildd/.ssh/ppa-reset-builder <email address hidden>

i.e. it's been blocked on that ssh for almost 5 hours

Probably wants a pretty low (minutes) timeout?

[Celso Providelo (cprov)]

r8324 (devel)

[LaMont Jones (lamont)]

Still happens, e.g. 11:38 london time today, vs doubah.

[Julian Edwards (julian-edwards)]

The timeout is currently set to 180 seconds. I've no idea why it doesn't trigger, it might be a different problem on the buildd-manager. Is this still happening and how often?

[Julian Edwards (julian-edwards)]

And bang on cue this happens:

https://pastebin.canonical.com/26206/

[Julian Edwards (julian-edwards)]

Oddly it's not honouring the timeout, it appears to bail out immediately after trying to send data down the socket.

[Michael Hudson-Doyle (mwhudson)]

Jelmer and I looked at this and it seemed impossible, but I don't remember if that was before or after we found that there are two implementations of resetting virtual builders -- lp.buildmaster.model.BuilderSlave.resume and buildmaster.manager.RecordingSlave.resumeSlave -- and only one of them honours the time out (the RecordingSlave one).

Now I know a bit more about how the manager works, it seems likely that this is indeed the problem -- but I don't know why and when the non-recording slave's resume method is called.

[Julian Edwards (julian-edwards)]

Me neither without analysing it further :(

[Julian Edwards (julian-edwards)]

This should be fixed in 10.04. See bug 563353 as well.