publish-distro.py hangs on librarian connectivity glitches

[Steve McInerney]

germanium: cronscripts/publishing/cron.publish-ppa was hung this morning, apparently for ~ 14 hours.
This coincided with a fast reboot on the server it was talking to:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1005 9139 0.0 0.0 4096 464 ? Ss 10:20 0:00 /bin/sh -c nice -n 5 /srv/launchpad.net/production/launchpad/cronscripts/publishing/cron.publish-ppa >> /srv/launchpad.net/production-logs/lp_publish/cron.ppa.log 2>&1
1005 9141 0.0 0.0 10728 712 ? SN 10:20 0:00 \_ /bin/bash /srv/launchpad.net/production/launchpad/cronscripts/publishing/cron.publish-ppa
1005 9552 0.1 4.1 597536 166912 ? SNl 10:20 1:08 \_ /usr/bin/python -S /srv/launchpad.net/production/launchpad/cronscripts/publishing/../../scripts/publish-distro.py --ppa -d ubuntu

lp_publish@germanium:~$ date -u
Sun Oct 16 23:59:32 UTC 2011

lp_publish@germanium:~$ strace -p 9552
Process 9552 attached - interrupt to quit
recvfrom(12, ^C <unfinished ...>
Process 9552 detached

lp_publish@germanium:~$ lsof -p 9552
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
...
publish-d 9552 lp_publish 12u IPv4 3916754937 0t0 TCP germanium.canonical.com:49681->launchpadlibrarian-net.nutmeg.canonical.com:www (ESTABLISHED)

suggest that some sort of failure/retry/timeout fix would be appropriate?

[William Grant (wgrant)]

This is probably a general issue in our librarian download handling.