Bzr timeouts on SSH connections from people.canonical.com

[David Planella]

I mentioned this on #launchpad last week, and today I'm experiencing it again.

I'm trying to check out a branch for a script to be run on the people.canonical.com server, and I'm continually getting timeouts. As an example:

$ bzr pull lp:giraffe
ssh: connect to host bazaar.launchpad.net port 22: Connection timed out
bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.

I started noticing this a few weeks ago, when the problem appeared on and off. Now it seems to be there all the time.

Here's the bzr version that runs on the server:

$ bzr --version
Bazaar (bzr) 2.1.1
  Python interpreter: /usr/bin/python 2.6.5
  Python standard library: /usr/lib/python2.6
  Platform: Linux-2.6.32-27-server-x86_64-with-Ubuntu-10.04-lucid
  bzrlib: /usr/lib/python2.6/dist-packages/bzrlib
  Bazaar configuration: /home/dpm/.bazaar
  Bazaar log file: /home/dpm/.bzr.log

[Robert Collins (lifeless)]

Getting timeouts from people.canonical.com to bazaar.launchpad.net is extremely worrying, and a timeout here is a regression from our normal reliability.

[John A Meinel (jameinel)]

I was told that the forking service was started on production, but the configuration was not enabled. If the configuration *was* enabled, that might explain this (since it is what happened when we rolled it out initially). While I doubt this is the actual problem, I figured we can at least have someone check the config in production to verify.

[John A Meinel (jameinel)]

Note that I can download the data from my home machine without a problem. However doing:
jameinel@lillypilly:~$ echo hello | ssh <email address hidden> -v bzr serve --inet --directory=/ --allow-writes

gives
OpenSSH_5.3p1 Debian-3ubuntu3.1.IS.10.04, OpenSSL 0.9.8k 25 Mar 2009
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to bazaar.launchpad.net [91.189.90.11] port 22.
debug1: connect to address 91.189.90.11 port 22: Connection timed out
ssh: connect to host bazaar.launchpad.net port 22: Connection timed out

This looks like a routing/firewall issue, not a bzr or Launchpad configuration issue.

[Martin Pool (mbp)]

Did the start of this problem coincide with the introduction of haproxy?
Is it possible that it's doing something strange that's causing the port to sometimes not be listening, or perhaps only on a specific ip that's not reached by the route from lillypilly?

[Martin Pool (mbp)]

rt 44341

[Robert Collins (lifeless)]

Ok, we're satisfied this isn't a code issue - the RT ticket will address the firewall in the datacentre. Until then you may be able to use http:// bazaar.launchpad.net instead.

[Martin Pool (mbp)]

For the benefit of the bug, no, haproxy isn't live yet so this probably is unrelated.