Bzr timeouts on SSH connections from people.canonical.com

Bug #727148 reported by David Planella on 2011-03-01
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Critical
Unassigned

Bug Description

I mentioned this on #launchpad last week, and today I'm experiencing it again.

I'm trying to check out a branch for a script to be run on the people.canonical.com server, and I'm continually getting timeouts. As an example:

$ bzr pull lp:giraffe
ssh: connect to host bazaar.launchpad.net port 22: Connection timed out
bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.

I started noticing this a few weeks ago, when the problem appeared on and off. Now it seems to be there all the time.

Here's the bzr version that runs on the server:

$ bzr --version
Bazaar (bzr) 2.1.1
  Python interpreter: /usr/bin/python 2.6.5
  Python standard library: /usr/lib/python2.6
  Platform: Linux-2.6.32-27-server-x86_64-with-Ubuntu-10.04-lucid
  bzrlib: /usr/lib/python2.6/dist-packages/bzrlib
  Bazaar configuration: /home/dpm/.bazaar
  Bazaar log file: /home/dpm/.bzr.log

David Planella (dpm) on 2011-03-01
description: updated
description: updated
Changed in launchpad:
status: New → Triaged
importance: Undecided → High
Robert Collins (lifeless) wrote :

Getting timeouts from people.canonical.com to bazaar.launchpad.net is extremely worrying, and a timeout here is a regression from our normal reliability.

Changed in launchpad:
importance: High → Critical
tags: added: regression
John A Meinel (jameinel) wrote :

I was told that the forking service was started on production, but the configuration was not enabled. If the configuration *was* enabled, that might explain this (since it is what happened when we rolled it out initially). While I doubt this is the actual problem, I figured we can at least have someone check the config in production to verify.

John A Meinel (jameinel) wrote :

Note that I can download the data from my home machine without a problem. However doing:
jameinel@lillypilly:~$ echo hello | ssh <email address hidden> -v bzr serve --inet --directory=/ --allow-writes

gives
OpenSSH_5.3p1 Debian-3ubuntu3.1.IS.10.04, OpenSSL 0.9.8k 25 Mar 2009
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to bazaar.launchpad.net [91.189.90.11] port 22.
debug1: connect to address 91.189.90.11 port 22: Connection timed out
ssh: connect to host bazaar.launchpad.net port 22: Connection timed out

This looks like a routing/firewall issue, not a bzr or Launchpad configuration issue.

summary: - Bzr timeouts on SSH connections
+ Bzr timeouts on SSH connections from people.canonical.com
Martin Pool (mbp) wrote :

Did the start of this problem coincide with the introduction of haproxy?
Is it possible that it's doing something strange that's causing the port to sometimes not be listening, or perhaps only on a specific ip that's not reached by the route from lillypilly?

Martin Pool (mbp) wrote :

rt 44341

Robert Collins (lifeless) wrote :

Ok, we're satisfied this isn't a code issue - the RT ticket will address the firewall in the datacentre. Until then you may be able to use http:// bazaar.launchpad.net instead.

Changed in launchpad:
status: Triaged → Invalid
Martin Pool (mbp) wrote :

For the benefit of the bug, no, haproxy isn't live yet so this probably is unrelated.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers