lp-serve (or bzr) should release connections that do not authenticate quickly

Bug #907515 reported by Gary Poster on 2011-12-21
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Launchpad itself
Critical
Unassigned

Bug Description

We had an incident in which codehosting connections spiked because of one person connecting 800+ times to our SSH servers, and then not authenticating and never letting go of the connection. https://wiki.canonical.com/IncidentReports/2011-12-15-LP-codehosting-connection-spike

Reviewing the logs by eye, it seems that connections are all authenticated within less than half a second. If lp-serve (or bzr?) were configured to release unauthenticated connections after a relatively short time (1 second might not be conservative enough, but 10 seconds seems like plenty) then this particular problem might not have happened.

Acting on this should of course include a more careful log analysis to determine maximum authentication times.

I'm marking this critical based on discussion on the team leads call, but it's worth noting that this only really helps with people making honest mistakes. if this were a malicious DoS, this would be very inadequate.

On 12/21/2011 09:44 PM, Gary Poster wrote:
> Public bug reported:
>
> We had an incident in which codehosting connections spiked because of
> one person connecting 800+ times to our SSH servers, and then not
> authenticating and never letting go of the connection.
> https://wiki.canonical.com/IncidentReports/2011-12-15-LP-codehosting-
> connection-spike
>
> Reviewing the logs by eye, it seems that connections are all
> authenticated within less than half a second. If lp-serve (or bzr?)
> were configured to release unauthenticated connections after a
> relatively short time (1 second might not be conservative enough, but 10
> seconds seems like plenty) then this particular problem might not have
> happened.
>
> Acting on this should of course include a more careful log analysis to
> determine maximum authentication times.
>
> I'm marking this critical based on discussion on the team leads call,
> but it's worth noting that this only really helps with people making
> honest mistakes. if this were a malicious DoS, this would be very
> inadequate.
lp-serve isn't started until the SSH authentication succeeds. The ssh
server code (which runs lp-serve) is all in lib/lp/codehosting/sshserver/.

Cheers,

Jelmer

INADA Naoki (songofacandy) wrote :

It may be a problem of Bazaar Explorer and TortoiseBZR.

Bazaar Explorer's checkout dialog uses lightweight checkout.
TortoiseBZR tries to open workingtree and get status of files.

If the user doesn't launch pageant, this cause authentication failed repeatedly.

Jelmer Vernooij (jelmer) wrote :

Isn't the connection properly closed if that happens though? And would this really keep open a couple of hundred connections?

William Grant (wgrant) on 2012-10-19
tags: added: codehosting-ssh
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers