lp-serve (or bzr) should release connections that do not authenticate quickly

[Gary Poster]

We had an incident in which codehosting connections spiked because of one person connecting 800+ times to our SSH servers, and then not authenticating and never letting go of the connection. https://wiki.canonical.com/IncidentReports/2011-12-15-LP-codehosting-connection-spike

Reviewing the logs by eye, it seems that connections are all authenticated within less than half a second. If lp-serve (or bzr?) were configured to release unauthenticated connections after a relatively short time (1 second might not be conservative enough, but 10 seconds seems like plenty) then this particular problem might not have happened.

Acting on this should of course include a more careful log analysis to determine maximum authentication times.

I'm marking this critical based on discussion on the team leads call, but it's worth noting that this only really helps with people making honest mistakes. if this were a malicious DoS, this would be very inadequate.

[Jelmer Vernooij (jelmer)]

On 12/21/2011 09:44 PM, Gary Poster wrote:
> Public bug reported:
>
> We had an incident in which codehosting connections spiked because of
> one person connecting 800+ times to our SSH servers, and then not
> authenticating and never letting go of the connection.
> https://wiki.canonical.com/IncidentReports/2011-12-15-LP-codehosting-
> connection-spike
>
> Reviewing the logs by eye, it seems that connections are all
> authenticated within less than half a second. If lp-serve (or bzr?)
> were configured to release unauthenticated connections after a
> relatively short time (1 second might not be conservative enough, but 10
> seconds seems like plenty) then this particular problem might not have
> happened.
>
> Acting on this should of course include a more careful log analysis to
> determine maximum authentication times.
>
> I'm marking this critical based on discussion on the team leads call,
> but it's worth noting that this only really helps with people making
> honest mistakes. if this were a malicious DoS, this would be very
> inadequate.
lp-serve isn't started until the SSH authentication succeeds. The ssh
server code (which runs lp-serve) is all in lib/lp/codehosting/sshserver/.

Cheers,

Jelmer

[methane (songofacandy)]

It may be a problem of Bazaar Explorer and TortoiseBZR.

Bazaar Explorer's checkout dialog uses lightweight checkout.
TortoiseBZR tries to open workingtree and get status of files.

If the user doesn't launch pageant, this cause authentication failed repeatedly.

[Jelmer Vernooij (jelmer)]

Isn't the connection properly closed if that happens though? And would this really keep open a couple of hundred connections?