bzr fails, perhaps account is locked

[Alexey Bychko]

since yesterday we're not able to use our dev bot account for bazaar. we can't do "bzr branch" in our CI system, manual repo fetch doesn't work also.

all info on this is short failure message:

-bash-4.2$ bzr branch lp:~sergei.glushchenko/percona-xtrabackup/2.1-ST43302-xb-bug1340717
ssh: connect to host bazaar.launchpad.net port 22: Connection refused
ConnectionReset reading response for 'BzrDir.open_2.1', retrying
ssh: connect to host bazaar.launchpad.net port 22: Connection refused
bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.

but it works perfectly in the same time on the same servers with other accounts

[Alexey Bychko (abychko)]

our account is lp-dev-merge-bot, a member of persona-core group

[Roel Van de Paar (roel11)]

Any help you can give us here at Percona would be very appreciated!

[William Grant (wgrant)]

It sounds like that server is behind a firewall that's rejecting SSH connections to bazaar.launchpad.net. It's not blocked at our end.

[William Grant (wgrant)]

Ah, I'm mistaken: we blocked that host over the weekend, as it had more than 50 connections to bazaar.launchpad.net open simultaneously. Do you know what would have been making such a large number of connections?

[Alexey Bychko (abychko)]

William, we have /24 internal network with build hosts NATed to the single IP. that is the reason for more than 50 connections

[William Grant (wgrant)]

I've unblocked the host. Sorry about that.

[Alexey Bychko (abychko)]

we're working on minimizing checkouts, some build jobs are really large (for example 160 checkouts)

[Alexey Bychko (abychko)]

it works, thank you for your help!

[Roel Van de Paar (roel11)]

William; thank you

[Alexey Bychko (abychko)]

William, we're not able to work today. seems our account is blocked again?

[William Grant (wgrant)]

Yes, it looks like ~lp-dev-merge-bot (is that your Jenkins?) had 50 connections open (the maximum we allow) each using several gigabytes of RAM. Can you convince it to not make so many simultaneous connections, or work out what it's doing to use so much RAM?