bazaar.launchpad.net host key is 1024-bit RSA

[Colin Watson]

We should rotate this to something stronger.

General notes on the software involved: the server is not OpenSSH, but a custom server based on Twisted Conch (the parts common to all of Launchpad's SSH endpoints are in the lazr.sshserver package). Perhaps unfortunately, our logging doesn't record client banners, but it's probably reasonable to expect that the majority of clients are OpenSSH since that's the default configuration on Ubuntu; that said, Bazaar does support a few other SSH clients, including Plink and Paramiko.

The main difficulty is that the SSH ecosystem doesn't have robust host key rotation support. There's a protocol extension implemented by the OpenSSH server (though not Twisted or lazr.sshserver; filed as https://bugs.launchpad.net/lazr.sshserver/+bug/1811363) that allows for it, but it has to be explicitly requested by setting the UpdateHostKeys client option (https://bugzilla.mindrot.org/show_bug.cgi?id=2894), so until such time as this is made the default only a tiny minority of users are likely to be using it. Giving all bazaar.launchpad.net users a scary host-key-changed prompt isn't acceptable either, partly because many users would interpret it quite reasonably as an indicator of an attack, and partly because it would require intervention in some automated setups.

I did some testing to see what options we have. Using an OpenSSH server, if you have two RSA host keys, then the first listed in sshd_config is preferred by clients that don't trust any host key for this server yet. However, OpenSSH clients that already trust the old host key only avoid the host-key-changed prompt if the old host key is listed first. There's therefore no way to do a reasonable rotation within the same key type given the current OpenSSH client (which is no doubt why the UpdateHostKeys mechanism was developed).

What we can do is use a different key type. If you have host keys of multiple types, then the client will pick the type it prefers (the OpenSSH client has a configurable preference list), and in that case the behaviour is as follows: clients that trust the old host key continue to do so; clients that don't trust any host key for this server yet pick the one of their preferred type; if UpdateHostKeys is set, then clients that trust the old host key add the new one to known_hosts as well. This is probably the best we can do given current OpenSSH, never mind whatever other clients do. https://bugzilla.mindrot.org/show_bug.cgi?id=2673 says that this is an artifact rather than a deliberate feature, but I'll take it.

This will require some work in lazr.sshserver to support other host key types (filed as https://bugs.launchpad.net/lazr.sshserver/+bug/1811359), followed by a small change in Launchpad to make use of it.

I would strongly prefer to get Ed25519 support into Twisted first; I've been working on this in spare moments for a while now. My reasoning for this is that the OpenSSH client prefers ECDSA host keys over Ed25519, perhaps for compatibility reasons of some kind, and so this would give us some additional key rotation opportunities using the same strategy. It seems silly to have to choose algorithms on this basis, but it's the best I can do at the moment.

(I also looked into host key certificates, but the client-side deployment mechanism seems pretty manual there; they appear to be more intended for use within an organisation where you can deploy /etc/ssh/ssh_known_hosts centrally with an appropriate @cert-authority line. I don't think they would work well for bazaar.launchpad.net.)

The alternative, of course, is to do a host key rollover that requires user intervention (at minimum, setting UpdateHostKeys=yes). I think it's likely that this will produce a prohibitive user support load, but it is a possibility if other options are exhausted.

[Colin Watson (cjwatson)]

Similar comments apply to git.launchpad.net's host key to some extent, although at least it's 2048-bit.

[Colin Watson (cjwatson)]

We should also offer at least one of the rsa-sha2-512 and rsa-sha2-256 host key algorithms rather than just ssh-rsa. This doesn't require changing the host key itself.

[Colin Watson (cjwatson)]

I believe that supporting rsa-sha2-* will require some work in Twisted; it appears that it only supports ssh-rsa for RSA keys today.