Record and expose SSH host keys

[Adam Collard]

It would be great if MAAS could assist me in identifying MITM attacks by allowing me to preseed my SSH known_hosts file with keys from the nodes that MAAS has deployed (and commissioned?).

Maybe through the use of ssh-keyscan or similar?

[Mike Pontillo (mpontillo)]

If an active man-in-the-middle attack is already taking place, ssh-keyscan as an end-user won't provide protection.

What you would expect is that, after a node is deployed, its host key won't change for the life of that deployment. The most useful thing I can think to do with ssh-keyscan is to invoke it (from its attached cluster), so we eliminate the risk of network layer attacks, and limit the attack surface to ARP/neighbor discovery poisoning (which could be prevented, to some extent, with managed switch features).

We could then make the expected public key available via the MAAS API, and then a script could be used to update a users' known_hosts file.

This isn't a seamless user experience, though, and we all know how rarely users actually check host keys. Will they bother to find out about the script and run it, to update their known_hosts?

A more user-friendly (albeit much more potentially dangerous) option would be for MAAS to escrow the key, and have curtin redeploy the same private host key to the node in subsequent deployments.

But how would we do that in a secure way? We don't want the MAAS database to contain private keys for the deployed node. That's a man-in-the-middle attack waiting to happen. I see a few options here:

 - Change curtin to shuffle data around on disk so that the private host keys are preserved from a previous deployment
 - Store the host keys inside some kind of hardware-specific NVRAM on the node (a hardware security device such as a TPM may be possible, but these devices have limited RAM and are more suited to holding keys - but it's not clear that the TPM state will survive a redeployment.)
 - Derive a key using information known only to the node, such as a value computed by a TPM, (or some other information not available in the MAAS database) - plus, possibly, a password entered by the user, or a random salt - to encrypt the private host key, and have MAAS escrow that.

All this sounds like a lot of trouble/risk for not much gain, so I'd suggest going with the simpler option of having the MAAS clusterd query nodes for their new public keys, and make that information available.

But I still think there may be some value in an escrow mechanism, just due to the fact that typical users are so negligent about SSH public key checking. Effectively, it may be more secure overall if the host key never changes. Assuming the MitM isn't already in progress. ;-)

[Mark Shuttleworth (sabdfl)]

A little digging in the SSH man pages turned up this:

     VerifyHostKeyDNS

Specifies whether to verify the remote key using DNS and SSHFP resource records. If this option is set to “yes”, the client will implicitly trust keys that match a secure fingerprint from DNS. Insecure fingerprints will be handled as if this option was set to “ask”. If this option is set to “ask”, information on fingerprint match will be displayed, but the user will still need to confirm new host keys according to the StrictHostKeyChecking option. The argument must be “yes”, “no”, or “ask”. The default is “no”. Note that this option applies to protocol version 2 only.

             See also VERIFYING HOST KEYS in ssh(1).

And correspondingly:

VERIFYING HOST KEYS

If the fingerprint is unknown, an alternative method of verification is available: SSH fingerprints verified by DNS. An additional resource record (RR), SSHFP, is added to a zonefile and the connecting client is able to match the fingerprint with that of the key presented.

In this example, we are connecting a client to a server, “host.example.com”. The SSHFP resource records should first be added to the zonefile for host.example.com:

           $ ssh-keygen -r host.example.com.

The output lines will have to be added to the zonefile. To check that the zone is answering fingerprint queries:

           $ dig -t SSHFP host.example.com

Finally the client connects:

           $ ssh -o "VerifyHostKeyDNS ask" host.example.com
           [...]
           Matching host key fingerprint found in DNS.
           Are you sure you want to continue connecting (yes/no)?

See the VerifyHostKeyDNS option in ssh_config(5) for more information.

[Mark Shuttleworth (sabdfl)]

So, in summary:

 * during or post-deployment, MAAS could do ssh-keygen -r <hostname>
 * the data from that goes into the relevant DNS zone file
 * the user .ssh/config includes:

Host *.maas.domain
      VerifyHostKeyDNS yes
      StrictHostKeyChecking no
      CheckHostIP no
      UserKnownHostsFile /dev/null

I think this will result in DNS-based key checking and no persistent key storage.

[Mike Pontillo (mpontillo)]

Interesting approach; I like it. But it seems that the prerequisite for enabling this should be requiring DNSSEC validation of the MAAS zone. Otherwise we may risk enabling a MitM rather than preventing it. (perhaps that's why it's not enabled by default.)

Ideally MAAS would be a [sub] CA, and would be able to sign nodes' public SSH keys to confirm or deny trust. In that situation, MAAS could sign host keys for new deployments, revoke obsolete keys when nodes are released, etc. (Too bad X.509 isn't widely used with SSH.)

[Mike Pontillo (mpontillo)]

For users who wish to trust their DNS to provide SSH keys, it might be nice to have a script and/or snippet in the UI that could be copy/pasted into .ssh/config with the appropriate values.

[Mike Pontillo (mpontillo)]

I experimented with this a little bit; some implementation notes:

 - See also: https://tools.ietf.org/html/rfc4255

 - "ssh-keygen -r <hostname>" grabs keys from the ~/.ssh/known_hosts file. (so ssh-keyscan - or another method of obtaining the public key - is still required)

 - "ssh-keygen -r <hostname> -f <filename>", as documented in the manual page, is *supposed* to read from <filename> as if it is a known_hosts file. However, (at least for me on Trusty) it seems to be broken. Only the default option of reading the key from the known_hosts file works. So we'd either need to rewrite the public-key-to-DNS conversion (have the Python code do that), or hack around it, such as by having a keyscan also populate known_hosts on the cluster.

[Andres Rodriguez (andreserl)]

Hi!

**This is an automated message**

We believe this is may no longer be an issue in the latest MAAS release. Due to the date of this bug report, we are marking this bug report as invalid. If you believe this is still an issue, please re-open this bug report.

Thanks

[Zack Williams (zdwzdw)]

I don't think the assessment above takes into account the extremely limited utility of a stolen host key.

If the MaaS server was to store (and possibly generate) host keys and there was a breach where this information was leaked to an attacker, the host keys are only useful if the attacker wanted to start up a new impersonating node using that same host key.

As successful impersonation would mean controlling the DNS (or IP address) of the impersonated node, at that point a stolen host key is no longer the most important security concern.

I would argue that the right implementation would be to *optionally* allow SSH host keys to be either autogenerated or added manually. This is already supported in cloud-init: http://cloudinit.readthedocs.io/en/latest/topics/examples.html#configure-instances-ssh-keys

The manual addition would allow users to set up an SSH CA or other secondary authentication system: https://www.digitalocean.com/community/tutorials/how-to-create-an-ssh-ca-to-validate-hosts-and-clients-with-ubuntu

[Jerzy Husakowski (jhusakowski)]

Feature request, please make sure it's captured on MAAS Discourse