juju does not extract system ssh fingerprints

[Kees Cook]

On first-time connect, the SSH host keys for machines are not known. This should be extracted from the system boot logs in EC2 and added to the user's known-hosts file automatically so that there is an end-to-end secure path to the EC2 instances, and they are not faced with:

$ juju bootstrap
2011-11-19 09:49:19,902 INFO Bootstrapping environment 'sample' (type: ec2)...
2011-11-19 09:49:22,348 INFO 'bootstrap' command finished successfully

$ juju status
2011-11-19 09:54:57,536 INFO Connecting to environment.
The authenticity of host 'ec2-50-112-8-6.us-west-2.compute.amazonaws.com (50.112.8.6)' can't be established.
ECDSA key fingerprint is 4c:2a:68:f2:33:50:f2:6f:81:bf:e3:f3:06:9d:23:70.
Are you sure you want to continue connecting (yes/no)?

[Kees Cook (kees)]

For example:

$ ec2-get-console-output --region us-west-2 i-103fe910 | grep -m1 -A3 FINGERPRINTS
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 2048 f8:bf:48:3a:a8:93:e1:1f:52:6d:08:ff:0a:a2:3d:eb /etc/ssh/ssh_host_rsa_key.pub (RSA)
ec2: 1024 1d:a6:d9:89:6f:7b:0a:a2:45:78:d4:ee:a8:23:c0:5c /etc/ssh/ssh_host_dsa_key.pub (DSA)

[Clint Byrum (clint-fewbar)]

Agreed Kees, this is related to bug #802117 which suggests that juju should use a different known_hosts file. We've also discussed generating a key for the host and then injecting it via userdata so that we don't have to wait for the 4+ minute lag time of the EC2 console. That would also help us handle the bare metal case where we don't necessarily have access to the console programmatically.

[Kees Cook (kees)]

Yeah. I'd be great to have some sort of "waitfor" subcommand, and it could wait for various status changes (unit or service finished installing/starting/upgrading, SSH connectivity, etc).

Right now, I'm using stuff from http://ubuntu-smoser.blogspot.com/2010/07/verify-ssh-keys-on-ec2-instances.html

It need to be much more robust, but this can give you a picture:

#!/bin/bash
set -e

INSTANCE="$1"
HOST="$2"
REGION="$3"
if [ -z "$REGION" ]; then
    REGION=us-west-1
fi

FINGERPRINT=""
while [ -z "$FINGERPRINT" ]; do
    FINGERPRINT=$(ec2-get-console-output --region $REGION $INSTANCE | grep -A2 ecdsa | tail -n1 | cut -d" " -f1,2)
    sleep 10
done
known_length=$(echo "$FINGERPRINT" | cut -d" " -f1)
known_fp=$(echo "$FINGERPRINT" | cut -d" " -f2)

pubkey=$(mktemp -t pubkey-XXXXXX)
fprint=$(mktemp -t fprint-XXXXXX)
ssh-keyscan $HOST > $pubkey
ssh-keygen -lk $pubkey > $fprint
read length fp hostname id < $fprint
if [ "x$length" != "x$known_length"] || [ "x$fp" != "x$known_fp" ]; then
    exit 1
fi

ssh-keygen -R "$HOST"
ssh-keygen -H -f $pubkey
cat $pubkey >> ~/.ssh/known_hosts
shred -u $pubkey $fprint

[Jim Baker (jimbaker)]

Support for managing known_hosts for an environment, including public keys of new machines, is being done with respect to bug 802117. So this should resolve this issue too.

Re "wait-for" functionality, this is mostly addressed by bug 849071, a branch for which is currently under review.

It would be desirable if juju status in particular could support some sort of long poll mechanism that would return upon a change to juju status. This really needs bug 843539 completed however.

[Scott Moser (smoser)]

just an fyi, kirkland in bikeshed has 'cloud-sandbox' which does much of what you're talking about.
 http://bazaar.launchpad.net/~bikeshed/bikeshed/trunk/view/head:/cloud-sandbox

It injects keys and then generates new ones as the ones injected are protected as they're plain text in user-data.

JuJu can actually do much better than this on any instance other than the bootstap node, as it could feed one-time use long urls to cloud-init as a '#include-once'.

[Kapil Thangavelu (hazmat)]

Nutshell on ssh fingerprint security..

- keep env specific known hosts file for use of juju ssh/scp
- for bootstrap, juju generates the host key and passes in via userdata (much the same way we do tls certs), client records this locally to env hosts file
- on non bootstrap machines record to state server their auto-gen'd sshd fingerprints when connecting.
- client access to non bootstrap machine, lookups fingerprint and records it to env known hosts file before launching ssh/scp with params.

re why not do the gen on non-bootstrap machines, its simpler to preserve the goal/move to multiple machine provisioning with a single api call which would have shared userdata.

[Andrew Wilkins (axwalk)]

hazmat: I have had essentially the same thoughts, but we also need to regenerate the ssh keys for the bootstrap host after bootstrap, otherwise units (or units in containers) on machine 0 can access the keys via the userdata/metadata IP (ec2, openstack?). It's not an issue for the certs, because they're not actually sent via userdata but instead via the SSH part of bootstrap.

[Kapil Thangavelu (hazmat)]

hi andrew, that's reasonable, re key regen, we can even trigger it at end
of sync bootstrap, and record to api state, so we pick it up on the client
same as non bootstrap servers, ie always contact api servers for unknown
fingerprint.

On Tue, Feb 25, 2014 at 5:08 PM, Andrew Wilkins <
<email address hidden>> wrote:

> hazmat: I have had essentially the same thoughts, but we also need to
> regenerate the ssh keys for the bootstrap host after bootstrap,
> otherwise units (or units in containers) on machine 0 can access the
> keys via the userdata/metadata IP (ec2, openstack?). It's not an issue
> for the certs, because they're not actually sent via userdata but
> instead via the SSH part of bootstrap.
>
> --
> You received this bug notification because you are subscribed to pyjuju.
> https://bugs.launchpad.net/bugs/892552
>
> Title:
> juju does not extract system ssh fingerprints
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/892552/+subscriptions
>

[Eric Snow (ericsnowcurrently)]

This seems like a new feature. Shouldn't this be targeting master (currently 1.26)?

Also, we should verify that this is still relevant. We've touched related code for other things and the treatment of SSH fingerprints might have changed at this point.

[Aaron Bentley (abentley)]

The symptom has changed. Now it says "Attempting to connect to 172.31.13.247:22
Warning: Permanently added '52.5.235.80' (ECDSA) to the list of known hosts."

This is because Juju is disabling host verification by messing with the known-hosts file. Host verification should not be disabled.

[Menno Finlay-Smits (menno.smits)]

Fixed as part of the work on bug 1456916