support newer mac algorithms in "juju ssh" reachability checks

[Paul Collins]

We've been rolling out stricter baseline sshd configurations. In particular we've selected the following MAC algorithms on the basis that they are good choices that are also supported by openssh as far back as Ubuntu 14.04 LTS (trusty):

<email address hidden>,<email address hidden>,<email address hidden>

However I recently discovered while finalizing a recent new deployment that "juju ssh" apparently doesn't support any of these algorithms (at least as of 2.1.2) when checking reachability. In our case, when reachability succeeds, the openssh client is used to make the actual connection, which supports the restricted MAC configuration.

04:21:28 INFO juju.cmd supercommand.go:63 running juju [2.1.2 gc go1.6]
[...]
04:21:28 DEBUG juju.network.ssh reachable.go:175 ssh: handshake failed: ssh: no common algorithm for client to server MAC; client offered: [hmac-sha2-256 hmac-sha1 hmac-sha1-96], server offered: [<email address hidden> <email address hidden> <email address hidden>]
04:21:28 DEBUG juju.network.ssh reachable.go:175 ssh: handshake failed: ssh: no common algorithm for client to server MAC; client offered: [hmac-sha2-256 hmac-sha1 hmac-sha1-96], server offered: [<email address hidden> <email address hidden> <email address hidden>]
04:21:28 DEBUG juju.cmd.juju.commands ssh_common.go:366 getting target "vault/0" address(es) failed: cannot connect to any address: [10.22.96.40:22 91.189.89.250:22 10.22.96.40:22] (retrying)
[...]

Please consider upgrading juju ssh's MAC algorithm suite.

We've used https://github.com/arthepsy/ssh-audit as a starting point for our current configuration, and here are our notes on Ubuntu release support: http://pastebin.ubuntu.com/24741456/

[Paul Collins (pjdc)]

Possible solutions could be to simplify the reachability check to succeed when an SSH banner is seen, or even to succeed merely if the TCP connection is successfully established.

[Haw Loeung (hloeung)]

Seems it used to just check if a TCP connection succeeded but was changed in commit 37034d884bd7b7aca7f94bb5684ce813a51c3be1 to do more than that.

FWIW, an updated version of Golang crypto SSH library includes <email address hidden>[1] but I think we should revert back to a simple TCP connect (or even SSH banner) as Paul suggests.

[1]https://github.com/golang/go/issues/17676

[John A Meinel (jameinel)]

A simple TCP connect is not sufficient. In many of the places that interact
with Canonical and MAAS installs they actually have overlapping IP address
ranges. MAAS machines are configured with a "private" subnet that is
10.X... which is *not* routable from the driving machine, but the machine
is also attached to the Canonical VPN which *also* has a 10.X... subnet
that *is* routable from the driving machine. Imagine the topology:

My laptop: 55.55.55.55
MAAS Controller: 10.100.0.1 and private address 10.200.0.1
MAAS Machine 1: NIC 1, 10.200.0.2, NIC 2: 10.250.0.2
MAAS Machine 2: NIC 1, 10.200.0.3, NIC 2: 10.250.0.3
Random Canonical Machine: 10.250.0.3

NIC 2 of both MAAS machines are on a private subnet that does not have a
gateway to the outside world. But they *are* reachable from their
10.200.0.* addresses.

If you're on the Canonical VPN, you can 'ping' and even 'ssh connect' to
10.250.0.3 but that happens to be something like a Canonical internal
webserver.
So when trying to connect from your laptop via the Canonical VPN to your
MAAS machine, you need to actively connect to 10.100.0.1 to talk to MAAS,
and then SSH to 10.200.0.3 but *not* to 10.250.0.3 which with your route
tables is not the machine you think it is.

SSH banner is not sufficient. We need to see the public keys that the host
is giving us match the public keys of the machine as registered as part of
setting up the machine. (Else you're actually connecting to the wrong
machine.)
Note also that 10.250.0.3 is not universally the wrong address. Imagine you
are sshing from machine 1 to machine 2, it is entirely plausible that there
is a firewall preventing internal traffic from routing to the 10.200.0.3
address, and you should be using 10.250.0.3 (we've seen this in places like
Comcast where "public" addresses are reachable from outside, but *not*
internally.)

It's certainly possible to get an update for more protocols, or possibly
find a way to get the public keys as long as it is one of the registered
keys rather than a protocol that the library supports? We could explore
that. However, there are times where we explicitly depend on being able to
use gocrypto to SSH to machines (from the windows client, for example).
We've also talked about preferring the internal SSH implementation for
consistency (it would also let us skip some of the round trips, and more
tightly control the "known hosts" configuration.)

On Fri, Jun 2, 2017 at 10:07 AM, Haw Loeung <email address hidden>
wrote:

> Seems it used to just check if a TCP connection succeeded but was
> changed in commit 37034d884bd7b7aca7f94bb5684ce813a51c3be1 to do more
> than that.
>
> FWIW, an updated version of Golang crypto SSH library includes hmac-
> <email address hidden>[1] but I think we should revert back to a
> simple TCP connect (or even SSH banner) as Paul suggests.
>
> [1]https://github.com/golang/go/issues/17676
>
> ** Bug watch added: github.com/golang/go/issues #17676
> https://github.com/golang/go/issues/17676
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1695098
>
> Title:
> support newer mac a...

[Haw Loeung (hloeung)]

On Fri, Jun 02, 2017 at 07:35:11AM -0000, John A Meinel wrote:
> A simple TCP connect is not sufficient. In many of the places that interact
> with Canonical and MAAS installs they actually have overlapping IP address
> ranges. MAAS machines are configured with a "private" subnet that is
> 10.X... which is *not* routable from the driving machine, but the machine
> is also attached to the Canonical VPN which *also* has a 10.X... subnet
> that *is* routable from the driving machine. Imagine the topology:

Sounds like over-engineering here to me.

> If you're on the Canonical VPN, you can 'ping' and even 'ssh connect' to
> 10.250.0.3 but that happens to be something like a Canonical internal
> webserver.
>
> So when trying to connect from your laptop via the Canonical VPN to your
> MAAS machine, you need to actively connect to 10.100.0.1 to talk to MAAS,
> and then SSH to 10.200.0.3 but *not* to 10.250.0.3 which with your route
> tables is not the machine you think it is.

So you'll juju deploy a charm/service/application, say apache. But
there's no way to test from said "driving machine". Well, you'll try
and maybe it'll work because you're testing HTTP/HTTPS against some
other server via the VPN. Should your browser also be engineered to
knock first and do various fingerprint checks to ensure the webserver
is the one you've just deployed via a charm?

So juju ssh/scp works, but nothing else. Juju is really just masking
the problem here, and that's a network problem. This should really be
a user issue to me. Two things:

* the Canonical company VPN has netblocks *reserved* for users to
  use. No routes will ever be pushed for those so you can be sure
  there's no overlaps.

* there's route weights/metrics one could modify to say which path
  should be preferred. All routes pushed via the Canonical company VPN
  has weights set quite high so local routes should be preferred.

> We've also talked about preferring the internal SSH implementation for
> consistency (it would also let us skip some of the round trips, and more
> tightly control the "known hosts" configuration.)
>

Right, reducing the number of code paths makes sense.

It certainly did puzzle us when we dug into the code and thought "hang
on, shouldn't it be using OpenSSH first before falling back to Go's
crypto SSH? Oh, this is in 'reachable'" heh.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 5 years, so we're marking it Expired. If you believe this is incorrect, please update the status.