Can't ssh into lxd container using a proxy

[Marc André Audet]

When I try to connect to an LXD container on an AWS EC2 machine with the command:

juju --debug ssh --proxy 0/lxd/4

This is the result:

INFO juju.cmd supercommand.go:56 running juju [2.3.8 gc go1.10.2]
DEBUG juju.cmd supercommand.go:57 args: []string{"juju", "--debug", "ssh", "--proxy", "0/lxd/4"}
INFO juju.juju api.go:67 connecting to API addresses: [REDACTED:17070 REDACTED:17070 REDACTED:17070 REDACTED:17070 REDACTED:17070 REDACTED:17070 REDACTED:17070 REDACTED:17070 REDACTED:17070]
DEBUG juju.api apiclient.go:843 successfully dialed "wss://REDACTED:17070/model/REDACTED/api"
INFO juju.api apiclient.go:597 connection established to "wss://REDACTED:17070/model/REDACTED/api"
DEBUG juju.cmd.juju.commands ssh_common.go:338 proxy-ssh enabled so not doing reachability scan
DEBUG juju.cmd.juju.commands ssh_common.go:380 using target "0/lxd/4" address "252.19.213.206"
DEBUG juju.utils.ssh ssh.go:305 using OpenSSH ssh client
ssh_exchange_identification: Connection closed by remote host
DEBUG juju.api monitor.go:35 RPC connection died
INFO cmd supercommand.go:465 command finished

Every configurations are using defaults and this is no a new controller with Juju 2.3.8.

I'm not 100% sure about the internals or if I'm supposed to do some configurations manually in order to make connections to LXD containers work.

Connection to the AWS EC2 machine works flawlessly with juju ssh 0.

[Alvaro Uria (aluria)]

I'm running into the same issue, using the Juju openstack provider (Juju 2.7.2 on bionic)
* juju client perspective: https://pastebin.ubuntu.com/p/pJgZfwHr74/
* machine-0.log from controller: https://pastebin.ubuntu.com/p/2Jpcr3JmHg/

On a single machine,
1) I have deployed "openstack-on-lxd" with Juju LXD provider
2) Then bootstrapped a Juju openstack environment on top of the previously deployed openstack service

FWIW, I have created lxdbr1 on the bare metal and a static route to reach the Neutron network via the "provider-router" running in the neutron-gateway unit.

Any OpenStack instance that I launch is reachable, and it also works via "juju bootstrap" and "juju deploy" + "juju ssh".

However, if I deployed "juju deploy cs:ubuntu ubuntu-lxd --to lxd:1",
1) I can "juju run --unit ubuntu-lxd/0 hostname" (it runs through the controller
2) I can "juju ssh -m controller 0"
3) I can "juju ssh -m default ubuntu/0" (a nova instance managed by Juju)
4) I cannot "juju ssh -m default ubuntu-lxd/0" (when proxy-ssh=false)
5) I cannot "juju ssh -m default ubuntu-lxd/0" (when proxy-ssh=true)

fan-252 interface is in place on all the openstack machines, and I can reach via ping and "nc <lxd-fan-ip> 22" from the controller node.

[Alvaro Uria (aluria)]

BTW, I ran several tcpdumps and didn't find any trace of the "juju ssh" connections. The only tcp/22 data sniffed was the ssh conn to run tcpdump command itself. I only saw traffic to ws:// on tcp/17070 and traffic from the agents toward the controller.

[Richard Harding (rharding)]

Hmmm, I think the issue we'll find here is that the model has a fan network setup but the controller itself isn't participating in it. You might need to see the controller with the same fan config as the model when you bootstrap and set model config so that there's a known route from the controller you're ssh'ing through (with the --proxy) to the lxd container that you're trying to reach in another model. By default there's not a known route from the controller to each lxd instance.

Juju is working because the lxd units reach out to the controller which has a reachable address. Things going the other way around will have issues. Setting common fan config would give that path.

[Alvaro Uria (aluria)]

Hi Rick. This is a Juju openstack environment and the controller has a floating IP. Other than that, the underlay and overlay networks match per the info below:

$ juju model-config -m controller | egrep 'fan|use-'
container-networking-method model fan
fan-config model 10.200.0.0/24=252.0.0.0/8
use-default-secgroup controller true
use-floating-ip controller true
use-openstack-gbp default false

$ juju model-config -m default | egrep 'fan|use-'
container-networking-method model fan
fan-config model 10.200.0.0/24=252.0.0.0/8
use-default-secgroup controller true
use-floating-ip default false
use-openstack-gbp default false

$ juju models --debug
[...]
connecting to API addresses: [10.200.0.152:17070 10.10.0.178:17070 252.152.0.1:17070]
[...]

$ juju ssh -m controller 0 'ip -o -4 a s;fanctl show'
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
2: ens2 inet 10.200.0.152/24 brd 10.200.0.255 scope global dynamic ens2\ valid_lft 49097sec preferred_lft 49097sec
3: fan-252 inet 252.152.0.1/8 scope global fan-252\ valid_lft forever preferred_lft forever
Bridge Underlay Overlay Flags
fan-252 10.200.0.152/24 252.0.0.0/8 enable dhcp

$ juju ssh -m default 0 'ip -o -4 a s;fanctl show'
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
2: ens2 inet 10.200.0.129/24 brd 10.200.0.255 scope global dynamic ens2\ valid_lft 56736sec preferred_lft 56736sec
3: fan-252 inet 252.129.0.1/8 scope global fan-252\ valid_lft forever preferred_lft forever
5: lxdbr0 inet 10.250.154.1/24 scope global lxdbr0\ valid_lft forever preferred_lft forever
Bridge Underlay Overlay Flags
fan-252 10.200.0.129/24 252.0.0.0/8 enable dhcp

If I ssh into the controller and I ping the overlay IPs on the units in the default model (even the LXD inside a nova instance, which only has the overlay IP as main IP).

[Alvaro Uria (aluria)]

Any ETA on when this issue could be addressed?
* client1: runs juju commands
* controller1: runs in an openstack environment
* default-node1: runs in the default model, in the same openstack environment

I can reach controller1 and default-node1 via "juju ssh" (no proxying).

Now, if I deploy a LXD container inside default-node1, I can't reach it. Note:
1) fan network is in place in controller1, default-node1 and the LXD unit
2) From controller1, I can reach the LXD unit via the fan network
3) "juju ssh" against the LXD unit does not work because client1 is not in the fan network
4) "juju ssh --proxy" against the LXD unit, from client1, doesn't work either (and it should)

Also note that "juju run" from client1 against the LXD container does work, because the request goes through controller1 which does have visibility via the fan network.

"juju ssh --proxy" should work in the same way as "juju run", or use ProxyCommand to connect via controller1.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.