container addressability Can't juju ssh to lxd-placed container

[Casey Marshall]

If I deploy into a container of a machine, I can't `juju ssh` into it.

Steps to reproduce:

1. Deploy some stuff. In my case, I deployed ceph on openstack.
2. juju deploy ubuntu --series trusty --to lxc:3

Then I get:

[Units]
ID WORKLOAD-STATUS JUJU-STATUS VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE
ceph/1 unknown idle 2.0-beta7.1 1 10.55.32.62
ceph/2 unknown idle 2.0-beta7.1 2 10.55.32.72
ceph/3 unknown idle 2.0-beta7.1 3 10.55.32.85
demo-client/2 unknown idle 2.0-beta7.1 3/lxd/0 10.0.4.147

3. Can't ssh in to the container

$ juju ssh demo-client/2
ssh: connect to host 10.0.4.147 port 22: Connection timed out

I *think* I remember being able to do this with lxc: placed units in juju 1.x.

See this to track the recent form of the regression
   http://reports.vapour.ws/releases/issue/5707efa7749a5658800b3243

[James Tunnicliffe (dooferlad)]

Just deployed OpenStack and "juju ssh ceph-mon/0" worked fine. It was created on 1/lxc/0 (this is a xenial machine, so it is a lxd managed lxc).

Tested with master + some routing fixes. I don't think the routing fixes will have had any impact though because they only fixed bonded interfaces.

[Casey Marshall (cmars)]

I think I mis-typed the deploy command above, sorry, it must have been `--to lxd:3`. So maybe it's only affecting LXD, not LXC placement?

Should I avoid the use of `--to lxd:<machine number>`?

I'm trying a container deployed with `--to lxc:3`.

[Cheryl Jennings (cherylj)]

Casey - if you're not using the MAAS provider, you will need to specify --proxy when using juju ssh to get to the containers. There was a change in beta6(maybe 5?) where it is no longer the default to proxy through the state server.

[Nate Finch (natefinch)]

This used to Just Work. Now it doesn't. That's bad. Having to specify your own proxy manually is really ugly. The whole point of juju ssh is to jump through hoops for you.

I'm not an expert in SSH, but it seems like, if I can ssh to the host machine, we should be able to set up the proxy automatically from the host to the container on that host.

At the very least, the error message is really bad:

$ juju ssh 1/lxc/0
ssh: connect to host 10.0.3.88 port 22: No route to host

If we explicitly aren't allowing people to ssh to the container, say so. This just looks like a bug, like we're giving SSH the wrong IP to connect to.

[James Tunnicliffe (dooferlad)]

Agreed. Why did we change this?

[Cheryl Jennings (cherylj)]

See bug #1566237 for discussion on the change.

I think it's a fair point to proxy through the host for containers by default.

[Richard Harding (rharding)]

The issue is that just because you can ssh to the container through the machine doesn't mean you should proxy through the controller. We need to limit folks that can access the controller directly like this. I think the longer term path is to find a way to enable proxying through the host machine in a lxd case.

I do think we can do a few things to help:

1) if the user ssh'ing is a controller admin, then proxy for them. It's ok, they could ssh there anyway

2) if the user is not and is a model user only, suggest they ssh directly to the host machine and then suggest ssh'ing to the container from there. I think we can provide these instructions without too much pain. This would only be required if the --proxy is indeed set to false for the controller.

[Anastasia (anastasia-macmood)]

I think that this has been fixed - last failure on master is seen on Apr 10 2016.

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]

[Stuart Bishop (stub)]

I just deployed kubernetes-core to aws, which places the easyrsa unit into an lxd container:

easyrsa/0* active idle 0/lxd/0 10.0.0.206 Certificate Authority connected.

juju ssh fails for this unit:

$ juju ssh easyrsa/0
ERROR cannot connect to any address: [10.0.0.206:22]

juju ssh works fine for all the non-lxd contained units.

[Stuart Bishop (stub)]

(the above is Juju 2.0.1, after bootstrapping an aws model and running 'juju deploy kubernetes-core')

[Reed O'Brien (reedobrien)]

That should be
$ juju ssh 0/lxd/0

not

$ juju ssh easyrsa/0

easyrsa/0 is the name of the unit, 0/lxd/0 is the name of the machine. Please confirm you can ssh to the machine rather than the unit.

Thanks.

[Stuart Bishop (stub)]

From the docs:

"The machine is identified by the argument which is either a 'unit name' or a 'machine id'."

It is expected by users that the unit name works here, and it is trivial for Juju to map the unit name to the machine if that is what it needs internally.

[Anastasia (anastasia-macmood)]

Related to bug # 1569106

[Kevin W Monroe (kwmonroe)]

@reedobrien et al, deploying to AWS with --to lxd placement, I cannot ssh to the unit name nor the machine.

$ juju version
2.1.2-xenial-amd64

Repro:

$ juju deploy ubuntu u1
$ juju deploy ubuntu uc1 --to lxd:0

$ juju status
Model Controller Cloud/Region Version
l1 aws-e aws/us-east-1 2.1.2

App Version Status Scale Charm Store Rev OS Notes
u1 16.04 active 1 ubuntu jujucharms 10 ubuntu
uc1 16.04 active 1 ubuntu jujucharms 10 ubuntu

Unit Workload Agent Machine Public address Ports Message
u1/0* active idle 0 52.205.139.88 ready
uc1/0* active idle 0/lxd/0 10.0.22.39 ready

Machine State DNS Inst id Series AZ
0 started 52.205.139.88 i-0608bcd8aa65ec94b xenial us-east-1a
0/lxd/0 started 10.0.22.39 juju-93ecfe-0-lxd-0 xenial

$ juju ssh 0/lxd/0
ERROR cannot connect to any address: [10.0.22.39:22]
$ juju ssh uc1/0
ERROR cannot connect to any address: [10.0.22.39:22]

[John A Meinel (jameinel)]

the machine is not externally routable, thus you aren't able to ssh to it. (the address of the machine is explicitly in a local-only network that is only valid from the host machine.)

we could proxy/use the host machine as a jump host. I think ideally we would try to make it so that containers can get addresses from the provider in their hosts network. This is quite provider specific, and often requires setting flags that enable/disable security settings (allow traffic to the machine for mac addresses that the provider didn't provision, etc.)

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 5 years, so we're marking it Expired. If you believe this is incorrect, please update the status.