`juju bootstrap|ssh` fails if `~/.ssh/config` uses control sockets

[Simon Déziel]

I cannot bootstrap a controller on a remote LXD server because Juju trips on SSH config options not compatible with the snap strict confinement.

$ juju bootstrap --debug juju-lxd overlord
...
20:09:43 INFO cmd bootstrap.go:627 Installing Juju agent on bootstrap instance
20:09:44 DEBUG juju.cloudconfig.instancecfg instancecfg.go:945 Setting numa ctl preference to false
Waiting for address
Attempting to connect to 10.76.66.92:22
Attempting to connect to fd42:b2a9:e8fd:1156:216:3eff:fe43:8a1:22
20:09:54 DEBUG juju.provider.common bootstrap.go:669 connection attempt for 10.76.66.92 failed: unix_listener: cannot bind to path /run/user/1000/ssh-ubuntu@10.76.66.92:22.sock.3u8OEGTRTRgG2KFg: Permission denied
20:09:54 DEBUG juju.provider.common bootstrap.go:669 connection attempt for fd42:b2a9:e8fd:1156:216:3eff:fe43:8a1 failed: unix_listener: cannot bind to path /run/user/1000/ssh-ubuntu@fd42:b2a9:e8fd:1156:216:3eff:fe43:8a1:22.sock.GxmLW5KDOrbi8NGn: Permission denied
...

Accompanying Apparmor denials:

Feb 22 20:10:20 sdeziel-lemur kernel: audit: type=1400 audit(1677114620.656:6111): apparmor="DENIED" operation="connect" class="file" profile="snap.juju.juju" name="/run/user/1000/openssh_agent" pid=3286411 comm="ssh" requested_mask="wr" denied_mask="wr" fsuid=1000 ouid=1000
Feb 22 20:10:20 sdeziel-lemur kernel: audit: type=1400 audit(1677114620.660:6112): apparmor="DENIED" operation="mknod" class="file" profile="snap.juju.juju" name="/run/user/1000/ssh-ubuntu@10.76.66.92:22.sock.n2KOi7bol7kcGm1F" pid=3286410 comm="ssh" requested_mask="c" denied_mask="c" fsuid=1000 ouid=1000
Feb 22 20:10:20 sdeziel-lemur kernel: audit: type=1400 audit(1677114620.668:6113): apparmor="DENIED" operation="mknod" class="file" profile="snap.juju.juju" name="/run/user/1000/ssh-ubuntu@fd42:b2a9:e8fd:1156:216:3eff:fe43:8a1:22.sock.G0DgptEIgohkbxar" pid=3286411 comm="ssh" requested_mask="c" denied_mask="c" fsuid=1000 ouid=1000

In the above, we see that Juju is denied access to the SSH agent. It also tries to create a socket according to the ControlPath directive in ~/.ssh/config:

$ grep -E 'Agent|Control' ~/.ssh/config
  ControlPersist 120s
  ControlMaster auto
  ControlPath /run/user/%i/ssh-%r@%h:%p.sock
  ControlPersist 1s
  AddKeysToAgent yes
  ForwardAgent no

Juju should probably not rely as much on the user's config.

Additional information:

$ snap list juju snapd
Name Version Rev Tracking Publisher Notes
juju 3.1.0 22136 3.1/stable canonical✓ -
snapd 2.58.2 18357 latest/stable canonical✓ snapd

$ uname -a
Linux sdeziel-lemur 5.19.0-32-generic #33~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Jan 30 17:03:34 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

[Ian Booth (wallyworld)]

The only line in the ssh config that the strict juju snap is unhappy with is the control path

ControlPath /run/user/%i/ssh-%r@%h:%p.sock

which points to a directory that is not accessible from a strict snap.

The fix here is for juju to add a ControlPath override to somewhere under ~/.local/share/juju when invoking ssh.

[Simon Déziel (sdeziel)]

Thank Ian!

I think the `AddKeysToAgent yes` is responsible for the attempt to connect to my local agent, what do you think?

[Ian Booth (wallyworld)]

I tested with 'AddKeysToAgent yes' with the strict snap and all I needed to change was the ControlPath in order for it to work. But juju doesn't strictly need this so we could also override that to No as well.

[Simon Déziel (sdeziel)]

The agent thing doesn't seem to be blocking, merely generating noise. I forgot to mention but I have this which I assume isn't stripped when the snap executes:

$ env | grep SSH
SSH_AGENT_LAUNCHER=openssh
SSH_AUTH_SOCK=/run/user/1000/openssh_agent

[John A Meinel (jameinel)]

When running out of a strictly confined snap, we won't be able to access arbitrary directories on your filesystem.

I'm against having *juju* rewrite any of these rules, as it really goes against the idea of ControlPath. The whole *point* of ControlPath is that you share connections to a given host, and if we rewrote where those sockets are, then you wouldn't be sharing connections.

I'm a little surprised that the default is '/run/user' and not '$HOME/.ssh/' somewhere.

If this really is a standard pattern, then like Juju has the $HOME plug, we might want a /run/user plug for our snap (but so would many other applications).

Maybe that is a general consideration for the equivalent HOME snap slot?

[John A Meinel (jameinel)]

I feel like we definitely don't want to be rewriting an SSH_CONFIG file, but we might want to add some obvious interactions like '/run/user' if that is a generic standard that people make a lot of use of. (When looking online about '/run/user' and snaps, I see a bunch of things like the QT file chooser wanting to use it, etc)

[Simon Déziel (sdeziel)]

The user is free to save the ControlSocket anywhere they want, I picked /run/user/$UID as that's a per-user tmpfs. Somewhere in '~/.ssh/' is probably (more) common but since this is one feature depending on user-defined filesystem object, I feel this should be taken into account.

I don't think a SSH_CONFIG is required. How about just calling `ssh -oControlMaster=no`?

[Ian Booth (wallyworld)]

@jam, I wasn't advocating we rewrite SSH_CONFIG at all - I think Juju could or should simply use a -o option when it invokes ssh. In my view, juju's use of ssh is an "internal detail" of how juju operates and is not necessarily tied to the user's general ssh config. Juju should be free to invoke ssh with whatever -o options it wants or needs.

[Nobuto Murata (nobuto)]

It doesn't work even the socket is under ~/.ssh so I filed a snapd issue:
https://bugs.launchpad.net/snapd/+bug/2031489

[Camille Rodriguez (camille.rodriguez)]

workaround if you get blocked on this issue :
add the following under ~/.ssh/ssh_config:

ControlPath ~/ssh-%r@%h:%p.sock