juju 3.0.0 unable to bootstrap on fresh machine

[Caner Derici]

This recently popped up in our pylibjuju CI tests with juju --channel=latest/edge.

Problem is, after

`sudo snap install juju --channel latest/edge --classic`
Warning: flag --classic ignored for strictly confined snap juju

`juju bootstrap localhost test `

fails with:

ERROR cannot load ssh client keys: mkdir /home/jenkins/.local/share: permission denied

This is probably because the juju latest/edge is a strictly confined snap, it doesn't have the permissions to create (parent) directories. @jnsgruk reported on MM that things start to work after manually running `mkdir -p ~/.local/share`.

[Alberto Mardegan (mardy)]

Hi Caner, I created a branch in snapd which could fix the issue: https://github.com/snapcore/snapd/pull/12111

But I'm not sure it does, because it depends on the setup you have in the test machine (and how you create the jenkins user). Can you please check what is the output of the command

    ps -fe | grep userd

in the test machine?

I can also try to create a snapd build for you to test. Which binary architecture are you using?

[Caner Derici (cderici)]

Hi Alberto, thanks for looking into this!

So, I'm not entirely sure about how the jenkins user is set up initially. This is on an ephemeral node (focal, amd64) on aws running for a regular CI checks for a PR. Here's the whole output, might be helpful:

https://pastebin.canonical.com/p/nSjC7tvk2c/plain/

I went ahead and manually created an ephemeral node and ran the same CI job and I was able to recreate the problem. So I may be able to test it manually if you could make a snapd binary with the fix. I also checked the user jenkins on the same machine:

jenkins@ip-172-31-23-203:~$ ps -fe | grep userd
jenkins 11843 1409 0 17:34 pts/0 00:00:00 grep --color=auto userd

[John A Meinel (jameinel)]

My understanding is that the interface will be updated to create ~/.local/share for us if it doesn't exist, so that juju itself can create ~/.local/share/juju.

I think it makes sense to have the confinement such that juju itself cannot create ~/.local/share (it shouldn't be able to muck around in any other directories in that path).

The short term workaround is to add a "mkdir -p ~/.local/share" in your CI scripts, but that doesn't work very well for arbitrary end users.

[John A Meinel (jameinel)]

In *Juju* we could trap this error, and check if we are in a confined snap, and then prompt our users as a workaround, rather than just failing mysteriously on fresh installs.

[Alberto Mardegan (mardy)]

Sorry for not following up on this earlier! The approach I initially thought of might have been able to fix a few cases, but not all of them: it was relying on the userd session daemon to be running, but that is not a certain fact, since we want snaps to be working even in those cases where the used it not logged in a graphical session.

The solution would be to make "snap run ..." create the directories: the /usr/bin/snap program is not subject to an AppArmor confinement, so it could create the directories before invoking snap-confine.

[John A Meinel (jameinel)]

@mardy, sorry I missed your response. It is unclear to me whether you are saying that there should be something that the juju snap should be doing, or whether snapd itself should be doing this. I don't think we can inject a step before snap-confine, but if there is a way to do so, please let us know.

[Alberto Mardegan (mardy)]

Hi John, no, it was not a suggestion for you, but something for the snapd team. Sorry for not making this clear :-)

[Juan M. Tirado (tiradojm)]

I will set this as released because it is no longer impacting pylibjuju.

[John A Meinel (jameinel)]

I think there is still something that juju itself can do, so not Fix Released.

Specifically on a fresh install, we should recognize that we are in a confined snap and give the user a better error message, asking them to create the parent directory. Rather than just having mkdir fail and fall over.

[Eric Chen (eric-chen)]

Hi, May I confirm the current plan for this issue?
Will the snap create the folder or it should be handled by user?