juju-local: systemd units fail on boot if /home is not accessible before logging in

[Martin Pitt]

I have juju-local deployed on current xenial. This works fine right after installation (except for bug 1534106), but after a fresh boot "juju status" hangs. I have two failed units:

$ systemctl --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
● juju-agent-martin-local.service loaded failed failed juju agent for machine-0
● juju-db-martin-local.service loaded failed failed juju state database

Unfortunately they don't do any error logging to the journal, they just exit with 127:

$ sudo systemctl status -l juju-agent-martin-local.service
● juju-agent-martin-local.service - juju agent for machine-0
   Loaded: loaded (/var/lib/juju/init/juju-agent-martin-local/juju-agent-martin-local.service; enabled; vendor preset: enabled)
   Active: failed (Result: start-limit) since Do 2016-01-21 08:51:31 CET; 27min ago
 Main PID: 1017 (code=exited, status=127)

Jan 21 08:51:31 donald systemd[1]: juju-agent-martin-local.service: Main process exited, code=exited, status=127/n/a
Jan 21 08:51:31 donald systemd[1]: juju-agent-martin-local.service: Unit entered failed state.
Jan 21 08:51:31 donald systemd[1]: juju-agent-martin-local.service: Failed with result 'exit-code'.
Jan 21 08:51:31 donald systemd[1]: juju-agent-martin-local.service: Service hold-off time over, scheduling restart.
Jan 21 08:51:31 donald systemd[1]: Stopped juju agent for machine-0.
Jan 21 08:51:31 donald systemd[1]: juju-agent-martin-local.service: Start request repeated too quickly.
Jan 21 08:51:31 donald systemd[1]: Failed to start juju agent for machine-0.
Jan 21 08:51:31 donald systemd[1]: juju-agent-martin-local.service: Unit entered failed state.
Jan 21 08:51:31 donald systemd[1]: juju-agent-martin-local.service: Failed with result 'start-limit'.

juju-db-martin-local.service repeatedly fails with exit code 1, but otherwise it's a similar output.

They can be manually started after booting with sudo systemctl start 'juju-*' though, and then things work. So it looks like they are missing some dependency?

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: juju-core 1.25.0-0ubuntu1
ProcVersionSignature: Ubuntu 4.3.0-6.17-generic 4.3.3
Uname: Linux 4.3.0-6-generic x86_64
ApportVersion: 2.19.3-0ubuntu3
Architecture: amd64
CurrentDesktop: Unity
Date: Thu Jan 21 09:16:05 2016
EcryptfsInUse: Yes
SourcePackage: juju-core
UpgradeStatus: No upgrade log present (probably fresh install)

[Martin Pitt (pitti)]

Oh, I see what's wrong: They run /var/lib/juju/init/juju-agent-martin-local/exec-start.sh, which runs

  /home/martin/.juju/local/tools/machine-0/jujud

This can't work, as my home directory does not exist until I log in (it's encrypted). This will equally affect /home on NFS and the like.

 * Do these juju and mongodb processes need to run as root? If they can run as (my) user, then these two services should become session services instead of system services.

 * If they do need to run as root, please put the data into /var/lib somewhere, as you can't assume access to /home directories with system services.

[Martin Pitt (pitti)]

This does not affect juju-2.0, only 1.0. With 2.0 there is no cluttering of the host any more but everything is put into the controller LXD container.