non-default lxc-dir breaks local provider

[Martin Pitt]

I tried to use the local provider according to https://juju.ubuntu.com/docs/config-local.html, in particular https://juju.ubuntu.com/docs/config-LXC.html . James Page already informed me that the documentation is outdated for trusty, and one needs to drop the "sudo" from "juju bootstrap". With that, bootstrapping fails due to not being able to start the container, when using a non-default LXC path (I can't use /var/lib/lxc, my root partition is too small).

I initially experienced that on my normal trusty workstation, but I reproduced in a current and pristine trusty cloud-image VM (I just ran "run-adt-test -sUl" from lp:auto-package-testing).

$ sudo apt-get install juju-local

Now set up LXC for a different container path:

$ sudo mkdir /srv/lxc
$ echo "lxc.lxcpath = /srv/lxc" | sudo tee /etc/lxc/lxc.conf

Note that this works just fine with all the LXC tools.

$ juju generate-config

Now create a different juju root:

$ sudo mkdir /srv/juju; sudo chown ubuntu:ubuntu /srv/juju
$ sed -i '/root-dir/ { s/# //; s_:.*$_: /srv/juju_ }' .juju/environments.yaml

$ juju switch local
$ juju bootstrap
Logging to /home/ubuntu/.juju/local/log/cloud-init-output.log on remote host
Starting MongoDB server (juju-db-ubuntu-local)
Bootstrapping Juju machine agent
Starting Juju machine agent (juju-agent-ubuntu-local)

So far so good:

$ juju status
environment: local
machines:
  "0":
    agent-state: started
    agent-version: 1.17.4.1
    dns-name: localhost
    instance-id: localhost
    series: trusty
services: {}

There are no actual containers running (sudo lxc-ls --fancy), but that's the same as with using the default /var/lib/lxc/.

$ juju deploy mysql
Added charm "cs:precise/mysql-36" to the environment.

After a while I see:

$ juju status
environment: local
machines:
  "0":
    agent-state: started
    agent-version: 1.17.4.1
    dns-name: localhost
    instance-id: localhost
    series: trusty
  "1":
    agent-state-info: '(error: error executing "lxc-start": command get_cgroup failed
      to receive response)'
    instance-id: pending
    series: precise
services:
  mysql:
    charm: cs:precise/mysql-36
    exposed: false
    relations:
      cluster:
      - mysql
    units:
      mysql/0:
        agent-state: pending
        machine: "1"

In .juju/local/log/machine-0.log I see:

2014-03-11 15:48:49 WARNING juju.worker.instanceupdater updater.go:231 cannot get instance info for instance "": no instances found
2014-03-11 15:49:16 ERROR juju.container.lxc lxc.go:134 container failed to start: error executing "lxc-start": command get_cgroup failed to receive response
2014-03-11 15:49:16 ERROR juju.provisioner provisioner_task.go:403 cannot start instance for machine "1": error executing "lxc-start": command get_cgroup failed to receive response

(Attaching the whole file)

Other than that, .juju/local/log/ contains a broken symlink "all-machines.log -> /var/log/juju-ubuntu-local/all-machines.log" and "cloud-init-output.log" without any error messages.

Note that this works with the default /var/lib/lxc/ path.

[Martin Pitt (pitti)]

In that situation I see:

$ sudo lxc-ls --fancy
NAME STATE IPV4 IPV6 AUTOSTART
------------------------------------------------------
ubuntu-local-machine-1 STOPPED - - YES

Trying to boot it manually reveals the problem:

$ sudo lxc-start -n ubuntu-local-machine-1
lxc-start: No such file or directory - failed to mount '/home/ubuntu/.juju/local/log' on '/usr/lib/x86_64-linux-gnu/lxc/var/log/juju'
lxc-start: failed to setup the mount entries for 'ubuntu-local-machine-1'
lxc-start: failed to setup the container
lxc-start: invalid sequence number 1. expected 2
lxc-start: failed to spawn 'ubuntu-local-machine-1'

For testing I commented out the lxc.mount.entry for /home/ubuntu/.juju/local/log in /srv/lxc/ubuntu-local-machine-1/config, and tried to start the container again. It boots now, and starts cloud-init. I can't log in though, as ubuntu/ubuntu does not work and I don't know which account it created, so I stopped the container manually. At this point I don't know how to tell juju to re-poke that instance to continue the setup.

Supposedly this is due to some AppArmor restriction? I don't see any REJECTs in dmesg, but failing to bind-mount with a non-default container path sounds like that?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Tim Penhey (thumper)]

It seems to me that the bug isn't in the different container directory, but the way that the log directory is mounted.

I have just hit this same problem locally, and I don't have a non-default directory.

What I do have though is two different users locally.

When the juju conatiner template is created, the log mount points are added. What this means is that other users on the same machine will not be able to mount the log directory as specified in the container lxc config file, and the cgroup setting will fail.

One solution to this is to have a template container per user rather than a shared one.

[Tim Penhey (thumper)]

I found that the code was committed back in march to not mount the log dir in the container.

My juju-precise-template lxc container had a line that include a mount to /home/tim/.juju/local/log, which caused cgroups to complain when logged in as my other local user.

Removing the juju-precise-template and getting juju to recreate it fixed the problem for me.

My template container was created about a week before the code landed in trunk.

@pitti, can I get you to take a look inside the config file for the lxc container to see if you have a similar problem?

[Tim Penhey (thumper)]

Gah, I should read the entire description before changing things. This was a different problem to me that had exactly the same error message.

[Martin Pitt (pitti)]

For the record, on current utopic I didn't see this "failed to mount '/home/ubuntu/.juju/local/log'" error message any more. I still have trouble with non-default lxcpath, but it's different now. I filed bug 1329049 about that yesterday.

So from my POV this can be closed, not sure if Tim still wants to keep this for the bug he's seeing?

[Jorge Niedbalski (niedbalski)]

Tim,

According to @pitti, this specific bug is fixed, i will jump to LP: #1329049.

Are you OK to mark this one as fix released?