'container failed to start' with local provider

[Liam Young]

juju-core 1.21-beta2-0ubuntu1~14.10.1~juju1

When using juju local provider on a fresh utopic install, bootstrap succeeds but adding a machine fails. juju status returns:

environment: local
machines:
  "0":
    agent-state: started
    agent-version: 1.21-beta2.1
    dns-name: localhost
    instance-id: localhost
    series: utopic
    state-server-member-status: has-vote
  "1":
    agent-state-info: 'container failed to start: container failed to start'
    instance-id: pending
    series: trusty
services: {}

/var/lib/juju/containers/liam-local-machine-1/container.log contains:

      lxc-start 1416342599.055 ERROR lxc_conf - conf.c:mount_entry:1880 - No such file or directory - failed to mount '/home/liam/.juju/local/storage/tools/releases' on '/usr/lib/x86_64-linux-gnu/lxc/var/lib/juju/storage/tools'
      lxc-start 1416342599.055 ERROR lxc_conf - conf.c:lxc_setup:4051 - failed to setup the mount entries for 'liam-local-machine-1'

(full log attached)

$ ls -l /home/liam/.juju/local/storage/tools/releases /usr/lib/x86_64-linux-gnu/lxc/var/lib/juju/storage/tools
ls: cannot access /home/liam/.juju/local/storage/tools/releases: No such file or directory
ls: cannot access /usr/lib/x86_64-linux-gnu/lxc/var/lib/juju/storage/tools: No such file or directory
$ ls -l /home/liam/.juju/local/storage/
total 0

[Ian Booth (wallyworld)]

The issue is that there was a template container plus config generated by Juju 1.20 or earlier. This template container is used (when lxc-clone is true) to make container creattion fast. But the template config is out of date with 1.21, because it contains a bind mount to a tools directory that no longer exists in 1.21.

So, an upgrade to 1.21 should check for existing lxc templates and do something - whether that's warning the user and advising them to delete the template, or attempting to parse and fix the lxc config, I'm not sure.

[Curtis Hovey (sinzui)]

Users who regularly develop with fresh images will not experience this.

I think users of local-provider lxc need to purge templates and cache regularly. The performance increase in 1.20.x and 1.21.x came at the expense fresh images. the defaults are now cloning a cached image and never upgrading the packages. Developers and testers need to change their workflow to ensure they have fresh images. When I start a development effort I do

    sudo lxc-destroy --name juju-trusty-lxc-template
    sudo rm -r /var/cache/lxc/cloud-trusty

And in Juju CI, we need to purge every 2 weeks.

[Douglas Sikora (doug-s1k0ra)]

+1 for Curtis #3

    sudo lxc-destroy --name juju-trusty-lxc-template
    sudo rm -r /var/cache/lxc/cloud-trusty

cleared it for me