LXD containers are not autostarting, because the code is setting user.boot.autostart instead of boot.autostart
There are quite a few things interacting here that make this non obvious and hard to test
- LXD containers autostart if they were last running.. running "poweroff" within the LXD host will save that state. Hard powering down the LXD host will not, e.g. echo b > /proc/sysrq-trigger -- in my test environment running poweroff on the host, or pressing it's power button, caused similar unclean shutdowns. This LXD behavior probably needs improving but is separate to the main issue here.
- the key misunderstanding appears to be that Metadata passed to lxdclient.InstanceSpec is all namespaced under user, it is not direct config. Using lxdclient.UserdataKey as the constant for "user-data" and calling the "user" namespace data "metadata" probably both contribute to that not being obvious
- there is also code in provider/lxd/environ.go to setup the "environment" and create an lxd profile with this same setting. This seems to not work or never run -- I don't fully understand when this code is called.
Relevant code in container/lxd/lxd.go:CreateContainer()
metadata := map[string]string{
lxdclient.UserdataKey: string(userData),
// An extra piece of info to let people figure out where this
// thing came from.
"user.juju-model": manager.modelUUID,
// Make sure these come back up on host reboot.
"boot.autostart": "true", //wrong.. this is prefixed with autostart
}
If we look at a deployed host, we'll see that both juju-model and boot.autostart are set under user:
root@maas-node02:/home/ubuntu# lxc config show juju-c9e3cb-0-lxd-0|grep user
user.boot.autostart: "true"
user.user-data: |
users:
user.user.juju-model: 9e133ad9-6c3b-4977-8716-fa984ac9e3cb
root@maas-node02:/home/ubuntu# lxc config show juju-c9e3cb-0-lxd-0|grep autostart
user.boot.autostart: "true"
root@maas-node02:/home/ubuntu# lxc profile list
default
docker
root@maas-node02:/home/ubuntu# lxc profile show default
name: default
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
Steps to reproduce (tested on rc3, xenial)
- Setup a virtual MAAS with 2 nodes, register it and the credentials in juju
- juju bootstrap maas maas
- juju add-machine
- juju deploy apache2 --to lxd:0
- (wait for deployment)
- juju ssh 0 sudo reboot
- (once restarted) juju ssh 0 sudo lxc list # observe lxc container is running
- juju ssh 0 "echo b | sudo tee /proc/sysrq-trigger"
- (once restarted) juju ssh 0 sudo lxc list # observe lxc container is not running
- (optional) juju ssh 0 sudo reboot # observe still not running, because it was not last saved as running
- juju ssh 0 sudo lxc list
- juju ssh 0 sudo lxc config show <instance id> |grep user
Suggested Solution:
I wanted to contribute a patch, but I'm not really sure on the most correct fix. There is no mechanism to pass config into the lxd client right now, so we could either
- Hardcode boot.autostart into lxdclient (seems like a bad idea)
- Convert metadata to config, and explicitly namespace the user parts
- Add an extra mechanism to pass config, separate to metadata
- Investigate when/why the profile is not created in the newEnviron() code and later used to set autostart instead -- then use profiles to set this config
While fixing this, I would also suggest fixing "user.juju-model", as this currently sets user.user.juju-model
Created related lxd bug, about the autostart behavior being confusing:
https:/