Can't bootstrap environment after latest lxd upgrade

[Casey Marshall]

Upgraded lxd today and I'm now unable to bootstrap the lxd provider. I suspect some changes in their API.

Platform: Ubuntu 14.04 amd64

Juju: built from source (recent master, 1c3373611a025791e28c191dca59eacd97a4b03d)

LXD: installed from ppa:ubuntu-lxc/lxd-stable
ii lxd 2.0.0~beta3-0ubuntu1~ubuntu14.04.1~ppa1 amd64 Container hypervisor based on LXC - daemon
ii lxd-client 2.0.0~beta3-0ubuntu1~ubuntu14.04.1~ppa1 amd64 Container hypervisor based on LXC - client
ii lxd-tools 2.0.0~beta3-0ubuntu1~ubuntu14.04.1~ppa1 amd64 Container hypervisor based on LXC - extra tools

---

$ juju bootstrap --upload-tools --verbose
Bootstrapping model "lxd"
Starting new instance for initial controller
Launching instance
Bootstrap failed, destroying model
ERROR failed to bootstrap model: cannot start bootstrap instance: can't get info for image 'ubuntu-trusty': json: cannot unmarshal string into Go value
 of type int64

$ juju bootstrap --upload-tools --debug
2016-02-19 00:00:02 INFO juju.cmd supercommand.go:59 running juju [2.0-alpha2 gc go1.5.2]
2016-02-19 00:00:04 DEBUG juju.container.lxd.lxdclient client.go:53 loading LXD client config from "/home/cmars/.config/lxc/juju-lxd"
2016-02-19 00:00:04 DEBUG juju.container.lxd.lxdclient client.go:64 using LXD remote "local"
2016-02-19 00:00:04 DEBUG juju.container.lxd.lxdclient config.go:164 initializing config dir "/home/cmars/.config/lxc/juju-lxd"
2016-02-19 00:00:04 DEBUG juju.container.lxd.lxdclient config.go:207 writing config file "/home/cmars/.config/lxc/juju-lxd/config.yml"
2016-02-19 00:00:04 DEBUG juju.container.lxd.lxdclient config.go:257 writing cert PEM file "/home/cmars/.config/lxc/juju-lxd/client.crt"
2016-02-19 00:00:04 DEBUG juju.container.lxd.lxdclient config.go:273 writing key PEM file "/home/cmars/.config/lxc/juju-lxd/client.key"
2016-02-19 00:00:04 DEBUG juju.container.lxd.lxdclient client.go:53 loading LXD client config from "/home/cmars/.config/lxc/juju-lxd"
2016-02-19 00:00:04 DEBUG juju.container.lxd.lxdclient client.go:64 using LXD remote "juju-remote"
2016-02-19 00:00:04 DEBUG juju.environs.configstore disk.go:326 writing cache file
2016-02-19 00:00:04 INFO juju.cmd.juju bootstrap.go:292 combined bootstrap constraints:
2016-02-19 00:00:04 INFO juju.network network.go:248 setting prefer-ipv6 to false
2016-02-19 00:00:04 INFO cmd cmd.go:129 Bootstrapping model "lxd"
2016-02-19 00:00:04 DEBUG juju.environs.bootstrap bootstrap.go:137 model "lxd" supports service/machine networks: false
2016-02-19 00:00:04 DEBUG juju.environs.bootstrap bootstrap.go:139 network management by juju enabled: true
2016-02-19 00:00:04 INFO cmd cmd.go:129 Starting new instance for initial controller
Launching instance
2016-02-19 00:00:04 DEBUG juju.provider.lxd environ_broker.go:47 StartInstance: "0", trusty
2016-02-19 00:00:04 DEBUG juju.provider.lxd environ_broker.go:73 tools: &tools.Tools{Version:version.Binary{Number:version.Number{Major:2, Minor:0, Tag
:"alpha", Patch:2, Build:1}, Series:"trusty", Arch:"amd64"}, URL:"", SHA256:"", Size:0}
2016-02-19 00:00:04 DEBUG juju.cloudconfig.instancecfg instancecfg.go:545 Setting numa ctl preference to false
2016-02-19 00:00:04 DEBUG juju.service discovery.go:59 discovered init system "upstart" from series "trusty"
2016-02-19 00:00:04 DEBUG juju.provider.lxd environ_broker.go:145 LXD user data; 1171 bytes
2016-02-19 00:00:04 INFO juju.provider.lxd environ_broker.go:129 starting instance "juju-091ab62e-0da6-4288-880b-55342c0107a1-machine-0" (image "ubuntu
-trusty")...
2016-02-19 00:00:05 DEBUG juju.cmd.juju common.go:88 Destroying model.
2016-02-19 00:00:05 INFO cmd cmd.go:129 Bootstrap failed, destroying model
2016-02-19 00:00:05 INFO juju.provider.common destroy.go:22 destroying model "lxd"
2016-02-19 00:00:05 INFO juju.provider.common destroy.go:33 destroying instances
2016-02-19 00:00:05 INFO juju.provider.common destroy.go:53 destroying storage
2016-02-19 00:00:05 ERROR cmd supercommand.go:448 failed to bootstrap model: cannot start bootstrap instance: can't get info for image 'ubuntu-trusty':
 json: cannot unmarshal string into Go value of type int64

[Casey Marshall (cmars)]

Also confirmed this on latest xenial server w/juju master & 2.0~alpha2 compiled from source. xenial also currently installs lxd 2.0.0~beta3. Looks like this release of LXD has introduced some API changes that are incompatible with Juju's LXD provider.

[Randall Ross (randall)]

I can confirm the same error:
randall@randall-vm:~$ lxc image list
+---------------+--------------+--------+--------------------------------------+--------+---------+------------------------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCH | SIZE | UPLOAD DATE |
+---------------+--------------+--------+--------------------------------------+--------+---------+------------------------------+
| ubuntu-trusty | 7336a7a1ae64 | no | Ubuntu 14.04 LTS server (20160217.1) | x86_64 | 98.49MB | Feb 19, 2016 at 8:50pm (UTC) |
+---------------+--------------+--------+--------------------------------------+--------+---------+------------------------------+
randall@randall-vm:~$ juju bootstrap --upload-tools
Bootstrapping model "lxd"
Starting new instance for initial controller
Launching instance
Bootstrap failed, destroying model
ERROR failed to bootstrap model: cannot start bootstrap instance: can't get info for image 'ubuntu-trusty': json: cannot unmarshal string into Go value of type int64

[Billy Olsen (billy-olsen)]

Hmm, not sure this is entirely Juju's fault. I think the API must have been broken somewhere along the way. In fact, the actual error that's displayed comes from the LXD code itself - however, it looks like the juju-core code is pinned to the 0.21 commit from the dependencies list

[Billy Olsen (billy-olsen)]

Updating the dependencies itself breaks juju due to ContainerStatus being new (and some other errors). I'm suspecting a new patch will land soon with an updated lxd client

[Randall Ross (randall)]

Confirmed bug is still present with today's updates applied:

juju-core: 2.0-alpha2-0ubuntu1~16.04.1~juju1
LXD: 2.0.0~beta3-0ubuntu2
LXC: 2.0.0~rc1-0ubuntu1

[Richard Harding (rharding)]

Yes, still the case on lxd 2.0beta3 and juju 2.0 beta1

[Randall Ross (randall)]

Thanks Rick. Is there an ETA for a fix? Are we waiting for LXD?

[John A Meinel (jameinel)]

I'm working on this.
LXD is saying that they aren't committing to a stable API until they actually go 2.0.0 final. I have a branch that starts the migration with a PR here: https://github.com/jameinel/juju/pull/6

However, the big issues are:
1) They flattened one of their structures which is nice, but in the process removed one of the fields we were using. We can work around this, but I want to know if it was intentional or an accident before I commit to doing the extra work.
2) We have LXD 0.21 in Wily with no plans (that I've heard of) to backport 2.0.0 to Wily. And because LXD API is breaking (and they aren't going to try and do a compatibility layer), we have to decide what we want to do in Juju. I think the answer is just "roll forward", and use something like a PPA on Wily. (trusty-backports does have 2.0.0beta3 right now, and https://launchpad.net/~ubuntu-lxc/+archive/ubuntu/lxd-stable?field.series_filter=wily 'stable' ppa has 2.0.0beta3 as well)
3) Getting LXD from backports on trusty causes some small problems because of dependency management. Namely "apt-get install lxd/trusty-backports" fails because it is missing dependencies that are only available in backports. you need at least "apt-get install lxd/trusty-backports python3-lxc/trusty-backports" and at least 1 or 2 other trusty-backports only packages.

[Andreas Hasenack (ahasenack)]

With lxd beta4 I'm getting a different error now:

$ juju bootstrap mycontroller lxd
ERROR invalid config: can't connect to the local LXD server: Response was missing `api_compat`

[John A Meinel (jameinel)]

So if you can get 2.0.0~beta4 (available in the PPA and on Xenial), it looks like this:
 http://reviews.vapour.ws/r/3973/
Should get us to the point of being able to bootstrap the LXD provider again.
It *won't* support 2.0.0~beta3 (architecture changed from an integer field to a string field), and definitely doesn't support 0.21 or 0.26.

[JuanJo Ciarlante (jjo)]

FYI still failing on xenial, even after updated as per above
(fyi rebooted to be sure latest LXD in place).

Versions:
jjo@jjo-kvm-xenial:~$ apt-cache policy juju-core2 lxd lxd-client
juju-core2:
  Installed: 2.0-beta1-0ubuntu1~16.04.1~juju1
  Candidate: 2.0-beta1-0ubuntu1~16.04.1~juju1
  Version table:
 *** 2.0-beta1-0ubuntu1~16.04.1~juju1 500
        500 http://ppa.launchpad.net/juju/devel/ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status
lxd:
  Installed: 2.0.0~beta4-0ubuntu7
  Candidate: 2.0.0~beta4-0ubuntu7
  Version table:
 *** 2.0.0~beta4-0ubuntu7 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status
lxd-client:
  Installed: 2.0.0~beta4-0ubuntu7
  Candidate: 2.0.0~beta4-0ubuntu7
  Version table:
 *** 2.0.0~beta4-0ubuntu7 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status

[JuanJo Ciarlante (jjo)]

forgot to add, --debug output from #11:

jjo@jjo-kvm-xenial:~$ juju bootstrap --debug foo lxd --upload-tools
2016-03-01 22:58:33 INFO juju.cmd supercommand.go:59 running juju [2.0-beta1 gc go1.6rc2]
2016-03-01 22:58:33 INFO cmd cmd.go:141 cloud "lxd" not found, trying as a provider name
2016-03-01 22:58:33 INFO cmd cmd.go:141 no credentials found, checking environment
2016-03-01 22:58:33 DEBUG juju.cmd.juju.commands bootstrap.go:330 preparing controller with config: map[type:lxd name:foo]
2016-03-01 22:58:42 DEBUG juju.container.lxd.lxdclient client.go:53 loading LXD client config from "/home/jjo/.config/lxc/juju-foo"
2016-03-01 22:58:42 DEBUG juju.container.lxd.lxdclient client.go:64 using LXD remote "local"
2016-03-01 22:58:42 ERROR cmd supercommand.go:448 invalid config: can't connect to the local LXD server: Response was missing `api_compat`

[Nicholas Roberts (niccolox)]

I get same error on Xenial

[Cheryl Jennings (cherylj)]

This will be fixed in 2.0-beta2 which we are aiming to release this week.

[Nicholas Roberts (niccolox)]

I updated Juju to the 2.0-beta2, LXD and LXC and get the same error on Xenial

NOTE: I am able to Juju 1.25 and get Wordpress running no problems with lan and public IP

same error

sudo update-alternatives --config juju
There are 2 choices for the alternative juju (providing /usr/bin/juju).

  Selection Path Priority Status
------------------------------------------------------------
  0 /usr/lib/juju-1.25.3/bin/juju 30 auto mode
* 1 /usr/lib/juju-1.25.3/bin/juju 30 manual mode
  2 /usr/lib/juju-2.0-beta2/bin/juju 30 manual mode

Press <enter> to keep the current choice[*], or type selection number: 2
update-alternatives: using /usr/lib/juju-2.0-beta2/bin/juju to provide /usr/bin/juju (juju) in manual mode
devekko@arizmendix:~$ juju bootstrap --debug foo lxd --upload-tools
2016-03-13 03:53:25 INFO juju.cmd supercommand.go:59 running juju [2.0-beta2 gc go1.6]
2016-03-13 03:53:25 INFO cmd cmd.go:141 cloud "lxd" not found, trying as a provider name
2016-03-13 03:53:25 INFO cmd cmd.go:141 no credentials found, checking environment
2016-03-13 03:53:25 DEBUG juju.cmd.juju.commands bootstrap.go:355 preparing controller with config: map[type:lxd name:foo]
2016-03-13 03:53:25 ERROR juju.utils network.go:37 cannot find network interface "lxcbr0": route ip+net: no such network interface
2016-03-13 03:53:25 ERROR cmd supercommand.go:448 invalid config: route ip+net: no such network interface

[Nicholas Roberts (niccolox)]

actually, not the same error, but same outcome, I cant bootstrap on juju 2

my ~/.config/lxc/config.yml

default-remote: local
remotes:
  images:
    addr: https://images.linuxcontainers.org:8443
    public: true
  local:
    addr: unix://
    public: false
aliases: {}

[Nicholas Roberts (niccolox)]

kind of similar to these old bugs https://bugs.launchpad.net/juju-core/+bug/1230306

[Nick Zatkovich (nzatkovich)]

I'm still getting this on Ubuntu 16.04, with ppa lxd (2.1) and ppa juju (2.0-beta16-xenial-amd64)

The lxd bridge is up and the lxd service is running.

2016-08-28 01:21:29 ERROR cmd supercommand.go:458 creating LXD client: Get https://10.189.28.1:8443/1.0: Unable to connect to: 10.189.28.1:8443

[wehde (www-wehde)]

I'm getting the same error as Nick Zatkovich.

Host:
Ubuntu 16.04 xenial
juju-2.0 beta17
setup lxd bridge with lxc profile edit default
added br0 as the parent and set the nic to bridged

lxc profile edit:
name: default
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic

/etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

iface enp5s0 inet manual

#primary bridge
auto br0
iface br0 inet static
    address 192.168.48.186
    netmask 255.255.248.0
    gateway 192.168.50.12
    dns-nameservers 8.8.8.8 8.8.4.4
    bridge_ports enp5s0
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0

error from lxd
2016-09-09 19:26:34 INFO juju.cmd supercommand.go:63 running jujud [2.0-beta17 gc go1.6.2]
2016-09-09 19:26:34 ERROR cmd supercommand.go:458 creating LXD client: Get https://192.168.50.12:8443/1.0: Unable to connect to: 192.168.50.12:8443
ERROR failed to bootstrap model: subprocess encountered error code 1

[Tim Kuhlman (timkuhlman)]

I was seeing a similar problem with beta18, upon investigation the traffic to :8443 was being blocked by iptables. Though lxd added some iptables rules it didn't include this port. Adding a new rule like 'sudo iptables -I INPUT 4 -i lxdbr0 -p tcp --dport 8443 -j ACCEPT' allowed bootstrap to finish.

[Chris MacNaughton (chris.macnaughton)]

The iptables change worked for me too

[Anastasia (anastasia-macmood)]

I cannot reproduce with Juju 2.1-rc1 and lxd 2.4. The problem appears to be fixed as a drive-by with other fixes in the area.