1.25.0 and 1.24.6 cannot bootstrap local environment on wily host

[Mike McCracken]

Version 1.25.0 (update: and 1.24.6) is having problems creating VMs on wily.

First noticed with the openstack-installer, where we bootstrap a local env and use add-machine to add three VMs.
The bootstrap came up, but the VMs were forever stuck waiting for agent initialization.

Then I tried to reproduce outside of the installer and bootstrap fails by itself.

Details below:

- [X] sudo apt-add-repository ppa:juju/stable
- [X] sudo apt install juju-core
got 1.25.0-wily-amd64

- [X] juju init
- [X] edit env file

*************** contents of new env file:
NOTE: this env file works with juju 1.24

  default: local
  environments:
    local:
      admin-secret: ubuntu
      apt-http-proxy: http://172.16.0.62:3142
      apt-https-proxy: http://172.16.0.62:3142
      authorized-keys-path: ~/.ssh/id_rsa.pub
      container: kvm
      default-series: trusty
      lxc-clone: true
      network-bridge: lxcbr0
      no-proxy: localhost
      type: local

- [X] sudo apt install juju-local uvtool-libvirt uvtool
- [X] juju bootstrap

*************** output
ubuntu@knotty-temper:~⟫ juju bootstrap
Bootstrapping environment "local"
Starting new instance for initial state server
Building tools to upload (1.25.0.1-wily-amd64)
Installing Juju agent on bootstrap instance
Logging to /home/ubuntu/.juju/local/cloud-init-output.log on remote host
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Installing package: cloud-utils
Installing package: cloud-image-utils
Installing package: tmux
Bootstrapping Juju machine agent
Starting Juju machine agent (juju-agent-ubuntu-local)
Bootstrap agent installed
ERROR failed getting all instances: exit status 1
ERROR saving bootstrap endpoint address: exit status 1

*************** ps | grep juju output after it died includes:
15366 ? Ssl 0:11 /usr/lib/juju/bin/mongod --auth --dbpath /home/ubuntu/.juju/local/db --sslOnNormalPorts --sslPEMKeyFile /home/ubuntu/.juju/local/server.pem --sslPEMKeyPassword xxxxxxx --port 37017 --noprealloc --syslog --smallfiles --journal --keyFile /home/ubuntu/.juju/local/shared-secret --replSet juju --ipv6 --oplogSize 1
15670 ? Ss 0:00 bash /var/lib/juju/init/juju-agent-ubuntu-local/exec-start.sh
15674 ? Sl 0:04 \_ /home/ubuntu/.juju/local/tools/machine-0/jujud machine --data-dir /home/ubuntu/.juju/local --machine-id 0 --debug

[Mike McCracken (mikemc)]

Here's an additional attempt with 'juju bootstrap --debug' for extra info

- [X] sudo apt-add-repository ppa:juju/stable
- [X] sudo apt install juju-core
got 1.25.0-wily-amd64
- [X] juju init
- [X] edit env file
*************** contents of new env file:
NOTE: this env file works with juju 1.24

#+BEGIN_SRC yaml
  default: local
  environments:
    local:
      admin-secret: ubuntu
      apt-http-proxy: http://172.16.0.62:3142
      apt-https-proxy: http://172.16.0.62:3142
      authorized-keys-path: ~/.ssh/id_rsa.pub
      container: kvm
      default-series: trusty
      lxc-clone: true
      network-bridge: lxcbr0
      no-proxy: localhost
      type: local
#+END_SRC
*************** END
- [X] sudo apt install juju-local uvtool-libvirt uvtool
- [X] juju bootstrap
*************** output
ubuntu@knotty-temper:~⟫ juju bootstrap
Bootstrapping environment "local"
Starting new instance for initial state server
Building tools to upload (1.25.0.1-wily-amd64)
Installing Juju agent on bootstrap instance
Logging to /home/ubuntu/.juju/local/cloud-init-output.log on remote host
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Installing package: cloud-utils
Installing package: cloud-image-utils
Installing package: tmux
Bootstrapping Juju machine agent
Starting Juju machine agent (juju-agent-ubuntu-local)
Bootstrap agent installed
ERROR failed getting all instances: exit status 1
ERROR saving bootstrap endpoint address: exit status 1

*************** END

*************** ps | grep juju output after it died includes:
15366 ? Ssl 0:11 /usr/lib/juju/bin/mongod --auth --dbpath /home/ubuntu/.juju/local/db --sslOnNormalPorts --sslPEMKeyFile /home/ubuntu/.juju/local/server.pem --sslPEMKeyPassword xxxxxxx --port 37017 --noprealloc --syslog --smallfiles --journal --keyFile /home/ubuntu/.juju/local/shared-secret --replSet juju --ipv6 --oplogSize 1
15670 ? Ss 0:00 bash /var/lib/juju/init/juju-agent-ubuntu-local/exec-start.sh
15674 ? Sl 0:04 \_ /home/ubuntu/.juju/local/tools/machine-0/jujud machine --data-dir /home/ubuntu/.juju/local --machine-id 0 --debug

*************** END

- [ ] try again clean with juju --debug bootstrap
see [[*Attempt 2 : add --debug][Attempt 2 : add --debug]]

- [ ] juju add machine with constraints and see if it comes up

*** Attempt 2 : add --debug

- [X] reimage a wily host
- [X] sudo apt-add-repository ppa:juju/stable
- [X] sudo apt update
- [X] sudo apt install juju-core juju-local uvtool-libvirt uvtool emacs24-nox
got juju-core 1.25.0
- [X] juju init
- [X] edit env file
*************** contents of new env file:
NOTE: this env file works with juju 1.24

#+BEGIN_SRC yaml
  default: local
  environments:
    local:
      admin-secret: ubuntu
      apt-http-proxy: http://172.16.0.62:3142
      apt-https-proxy: http://172.16.0.62:3142
      authorized-keys-path: ~/.ssh/id_rsa.pub
      container: kvm
      default-series: trusty
      lxc-clone: true
      network-bridge: lxcbr0
      no-proxy: localhost
      type: local
#+END_SRC
*************** END

- [X] juju --debug bootstrap
*************** output
ubuntu@knotty-temper:~⟫ juju ...

[Mike McCracken (mikemc)]

OK, this is also happening with 1.24:

history
    1 sudo apt update
    2 apt list --upgradable
    3 sudo apt install juju-core juju-local uvtool-libvirt uvtool emacs24-nox
    4 juju --version
    5 juju init
    6 rm .juju/environments.yaml
    7 emacs .juju/environments.yaml
    8 juju --debug bootstrap
    9 juju --version
   10 history

ubuntu@knotty-temper:~⟫ juju --debug bootstrap
2015-10-30 19:46:52 INFO juju.cmd supercommand.go:37 running juju [1.24.6-wily-amd64 gc]
2015-10-30 19:46:52 DEBUG juju.environs.configstore disk.go:109 Made dir /home/ubuntu/.juju/environments
2015-10-30 19:46:52 INFO juju.provider.local environprovider.go:246 checking state port
2015-10-30 19:46:52 INFO juju.provider.local environprovider.go:246 checking API port
2015-10-30 19:46:52 INFO juju.provider.local environprovider.go:39 opening environment "local"
2015-10-30 19:46:52 DEBUG juju.container.kvm kvm.go:71 kvm-ok output:
INFO: /dev/kvm exists
KVM acceleration can be used

2015-10-30 19:46:52 INFO juju.provider.local environprovider.go:77
Attribute "apt-http-proxy" is set to (http://172.16.0.62:3142)
2015-10-30 19:46:52 INFO juju.provider.local environprovider.go:77
Attribute "apt-https-proxy" is set to (http://172.16.0.62:3142)
2015-10-30 19:46:52 DEBUG juju.provider.local environ.go:325 found "10.0.3.1" as address for "lxcbr0"
2015-10-30 19:46:52 DEBUG juju.environs.configstore disk.go:308 writing jenv file
2015-10-30 19:46:52 DEBUG juju.environs.configstore disk.go:432 writing jenv file to /home/ubuntu/.juju/environments/local.jenv
2015-10-30 19:46:52 INFO juju.network network.go:194 setting prefer-ipv6 to false
2015-10-30 19:46:52 INFO juju.cmd cmd.go:113 Bootstrapping environment "local"
2015-10-30 19:46:52 DEBUG juju.environs.bootstrap bootstrap.go:98 environment "local" supports service/machine networks: false
2015-10-30 19:46:52 DEBUG juju.environs.bootstrap bootstrap.go:100 network management by juju enabled: true
2015-10-30 19:46:52 DEBUG juju.provider.local environ.go:325 found "10.0.3.1" as address for "lxcbr0"
2015-10-30 19:46:52 INFO juju.cmd cmd.go:113 Starting new instance for initial state server
2015-10-30 19:46:52 INFO juju.environs.bootstrap bootstrap.go:184 newest version: 1.24.6.1
2015-10-30 19:46:52 INFO juju.environs.bootstrap bootstrap.go:212 picked bootstrap tools version: 1.24.6.1
2015-10-30 19:46:52 INFO juju.cmd cmd.go:113 Building tools to upload (1.24.6.1-wily-amd64)
2015-10-30 19:46:52 DEBUG juju.environs.sync sync.go:304 Building tools
2015-10-30 19:46:52 DEBUG juju.environs.tools build.go:122 looking for: juju
2015-10-30 19:46:52 DEBUG juju.environs.tools build.go:161 checking: /usr/bin/jujud
2015-10-30 19:46:52 INFO juju.environs.tools build.go:167 found existing jujud
2015-10-30 19:46:52 INFO juju.environs.tools build.go:177 target: /tmp/juju-tools286779638/jujud
2015-10-30 19:46:52 DEBUG juju.environs.tools build.go:232 forcing version to 1.24.6.1
2015-10-30 19:46:52 DEBUG juju.environs.tools build.go:38 adding entry: &tar.Header{Name:"FORCE-VERSION", Mode:436, Uid:0, Gid:0, Size:8, ModTime:time.Time{sec:63581831212, nsec:960076039, loc:(*time.Location)(0x2eac420)}, Typeflag:0x30, Linkname:"", Uname:"ub...

[Mike McCracken (mikemc)]

Some further info, from a system with 1.24.6.1

Just verifying that bootstrap did return an exit code of 1, and that the env was unusable afterwards, and un-bootstrappable as well.

I did this verification because the earlier attempt (see prev comment #2 for the bootstrap log) with 1.24 gave the same errors from bootstrap, but actually left the env in a usable state, after which add-machine did work.

>> annotate-output juju --debug bootstrap:
...
22:35:10 E: Bootstrapping Juju machine agent
22:35:26 E: Starting Juju machine agent (juju-agent-ubuntu-local)
22:35:26 E: 2015-10-30 22:35:26 INFO juju.cmd cmd.go:113 Bootstrap agent installed
22:35:26 E: 2015-10-30 22:35:26 ERROR juju.container.kvm kvm.go:194 failed getting all instances: exit status 1
22:35:26 E: 2015-10-30 22:35:26 ERROR juju.cmd supercommand.go:430 saving bootstrap endpoint address: exit status 1
22:35:26 I: Finished with exitcode 1

1 ubuntu@knotty-temper:~⟫ juju status
ERROR failed getting all instances: exit status 1
ERROR Unable to connect to environment "local".
Please check your credentials or use 'juju bootstrap' to create a new environment.

Error details:
exit status 1

# this is the second try for a bootstrap:

1 ubuntu@knotty-temper:~⟫ annotate-output juju --debug bootstrap
22:37:54 I: Started juju --debug bootstrap
22:37:54 E: 2015-10-30 22:37:54 INFO juju.cmd supercommand.go:37 running juju [1.24.6-wily-amd64 gc]
22:37:54 E: 2015-10-30 22:37:54 DEBUG juju.environs open.go:70 ConfigForName found bootstrap config map[string]interface {}{"apt-https-proxy":"http://172.16.0.62:3142", "firewall-mode":"instance", "no-proxy":"localhost", "syslog-port":6514, "agent-version":"1.24.6", "authorized-keys":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/+AR2qgCQ598Ct52hnatqMgv9phYW9pUn7CFa9LHLD4/3DUYlluV6Q6yn7wG+hA5A1NNM5hmL6WkGJ8/8vLPSCmUhIjpls8MF4OkgJuoQxa4Z/A/cGDF/vTAq6GtWkV579ZDCg1MClmBp8ucmUO0kX1xWaKFF6tanJvkiXPRanCI6KBd6GHIF58j5UU2onxQlxUqIzA/2FyYo7zoqQSosaGXD+xTn6OOpoUNewi0dzghzQdglFYqhO60jfVhm5kxUTuKV8EscRpVbCS9YqMwF2TWMJ1EJTLu45lh3IKERTPzPMhwH/OZqV9ghWsmwiLCp0nwZ7FM2mjXz+My9zeTz juju-client-key\n", "lxc-clone":true, "set-numa-control-policy":false, "state-port":37017, "apt-http-proxy":"http://172.16.0.62:3142", "bootstrap-addresses-delay":10, "tools-metadata-url":"", "default-series":"trusty", "logging-config":"<root>=DEBUG;unit=DEBUG", "proxy-ssh":false, "agent-metadata-url":"", "bootstrap-retry-delay":5, "bootstrap-timeout":600, "container":"kvm", "image-metadata-url":"", "lxc-clone-aufs":false, "prefer-ipv6":false, "root-dir":"/home/ubuntu/.juju/local", "allow-lxc-loop-mounts":false, "api-port":17070, "test-mode":false, "type":"local", "disable-network-management":false, "ignore-machine-addresses":false, "image-stream":"", "name":"local", "namespace":"ubuntu-local", "network-bridge":"lxcbr0", "ca-private-key":"-----BEGIN RSA PRIVATE KEY-------\n"}
22:37:54 E: 2015-10-30 22:37:54 WARNING juju.cmd.juju common.go:86 ignoring environments.yaml: using bootstrap config in file "/home/ubuntu/.juju/environments/local.jenv"
22:37:54 E: 2015-10-30 22:37:54 DEBUG juju.environs open.go:70 ConfigForName found bootstrap config map[string]interface {}{"root-dir":"/home/ubuntu/.juju...

[Mike McCracken (mikemc)]

After rebooting the machine running 1.24.6.1 from comment #3, when it comes back up, 'juju status' works and shows me a running env which accepts 'add-machine'

[Cheryl Jennings (cherylj)]

I think you might be hitting a couple issues here. The first, where the bootstrap succeeds, but the kvm guests stay in pending is bug #1510619.

The other issue, where bootstrap fails with "failed getting all instances: exit status 1", seems to match the symptoms of bug #1308088.