juju deploy is not working on juju 2.0

[Dilip Renkila]

I have a xenial server with juju 2.0 installed. I am trying to use lxd local provider to test workloads on my machine. I bootstrapped the environment without any errors but deploying charms returns an error.

steps to reproduce

$ juju version
2.0-beta6-xenial-amd64

$juju bootstrap lxd-test lxd

$ juju deploy keystone

$ juju status
[Services]
NAME STATUS EXPOSED CHARM
keystone unknown false cs:xenial/keystone-0

[Relations]
SERVICE1 SERVICE2 RELATION TYPE
keystone keystone cluster peer

[Units]
ID WORKLOAD-STATUS JUJU-STATUS VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE
keystone/0 unknown allocating 0 Waiting for agent initialization to finish

[Machines]
ID STATE DNS INS-ID SERIES AZ
0 error pending xenial

$ juju debug-log --debug
2016-05-17 20:38:57 INFO juju.cmd supercommand.go:60 running juju [2.0-beta6 gc go1.6.1]
2016-05-17 20:38:57 INFO juju.juju api.go:213 connecting to API addresses: [10.215.229.154:17070]
2016-05-17 20:38:57 INFO juju.api apiclient.go:494 dialing "wss://10.215.229.154:17070/model/85ff52dd-0a27-4935-8c08-959fdfac87f0/api"
2016-05-17 20:38:57 INFO juju.api apiclient.go:271 connection established to "wss://10.215.229.154:17070/model/85ff52dd-0a27-4935-8c08-959fdfac87f0/api"
2016-05-17 20:38:57 DEBUG juju.juju api.go:362 API hostnames unchanged - not resolving
2016-05-17 20:38:57 DEBUG juju.juju api.go:147 failed to connect via bootstrap config: aborted

[Cheryl Jennings (cherylj)]

Can you paste the output of juju status --format=yaml? That should give us more information on the error.

[Dilip Renkila (dilip-renkila278)]

$ juju status --format=yaml
model: default
machines:
  "0":
    juju-status:
      current: error
      message: 'Failed to get device attributes: no such file or directory'
      since: 17 May 2016 16:22:10-04:00
    instance-id: pending
    machine-status:
      current: provisioning error
      message: 'Failed to get device attributes: no such file or directory'
      since: 17 May 2016 16:22:10-04:00
    series: xenial
services:
  keystone:
    charm: cs:xenial/keystone-0
    exposed: false
    service-status:
      current: unknown
      message: Waiting for agent initialization to finish
      since: 17 May 2016 16:21:58-04:00
    relations:
      cluster:
      - keystone
    units:
      keystone/0:
        workload-status:
          current: unknown
          message: Waiting for agent initialization to finish
          since: 17 May 2016 16:21:58-04:00
        juju-status:
          current: allocating
          since: 17 May 2016 16:21:58-04:00
        machine: "0"

[Curtis Hovey (sinzui)]

The error is internal to lxd. This might not be a juju bug. Does LXD work fine when you directly used it? You can you run these commands:

# If you haven't already used lxd?
lxc image copy ubuntu:xenial local: --alias xenial

# Once xenial is registerest, these commmands should be repeatable.
lxc launch xenial me
lxc list
lxc exec me -- hostname
lxc stop me
lxc delete me

[Dilip Renkila (dilip-renkila278)]

can you please tell me how that is internal to lxd ?
LXD works fine

[Dilip Renkila (dilip-renkila278)]

I tried in another way deploying charms to lxc containers in a MAAS provider. I noticed that the containers are started and got an ip address but juju status remains as "Waiting for agent initialization to finish"

$juju status

[Services]
NAME STATUS EXPOSED CHARM
ceph unknown false cs:xenial/ceph-1
glance active false cs:xenial/glance-1
juju-gui unknown false cs:trusty/juju-gui-54
keystone active false cs:xenial/keystone-1
lxd true cs:xenial/lxd-1
mysql unknown true cs:trusty/mysql-38
nova-cloud-controller active true cs:xenial/nova-cloud-controller-1
nova-compute active true cs:xenial/nova-compute-1
openstack-dashboard terminated true cs:xenial/openstack-dashboard-1
rabbits active false cs:xenial/rabbitmq-server-1
redis unknown false cs:trusty/redis-0
wordpress unknown false cs:trusty/wordpress-4

[Relations]
SERVICE1 SERVICE2 RELATION TYPE
ceph ceph mon peer
glance glance cluster peer
glance keystone identity-service regular
glance lxd container subordinate
glance mysql shared-db regular
glance nova-cloud-controller image-service regular
glance nova-compute image-service regular
glance rabbits amqp regular
keystone keystone cluster peer
keystone lxd container subordinate
keystone mysql shared-db regular
keystone nova-cloud-controller identity-service regular
lxd lxd lxd-migration peer
lxd nova-cloud-controller juju-info regular
lxd nova-compute lxd regular
lxd rabbits juju-info regular
mysql mysql cluster peer
mysql nova-cloud-controller shared-db regular
nova-cloud-controller lxd container subordinate
nova-cloud-controller nova-cloud-controller cluster peer
nova-cloud-controller nova-compute cloud-compute regular
nova-cloud-controller rabbits amqp regular
nova-compute lxd lxd subordinate
nova-compute nova-compute compute-peer peer
nova-compute rabbits amqp regular
rabbits lxd container subordinate
rabbits rabbits cluster peer
wordpress wordpress loadbalancer peer

[Units]
ID WORKLOAD-STATUS JUJU-STATUS VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE
ceph/0 unk...

[Curtis Hovey (sinzui)]

'Failed to get device attributes: no such file or directory' is raised by LXD when it cannot create devices for the containers file system. Since LXD does work for you, I suspect Juju is configuring the container/file system wrongly.

[Cheryl Jennings (cherylj)]

If you're still getting "Waiting for agent initialization to finish", can you attach the contents of /var/log/juju/* for the machines hosting services that are still waiting?

[Dilip Renkila (dilip-renkila278)]

I am attaching the /var/lib/juju/containers/* files.They were no logs regarding to that machine ie.. container. But I observed that the containers are geting a valid ip from MAAS DHCP network and they are not pingable , shows failure downloading tools from juju state server. I noticed that agent tools were not downloaded so it shows a Waiting message.

Can you suggest me that where i have look for this agent networking part in juju source code?

[Dilip Renkila (dilip-renkila278)]

this is /var/log/juju/* .

Please give me a brief insight of how juju works when services are deployed into containers?

[Cheryl Jennings (cherylj)]

When you deploy to a container, the template container will be created first and used to clone the juju containers. Once the juju containers start, they will use the cloud-config from that file you posted in comment #8 to do some set up as part of cloud init.

That set up will configure the network, set up the system, and download and run the juju binaries. The output of those operations will be in /var/lib/juju/containers/<container-name>/console.log. It sounds like you saw there that the tools were failing to download?

If that's the case, it would be helpful to know:
- version of juju
- version of MAAS
- /etc/network/interfaces for the container that's failing to download tools

[Dilip Renkila (dilip-renkila278)]

####/etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual
  dns-nameservers 192.99.122.200
  dns-search maas
  pre-up ip address add 10.1.0.9/24 dev eth0 || true
  up ip route replace 10.1.0.0/24 dev eth0 || true
  down ip route del 10.1.0.0/24 dev eth0 || true
  post-down address del 10.1.0.9/24 dev eth0 || true
  up ip route replace default via 10.1.0.1 || true
  down ip route del default via 10.1.0.1 || true

####MAAS Version 2.0.0 (beta3+bzr4941)
####JUJU 2.0-beta7-xenial-amd64

[Cheryl Jennings (cherylj)]

Could you also attach the /var/lib/juju/containers/<name>/console.log for that container?

[Dilip Renkila (dilip-renkila278)]

I have attached console.log

[Cheryl Jennings (cherylj)]

Looking at the log, I see that we fail to get tools. Here are a couple things that stand out in the log:

Jun 5 14:49:06 juju-machine-1-lxc-0 pollinate[460]: ERROR: Network communication failed [28]\n14:48:46.294760 * Hostname was NOT found in DNS cache

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/Release.gpg Could not resolve 'archive.ubuntu.com'

Attempt 92 to download tools from https://10.1.0.254:17070/model/9e7966d9-8127-4b1d-87e7-ce882ab1a601/tools/2.0-beta7.1-trusty-amd64...
+ curl -sSfw tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s --noproxy * --insecure -o /var/lib/juju/tools/2.0-beta7.1-trusty-amd64/tools.tar.gz https://10.1.0.254:17070/model/9e7966d9-8127-4b1d-87e7-ce882ab1a601/tools/2.0-beta7.1-trusty-amd64
curl: (7) Failed to connect to 10.1.0.254 port 17070: No route to host
tools from https://10.1.0.254:17070/model/9e7966d9-8127-4b1d-87e7-ce882ab1a601/tools/2.0-beta7.1-trusty-amd64 downloaded: HTTP 000; time 2.203s; size 0 bytes; speed 0.000 bytes/s + echo Download failed, retrying in 15s
Download failed, retrying in 15s

I think this is a dup of bug 1575940, which will be fixed in the next beta release (estimated release is this Thursday, Jun 30)

[Cheryl Jennings (cherylj)]

@Dilip - you can confirm by inspecting /etc/resolv.conf in the container to see if there is a search domain