juju deploy and add-machine not installing agent

[Keith Lee]

Versions
   - Ubuntu Server 14.04.3 LTS
   - juju 1.24.4
   - maas 1.8.0+bzr4001

Steps
   - MAAS installed
   - nodes commisioned and ready
   - local copy of tools and charms created
   - ran "juju bootstrap --upload-tools --debug" on MAAS
   - node bootstraps but then reports "/var/lib/juju/nonce.txt does not exist"
   - ssh on to machine 0 and create nonce.txt with contents of "user-admin:bootstrap"
   - bootstrap then continues and sucessfully completes
   - run "juju deploy --repository=/home/keith/charms local:trusty/mysql --debug"
   - juju status is then stuck at pending/allocating/Waiting for agent initialization to finish. see output below
   - can ssh to machine 1
   - MAAS, machine 0 and 1 can all sucessfully ping each other using public-address
   - There are no machine 1 logs on machine 0 (juju)
   - There are no juju logs on machine 1
   - Same behaviour for other charms
   - Same behaviour if use "juju add-machine"
   - Charms deploy sucessfuly if deployed to already running machine 0 using the --to=0

juju status output
    environment: maas
    machines:
      "0":
        agent-state: started
        agent-version: 1.24.4.1
        dns-name: gracious-rabbits.maas
        instance-id: /MAAS/api/1.0/nodes/node-8af709d4-3b82-11e5-8107-005056bc64c7/
        series: trusty
        hardware: arch=amd64 cpu-cores=1 mem=1024M
        state-server-member-status: has-vote
      "1":
        agent-state: pending
        dns-name: first-circle.maas
        instance-id: /MAAS/api/1.0/nodes/node-468e9dac-3b87-11e5-89b6-005056bc64c7/
        series: trusty
        hardware: arch=amd64 cpu-cores=1 mem=1024M
    services:
      mysql:
        charm: local:trusty/mysql-326
        exposed: false
        service-status:
          current: unknown
          message: Waiting for agent initialization to finish
          since: 13 Aug 2015 22:29:26+01:00
        relations:
          cluster:
          - mysql
        units:
          mysql/0:
            workload-status:
              current: unknown
              message: Waiting for agent initialization to finish
              since: 13 Aug 2015 22:29:26+01:00
            agent-status:
              current: allocating
              since: 13 Aug 2015 22:29:26+01:00
            agent-state: pending
            machine: "1"
            public-address: first-circle.maas
    networks:
      maas-eth0:
        provider-id: maas-eth0
        cidr: 192.168.248.0/24

environments.yaml
    default: maas
    environments:
        maas:
            type: maas
            maas-server: 'http://192.168.248.2/MAAS/'
            maas-oauth: 'vwxwGddnHH7KNwVZwy:NPezs7Gr2M4Zx6qSEd:F5QXRdnTcntPLcmq6dK2Mxd8mphwR9p4'
            authorized-keys-path: ~/.ssh/id_rsa.pub
            bootstrap-timeout: 600
            admin-secret: xxxxxxx
            agent-metadata-url: http://192.168.248.2/tools

cloud-init-ouput.log from machine 1
http://pastebin.com/i2VPpcnv

[Keith Lee (keith-lee-m)]

Retested above with juju 1.22.6 and no issues with that earlier release

[Ryan Beisner (1chb1n)]

Linking AU! post just for reference:
http://askubuntu.com/questions/661170/juju-deploy-not-installing-juju-agent

[Ian Booth (wallyworld)]

The attached cloud init log for machine 1 appears to compete successfully. We'd need to see Juju's logs for machine 1 (and preferably also machine 0) to figure out what's happening.

[Manoj Iyer (manjo)]

This problem exists in juju version 2.0-alpha1-xenial-arm64

$ apt-cache policy juju-core
juju-core:
  Installed: 2.0-alpha1-0ubuntu1~16.04.1~juju1

I have to add-machine first and the deploy the charm. If I just deploy the charm the machine "agent-state: pending" prevents the charm from finishing deployment.

[Manoj Iyer (manjo)]

Also, This is with the LXD provider. Where LXD version is:

$ apt-cache policy lxd
lxd:
  Installed: 2.0.0~beta1-0ubuntu3

and .. bootstrap node was xenial machines (units) were trusty.

[Cheryl Jennings (cherylj)]

@manjo - can you attach the machine-*.log files from the machines that aren't coming up (if they exist), and the /var/log/cloud-init-output.log for those machines?

[Manoj Iyer (manjo)]

@cheryl, after I did a host distupgrade (xenial), synced lxd-images, I no longer have this issue as of this afternoon. "juju deploy charm" works like a charm without needing to do an "add machine". Retested this a couple of times by destroying service and redeploying and also destroying the controller, rebootstrap, redeploy. In all cases it works now.

[Launchpad Janitor (janitor)]

[Expired for juju-core because there has been no activity for 60 days.]