[juju 1.25] agent fails to install on xenial node

[Martin Pitt]

I'm trying to deploy a xenial-based service in my prod-ues-proposed environment in Canonical Prodstack. I've done this in the past a few times, so this worked before. Now it fails to install the agent on the newly created machine, even before it gets to my actual charm:

$ juju status autopkgtest-lxd-worker
environment: prod-ues-proposed-migration
machines:
  "7":
    agent-state: started
    agent-version: 1.25.6
    dns-name: 10.25.180.152
    instance-id: 15543edf-f2da-449c-b21e-06d5c8a278fd
    instance-state: ACTIVE
    series: xenial
    hardware: arch=amd64 cpu-cores=1 mem=8192M root-disk=51200M availability-zone=nova
services:
  autopkgtest-lxd-worker:
    charm: local:xenial/autopkgtest-cloud-worker-1
    exposed: false
    service-status:
      current: unknown
      message: Waiting for agent initialization to finish
      since: 29 Sep 2016 14:21:26Z
    relations:
      amqp:
      - rabbitmq-server
      juju-info:
      - ksplice
      - landscape-client
    units:
      autopkgtest-lxd-worker/3:
        workload-status:
          current: unknown
          message: Waiting for agent initialization to finish
          since: 29 Sep 2016 14:21:26Z
        agent-status:
          current: failed
          message: install local:xenial/autopkgtest-cloud-worker-1
          since: 29 Sep 2016 14:24:26Z
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "7"
        public-address: 10.25.180.152

There is no ERROR in machine-7.log (but I'll attach it anyway), but in unit-autopkgtest-lxd-worker-3.log I see the following:

2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "upgrader" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "leadership-tracker" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "uniter" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "api-caller" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "proxy-config-updater" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.logsender manifold.go:32 log sender manifold disabled by feature flag
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "log-sender" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "rsyslog-config-updater" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "logging-config-updater" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "api-address-updater" manifold worker: dependency not available
ubuntu@juju-prod-ues-proposed-migration-machine-7:/var/log/juju$ sudo egrep 'WARN|ERR' unit-autopkgtest-lxd-worker-3.log|head -n 30
^C
ubuntu@juju-prod-ues-proposed-migration-machine-7:/var/log/juju$ sudo egrep 'WARN|ERR' unit-autopkgtest-lxd-worker-3.log|head -n30
sudo: unable to resolve host juju-prod-ues-proposed-migration-machine-7
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "upgrader" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "leadership-tracker" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "uniter" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "api-caller" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "proxy-config-updater" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.logsender manifold.go:32 log sender manifold disabled by feature flag
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "log-sender" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "rsyslog-config-updater" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "logging-config-updater" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "api-address-updater" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "uniter" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "upgrader" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "leadership-tracker" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.logsender manifold.go:32 log sender manifold disabled by feature flag
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "log-sender" manifold worker: dependency not available
2016-09-29 14:23:29 WARNING juju.worker.dependency engine.go:304 failed to start "uniter" manifold worker: dependency not available
2016-09-29 14:23:29 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "<root>=DEBUG" to "<root>=WARNING;unit=DEBUG"
2016-09-29 14:23:30 ERROR juju.worker.uniter.filter filter.go:137 tomb: dying

The last line repeats over and over, and from then on the state is stable.

$ juju --version
1.25.6-trusty-amd64

[Martin Pitt (pitti)]

This instance is using the basenode, landscape-client, and ksplice default charms, in case it makes a difference.

The actual charm and deployment script is in https://git.launchpad.net/~ubuntu-release/+git/autopkgtest-cloud/tree/deployment (but as it doesn't really get to that yet I suppose it's not very relevant).

[Martin Pitt (pitti)]

FTR, I naïvely tried

$ juju resolved --retry autopkgtest-lxd-worker/3
ERROR unit "autopkgtest-lxd-worker/3" is not in an error state

[Junien F (axino)]

Martin, could this be related to https://bugs.launchpad.net/juju-core/1.25/+bug/1613992 and/or https://bugs.launchpad.net/juju-core/+bug/1626304 ?

[Martin Pitt (pitti)]

Confirmed, the workaround in bug 1626304 did it, thanks for pointing out!