juju-deployer failed with "Deployment failed: 'units'"

[Ashley Lai]

When running a deployment of the bundle below, it failed with "Deployment failed" 'units'".

oil_deployment:
  default: true
  relations:
  - - keystone
    - mysql
  - - nova-cloud-controller
    - glance
  - - nova-cloud-controller
    - keystone
  - - nova-cloud-controller
    - mysql
  - - nova-cloud-controller
    - rabbitmq-server
  - - glance
    - keystone
  - - glance
    - mysql
  - - cinder
    - glance
  - - cinder
    - keystone
  - - cinder
    - nova-cloud-controller
  - - cinder
    - mysql
  - - cinder
    - rabbitmq-server
  - - openstack-dashboard
    - keystone
  - - heat
    - mysql
  - - heat
    - rabbitmq-server
  - - heat
    - keystone
  - - ceilometer:identity-service
    - keystone:identity-service
  - - ceilometer
    - rabbitmq-server
  - - ceilometer
    - mongodb
  - - ceilometer-agent
    - ceilometer
  - - nova-compute:amqp
    - rabbitmq-server:amqp
  - - nova-compute
    - nova-cloud-controller
  - - nova-compute
    - glance
  - - swift-proxy
    - keystone
  - - swift-proxy
    - swift-storage
  - - ceilometer-agent
    - nova-compute
  - - swift-proxy
    - glance
  - - nova-compute
    - mysql
  - - nova-compute
    - rabbitmq-server:amqp
  - - neutron-gateway
    - mysql
  - - neutron-gateway
    - nova-cloud-controller
  - - neutron-gateway:amqp
    - rabbitmq-server
  - - neutron-api
    - mysql
  - - neutron-api
    - rabbitmq-server
  - - neutron-api
    - nova-cloud-controller
  - - neutron-api
    - keystone
  - - vsm:dvs
    - vem:dvs
  - - vem
    - nova-compute
  - - vem
    - neutron-gateway
  series: trusty
  services:
    ceilometer:
      branch: lp:charms/ceilometer
      to:
      - lxc:nova-compute=0
    ceilometer-agent:
      branch: lp:charms/ceilometer-agent
    cinder:
      branch: lp:~openstack-charmers/charms/trusty/cinder/next
      options:
        block-device: /dev/sdb /srv/cinder.img|10G
        glance-api-version: 2
        overwrite: 'true'
        remove-missing: true
    glance:
      branch: lp:~openstack-charmers/charms/trusty/glance/next
      constraints: mem=1G
      to:
      - lxc:nova-compute=1
    heat:
      branch: lp:charms/heat
      to:
      - lxc:nova-compute=2
    keystone:
      branch: lp:~gnuoy/charms/trusty/keystone/next-1385105
      constraints: mem=1G
      options:
        admin-password: openstack
        admin-token: ubuntutesting
      to:
      - lxc:cinder=0
    mongodb:
      branch: lp:charms/mongodb
      to:
      - lxc:neutron-gateway=0
    mysql:
      branch: lp:~lutostag/charms/trusty/mysql/fix-six-bug
      constraints: mem=1G
      options:
        dataset-size: 50%
      to:
      - lxc:nova-cloud-controller=0
    neutron-api:
      branch: lp:~springfield-team/charms/trusty/neutron-api/n1kv
      constraints: mem=1G
      options:
        n1kv-vsm-ip: '10.245.0.152'
        n1kv-vsm-password: ubuntu
        n1kv-vsm-username: ubuntu
        neutron-plugin: n1kv
        neutron-security-groups: false
        worker-multiplier: 0
      to:
      - lxc:vem=0
    neutron-gateway:
      branch: lp:~openstack-charmers/charms/trusty/quantum-gateway/next
      constraints: mem=1G
      options:
        instance-mtu: 1300
        plugin: n1kv
    nova-cloud-controller:
      branch: lp:~openstack-charmers/charms/trusty/nova-cloud-controller/next
      constraints: mem=1G
      options:
        console-access-protocol: novnc
        network-manager: Quantum
        quantum-plugin: n1kv
        quantum-security-groups: 'no'
    nova-compute:
      branch: lp:~lutostag/charms/trusty/nova-compute/oil
      constraints: mem=4G
      num_units: 3
      options:
        config-flags: auto_assign_floating_ip=True,compute_driver=libvirt.LibvirtDriver
        enable-live-migration: true
        enable-resize: true
        migration-auth-type: none
        virt-type: kvm
    openstack-dashboard:
      branch: lp:charms/openstack-dashboard
      constraints: mem=1G
      options:
        profile: cisco
      to:
      - lxc:vsm=0
    rabbitmq-server:
      branch: lp:charms/rabbitmq-server
      constraints: mem=1G
      to:
      - lxc:nova-compute=0
    swift-proxy:
      branch: lp:~lmic/charms/trusty/swift-proxy/fix-six-bug
      options:
        replicas: 3
        swift-hash: fdfef9d4-8b06-11e2-8ac0-531c923c8fae
        zone-assignment: auto
      to:
      - lxc:nova-compute=1
    swift-storage:
      branch: lp:~openstack-charmers/charms/trusty/swift-storage/next
      num_units: 3
      options:
        block-device: /srv/swift.img|10G
        overwrite: 'true'
        zone: 1
      to:
      - nova-compute=0
      - nova-compute=1
      - nova-compute=2
    vem:
      branch: lp:charms/vem
      options:
        host_mgmt_intf: br0
        n1kv-source: ppa:cisco-n1kv/n1kv-updates
        n1kv-vsm-domain-id: 500
        n1kv-vsm-ip: '10.245.0.152'
        uplink_profile: phys eth1 profile sys-uplink
    vsm:
      branch: lp:charms/vsm
      options:
        n1kv-phy-intf-bridge: eth0
        n1kv-source: ppa:cisco-n1kv/n1kv-updates
        n1kv-vsm-domain-id: 500
        n1kv-vsm-mgmt-gateway: '10.245.0.10'
        n1kv-vsm-mgmt-ip: '10.245.0.152'
        n1kv-vsm-mgmt-netmask: '255.255.192.0'
        n1kv-vsm-name: vsm-primary
        n1kv-vsm-password: ubuntu
        n1kv-vsm-role: primary

[Kapil Thangavelu (hazmat)]

without the actual output/traceback this isn't very helpful.

[Ashley Lai (alai)]

Please see the traceback below. Also juju status does not show neutron-api so it is not deployed.

2015-02-23 18:27:55,614 [INFO] deployer.import: Deploying service mysql using local:trusty/mysql
2015-02-23 18:28:04,224 [INFO] deployer.import: Deploying service neutron-api using local:trusty/neutron-api
2015-02-23 18:28:04,227 [ERROR] oil_ci.deploy.oil_deployer: Deployment failed: 'units'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oil_ci/deploy/oil_deployer.py", line 140, in deploy
    debug=True, verbose=True)
  File "/usr/lib/python2.7/dist-packages/oil_ci/juju/juju_deployer.py", line 85, in run_deployer
    importer.Importer(env, deploy, options=opts).run()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 196, in run
    self.deploy_services()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 110, in deploy_services
    placement.get(0))
  File "/usr/lib/python2.7/dist-packages/deployer/service.py", line 141, in get
    svc_units = with_service['units']
KeyError: 'units'
2015-02-23 18:28:04,259 [INFO] oil_ci.deploy.oil_deployer: Saving deployment artifacts to: ./artifacts
+ exit

[Jason Hobbs (jason-hobbs)]

What version of juju-core and juju-deployer is this with?

[Ashley Lai (alai)]

jenkins@s9:~$ dpkg -l | grep juju-core
ii juju-core 1.21.3-0ubuntu1~12.04.1~juju1 Juju is devops distilled - client
ii python-jujuclient 0.50.1-2 Python API client for juju-core
jenkins@s9:~$ dpkg -l | grep juju-deployer
ii juju-deployer 0.4.3-0ubuntu1~ubuntu12.04.1~ppa1 Deploy complex stacks of services using Juju
jenkins@s9:~$

[David Britton (dpb)]

@alai: did this fail on a fresh install (clean bootstrap with nothing else), or where there empty services out there already? This is ringing a bell for me.

[David Britton (dpb)]

@alai: also, do you have a `juju status` from after or before it failed?

[Ashley Lai (alai)]

Here the 'juju status' after it failed (note the bundle for this deployment is on the bottom):

machines:
  "0":
    agent-state: started
    agent-version: 1.21.3
    dns-name: lambert.oil
    instance-id: /MAAS/api/1.0/nodes/node-f645c4dc-ae16-11e3-9074-00163efc5068/
    series: trusty
    hardware: arch=amd64 cpu-cores=8 mem=32768M tags=hardware-hp-proliant-DL320E,oil,bug1346547,hw-alai-staging
    state-server-member-status: has-vote
  "1":
    agent-state: started
    agent-version: 1.21.3
    dns-name: anahuac.oil
    instance-id: /MAAS/api/1.0/nodes/node-f626062e-ae16-11e3-b194-00163efc5068/
    series: trusty
    containers:
      1/lxc/0:
        agent-state: started
        agent-version: 1.21.3
        dns-name: 10.245.45.165
        instance-id: juju-machine-1-lxc-0
        series: trusty
        hardware: arch=amd64
    hardware: arch=amd64 cpu-cores=8 mem=32768M tags=hardware-hp-proliant-DL320E,disable_biosdevname,transient-error-3,hw-alai-staging
  "2":
    agent-state: started
    agent-version: 1.21.3
    dns-name: krastin.oil
    instance-id: /MAAS/api/1.0/nodes/node-f4a5be14-3f59-11e4-b38a-00163eca07b6/
    series: trusty
    containers:
      2/lxc/0:
        agent-state: started
        agent-version: 1.21.3
        dns-name: 10.245.38.196
        instance-id: juju-machine-2-lxc-0
        series: trusty
        hardware: arch=amd64
    hardware: arch=amd64 cpu-cores=4 mem=8192M tags=hardware-hp-proliant-DL360E-G8,hw-alai-staging
  "3":
    agent-state: started
    agent-version: 1.21.3
    dns-name: morgens.oil
    instance-id: /MAAS/api/1.0/nodes/node-0f5dc0ee-ae1e-11e3-9074-00163efc5068/
    series: trusty
    containers:
      3/lxc/0:
        agent-state: started
        agent-version: 1.21.3
        dns-name: 10.245.45.164
        instance-id: juju-machine-3-lxc-0
        series: trusty
        hardware: arch=amd64
    hardware: arch=amd64 cpu-cores=4 mem=32768M tags=hardware-dell-poweredge-R210,oil-slave-3,hw-alai-staging
  "4":
    agent-state: started
    agent-version: 1.21.3
    dns-name: gulon.oil
    instance-id: /MAAS/api/1.0/nodes/node-0f67ca28-ae17-11e3-b194-00163efc5068/
    series: trusty
    containers:
      4/lxc/0:
        agent-state: started
        agent-version: 1.21.3
        dns-name: 10.245.38.212
        instance-id: juju-machine-4-lxc-0
        series: trusty
        hardware: arch=amd64
    hardware: arch=amd64 cpu-cores=4 mem=32768M tags=oil-slave-1,hardware-hp-proliant-DL360E-G8,bug1346547,hw-alai-staging
  "5":
    agent-state: started
    agent-version: 1.21.3
    dns-name: suangi.oil
    instance-id: /MAAS/api/1.0/nodes/node-f665a2ca-ae16-11e3-9074-00163efc5068/
    series: trusty
    containers:
      5/lxc/0:
        agent-state: started
        agent-version: 1.21.3
        dns-name: 10.245.38.199
        instance-id: juju-machine-5-lxc-0
        series: trusty
        hardware: arch=amd64
    hardware: arch=amd64 cpu-cores=8 mem=32768M tags=hardware-hp-proliant-DL320E,oil-slave-3,bug1346547,hw-alai-staging
  "6":
    agent-state: started
    agent-version: 1.21.3
    dns-name: mier.oil
    instance-id: /MAAS/api/1.0/nodes/node-cb70a76c-f8bd-11e3-a5f6-00163eca07b6/
    series: trusty...

[Ashley Lai (alai)]

This was deployed on a clean bootstrap with nothing on it.

[Ashley Lai (alai)]

I'm hitting this error again on the Metaswitch deployment:

2015-03-11 21:11:38 [INFO] deployer.import: Deploying service neutron-api using local:trusty/neutron-api
Traceback (most recent call last):
  File "/usr/bin/juju-deployer", line 9, in <module>
    load_entry_point('juju-deployer==0.4.3', 'console_scripts', 'juju-deployer')()
  File "/usr/lib/python2.7/dist-packages/deployer/cli.py", line 130, in main
    run()
  File "/usr/lib/python2.7/dist-packages/deployer/cli.py", line 228, in run
    importer.Importer(env, deployment, options).run()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 196, in run
    self.deploy_services()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 110, in deploy_services
    placement.get(0))
  File "/usr/lib/python2.7/dist-packages/deployer/service.py", line 141, in get
    svc_units = with_service['units']
KeyError: 'units'

[Francesco Banconi (frankban)]

This seems to be related to the fact that, in the provided bundles, some service have unit placements pointing to a subordinate charm. Perhaps I am missing something, but this should not be allowed: since IIRC subordinates don't have units, pointing to them for unit placement does not make much sense.
For instance, the calico.yaml bundle has neutron-api placed to lxc:neutron-calico=0, the latter being a subordinate.

What's the expected deployer behavior in such cases? I assumed an error should be returned, but, as I said, I can be wrong.
Of course the error here is not helpful, so I attached a branch that improves unit placement validation, so that the bundle deployment fails sooner rather than later and with a more meaningful error, e.g.:
Cannot place to a subordinate service: neutron-api -> neutron-calico

[Ryan Harper (raharper)]

This isn't a deployer bug, it's when we attempt to place additional units against a subordinate charm. The fix is in the oil-ci code, specifically you need to update /etc/oil-ci/policy/placement.yaml and be sure you've listed all of the subordinate charms in your bundle.

[Tim Van Steenburgh (tvansteenburgh)]

r140 adds a check that prevents this (don't deploy bundle if unit placement points to subordinate).