juju deployment of systems that does not meet maas base constraints

[Larry Michel]

When doing an OpenStack deployment, our maas base constraints of hw-staging was totally ignored. All the servers selected for the bootstrap node and the OpenStack deployment did not meet the constraints.

Some console output showing maas base tag:
============================================================================
2015-01-14 22:55:14,199651: Base MAAS tag: hw-staging
2015-01-14 22:55:14,200009: converted slave label ci-oil-slave1 to maas user oil-slave-1
2015-01-14 22:55:15,111365: found 34 nodes with tag=hw-staging
2015-01-14 22:55:15,111848: restricting to 0 nodes owned by oil-slave-1
2015-01-14 22:55:15,112040: no nodes owned by oil-slave-1 in tag hw-staging
Exiting with: 0
+ rm -rf /var/lib/jenkins/.juju/environments
+ mv run_config artifacts/run_config.parameter
+ oil-ci generate run_config --out run_config --environment ci-oil-slave1
2015-01-14 22:55:15,548 [INFO] oil_ci.cli: Constructing parameters from jenkins job environment
2015-01-14 22:55:15,549 [INFO] oil_ci.cli: Generating run config with deploy parameters: {'block-storage': 'cinder-iscsi', 'networking': 'neutron-openvswitch', 'openstack-release': 'juno', 'image-storage': 'glance-ceph', 'ubuntu-release': 'trusty', 'compute': 'nova-kvm'}
2015-01-14 22:55:15,549 [INFO] oil_ci.cli: Generating run_config networking for env: ci-oil-slave1
2015-01-14 22:55:15,549 [INFO] oil_ci.cli: Allocating networks for env: ci-oil-slave1
2015-01-14 22:55:15,603 [INFO] oil_ci.cli: OIL Network Allocating 2 ranges of 32 IPs per environment
2015-01-14 22:55:15,618 [INFO] oil_ci.cli: Calculating deployment cost
2015-01-14 22:55:15,914 [INFO] oil_ci.placement.density: ceph units of 3 to machines: [1, 2, 3]
2015-01-14 22:55:15,915 [INFO] oil_ci.placement.density: Base services:
2015-01-14 22:55:15,915 [INFO] oil_ci.placement.density: 3 units of nova-compute to: ['ceph=0', 'ceph=1', 'ceph=2']
2015-01-14 22:55:15,915 [INFO] oil_ci.placement.density: LXCed services:
2015-01-14 22:55:15,915 [INFO] oil_ci.placement.density: 1 units of ceilometer to: ['lxc:ceph=0']
2015-01-14 22:55:15,915 [INFO] oil_ci.placement.density: 1 units of glance to: ['lxc:ceph=1']
2015-01-14 22:55:15,916 [INFO] oil_ci.placement.density: 1 units of heat to: ['lxc:ceph=2']
2015-01-14 22:55:15,916 [INFO] oil_ci.placement.density: 1 units of keystone to: ['lxc:cinder=0']
2015-01-14 22:55:15,916 [INFO] oil_ci.placement.density: 1 units of mongodb to: ['lxc:neutron-gateway=0']
2015-01-14 22:55:15,916 [INFO] oil_ci.placement.density: 1 units of mysql to: ['lxc:nova-cloud-controller=0']
2015-01-14 22:55:15,916 [INFO] oil_ci.placement.density: 1 units of neutron-api to: ['lxc:ceph=0']
2015-01-14 22:55:15,916 [INFO] oil_ci.placement.density: 1 units of openstack-dashboard to: ['lxc:ceph=1']
2015-01-14 22:55:15,917 [INFO] oil_ci.placement.density: 1 units of rabbitmq-server to: ['lxc:ceph=2']
2015-01-14 22:55:15,922 [INFO] oil_ci.placement.density: Deployment will require 7 machines (6 + bootstrap)
+ cp -av run_config artifacts/run_config
`run_config' -> `artifacts/run_config'
+ cp -av oil_nodes artifacts/oil_nodes
`oil_nodes' -> `artifacts/oil_nodes'
+ oil-ci do deploy --environment ci-oil-slave1 --run_config artifacts/run_config --artifacts_dir ./artifacts
2015-01-14 22:55:16,753 [INFO] oil_ci.deploy.oil_deployer: Bootstrapping new juju environment
2015-01-14 22:55:17,011 [INFO] oil_ci.juju.client: Boostrapping new environment
Bootstrapping environment "ci-oil-slave1"
Starting new instance for initial state server
Launching instance
WARNING no architecture was specified, acquiring an arbitrary node
 - /MAAS/api/1.0/nodes/node-8bdda5f8-6bb0-11e4-83f3-00163eca07b6/
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to mokoi-vm-02.oil:22
Attempting to connect to mokoi-vm-02.oil:22
============================================================================

For example (I'll attach entire yaml file as well):
============================================================================
environment: ci-oil-slave1
machines:
  '0':
    agent-state: started
    agent-version: 1.21-beta4
    dns-name: mokoi-vm-02.oil
    hardware: arch=amd64 cpu-cores=2 mem=12288M tags=virtual
    instance-id: /MAAS/api/1.0/nodes/node-8bdda5f8-6bb0-11e4-83f3-00163eca07b6/
    series: precise
    state-server-member-status: has-vote
  '1':
    agent-state: started
    agent-version: 1.21-beta4
    containers:
      1/lxc/0:
        agent-state: started
        agent-version: 1.21-beta4
        dns-name: 10.245.44.246
        hardware: arch=amd64
        instance-id: juju-machine-1-lxc-0
        series: trusty
      1/lxc/1:
        agent-state: started
        agent-version: 1.21-beta4
        dns-name: 10.245.44.248
        hardware: arch=amd64
        instance-id: juju-machine-1-lxc-1
        series: trusty
    dns-name: zahhak.oil
    hardware: arch=amd64 cpu-cores=4 mem=32768M tags=hardware-dell-poweredge-R210,disable_biosdevname,hw-standby,bug1398879
    instance-id: /MAAS/api/1.0/nodes/node-6491e170-ae10-11e3-9074-00163efc5068/
    series: trusty
==========================================================================

We are using juju 1.21-beta4 and maas version 1.7.1~rc1+bzr3313-0ubuntu1~trusty1

Note that this issue has not been observed with Juju 1.20.x and maas version 1.7.1~rc1+bzr3313-0ubuntu1~trusty1

[Larry Michel (lmic)]

juju-status.yaml file

[Larry Michel (lmic)]

output from 'maas root tag nodes -k hw-staging'

[John George (jog)]

I was able to confirm the reported behaviour.

1.20.14 honours '--contraints tag=...' and errors when the request can not be fulfilled.

$ juju --show-log bootstrap -e maas-env2-weekly --constraints tags=medium
2015-01-15 00:31:38 INFO juju.cmd supercommand.go:37 running juju [1.20.14-trusty-amd64 gc]
2015-01-15 00:31:43 INFO juju.environs.bootstrap bootstrap.go:48 bootstrapping environment "maas-env2-weekly"
2015-01-15 00:31:43 INFO juju.environs.tools tools.go:187 filtering tools by released version
2015-01-15 00:31:43 INFO juju.environs.tools tools.go:87 reading tools with major.minor version 1.20
2015-01-15 00:31:43 INFO juju.environs.tools tools.go:98 filtering tools by series: trusty
2015-01-15 00:31:43 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:31:43 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:31:43 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:31:43 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:31:43 INFO juju.environs.bootstrap bootstrap.go:60 newest version: 1.20.14
2015-01-15 00:31:44 INFO juju.environs.bootstrap bootstrap.go:88 picked bootstrap tools version: 1.20.14
Launching instance
2015-01-15 00:31:50 ERROR juju.provider.common bootstrap.go:136 bootstrap failed: cannot start bootstrap instance: cannot run instances: gomaasapi: got error back from server: 409 CONFLICT (No available node matches constraints: tags=medium)
Bootstrap failed, destroying environment
2015-01-15 00:31:50 INFO juju.provider.common destroy.go:15 destroying environment "maas-env2-weekly"
2015-01-15 00:31:50 ERROR juju.cmd supercommand.go:323 cannot start bootstrap instance: cannot run instances: gomaasapi: got error back from server: 409 CONFLICT (No available node matches constraints: tags=medium)

1.21-beta4 does not:
juju --show-log bootstrap -e maas-env2-weekly --constraints tags=medium
2015-01-15 00:34:35 INFO juju.cmd supercommand.go:37 running juju [1.21-beta4-trusty-amd64 gc]
2015-01-15 00:34:35 WARNING juju.environs config.go:127 Config attribute "tools-metadata-url" (https://swift.canonistack.canonical.com/v1/AUTH_526ad877f3e3464589dc1145dfeaac60/juju-dist/weekly/tools) is deprecated.
It is replaced by "agent-metadata-url" attribute.
Your configuration should be updated to set "agent-metadata-url" as follows
agent-metadata-url: https://swift.canonistack.canonical.com/v1/AUTH_526ad877f3e3464589dc1145dfeaac60/juju-dist/weekly/tools.
2015-01-15 00:34:35 INFO juju.network network.go:106 setting prefer-ipv6 to false
Bootstrapping environment "maas-env2-weekly"
2015-01-15 00:34:40 INFO juju.environs.tools tools.go:86 reading tools with major.minor version 1.21
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
Starting new instance for initial state server
Launching instance
.
.
...

[John George (jog)]

oops here is the full output from 1.21-beta4:

juju --show-log bootstrap -e maas-env2-weekly --constraints tags=medium
2015-01-15 00:34:35 INFO juju.cmd supercommand.go:37 running juju [1.21-beta4-trusty-amd64 gc]
2015-01-15 00:34:35 WARNING juju.environs config.go:127 Config attribute "tools-metadata-url" (https://swift.canonistack.canonical.com/v1/AUTH_526ad877f3e3464589dc1145dfeaac60/juju-dist/weekly/tools) is deprecated.
It is replaced by "agent-metadata-url" attribute.
Your configuration should be updated to set "agent-metadata-url" as follows
agent-metadata-url: https://swift.canonistack.canonical.com/v1/AUTH_526ad877f3e3464589dc1145dfeaac60/juju-dist/weekly/tools.
2015-01-15 00:34:35 INFO juju.network network.go:106 setting prefer-ipv6 to false
Bootstrapping environment "maas-env2-weekly"
2015-01-15 00:34:40 INFO juju.environs.tools tools.go:86 reading tools with major.minor version 1.21
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
Starting new instance for initial state server
Launching instance
2015-01-15 00:34:41 WARNING juju.provider.maas environ.go:576 no architecture was specified, acquiring an arbitrary node
2015-01-15 00:34:46 INFO juju.provider.maas environ.go:884 could not acquire a node in zone "default", trying another zone
2015-01-15 00:34:46 WARNING juju.provider.maas environ.go:576 no architecture was specified, acquiring an arbitrary node
 - /MAAS/api/1.0/nodes/node-4ddd31ea-9790-11e4-8d20-525400985998/
2015-01-15 00:34:47 INFO juju.environs.bootstrap bootstrap.go:178 newest version: 1.21-beta5
2015-01-15 00:34:47 WARNING juju.environs.bootstrap bootstrap.go:201 failed to find 1.21-beta4 tools, will attempt to use 1.21-beta5
2015-01-15 00:34:47 INFO juju.environs.bootstrap bootstrap.go:206 picked bootstrap tools version: 1.21-beta5
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to maas-node-107.maas:22
Attempting to connect to maas-node-107.maas:22
Attempting to connect to 10.0.60.120:22
Warning: Permanently added '10.0.60.120' (ECDSA) to the list of known hosts.
Logging to /var/log/cloud-init-output.log on remote host
Running apt-get update
Running apt-get upgrade
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://swift.canonistack.canonical.com/v1/AUTH_526ad877f3e3464589dc1145dfeaac60/juju-dist/weekly/tools/releases/juju-1.21-beta5-trusty-amd64.tgz]>
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
Bootstrap complete