Comment 2 for bug 1835823

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote : Re: maas reported 409 CONFLICT from "allocate" while the node matching constraints was available

Hmm, indeed. The "prometheus" application (prometheus2 charm) in the bundle does not even have the "internal" endpoint in its metadata yaml - there are a lot of endpoints that are related to nova-compute-kvm's metadata.yaml file.

Based on what I see it looks like a Juju bug. There were 6 nova-compute-kvm units but only 5 placement directives were specified in the bundle. While Juju reported that it will add a new machine for the 6th unit ("add unit nova-compute-kvm/5 to new machine 13") it actually used machine 3 which is allocated to prometheus.

nova-compute-kvm/5 waiting allocating 3 waiting for machine

prometheus/0 waiting allocating 3 waiting for machine

From the bundle:

variables:
  oam-space: &oam-space oam-space
  internal-space: &internal-space internal-space

machines: # see https://oil-jenkins.canonical.com/artifacts/808239c1-7b06-4281-9267-0f09421604a1/config/config/bundle.yaml
# ...

applications:
# ...

# 6 units, 5 machines which is is a bundle problem but either way Juju should have caught it
  nova-compute-kvm:
    charm: cs:nova-compute
    num_units: 6
    bindings:
      "": *oam-space
      internal: *internal-space
# ...
    to:
    - 1000
    - 1001
    - 1002
    - 1003
    - 1004

  prometheus:
    charm: cs:prometheus2
    series: bionic
    bindings:
      "": *oam-space
    num_units: 1
    to:
    - 9

Deployment-time messages:

19:54:03 DEBUG juju.cmd.juju.application bundle.go:838 created new machine 6 for holding ceph-mon, ceph-osd, ceph-radosgw, cinder, glance, heat, keystone, mysql, nova-compute-kvm and openstack-service-checks units

19:54:03 DEBUG juju.cmd.juju.application bundle.go:838 created new machine 7 for holding ceph-mon, ceph-osd, ceph-radosgw, cinder, designate-bind, glance, heat, keystone, mysql, nova-compute-kvm and prometheus-ceph-exporter units

19:54:04 DEBUG juju.cmd.juju.application bundle.go:838 created new machine 8 for holding ceph-mon, ceph-osd, ceph-radosgw, cinder, designate-bind, glance, heat, keystone, mysql, nova-compute-kvm and prometheus-openstack-exporter units

19:54:04 DEBUG juju.cmd.juju.application bundle.go:838 created new machine 9 for holding aodh, ceilometer, ceph-osd, designate, gnocchi, neutron-api, nova-cloud-controller, nova-compute-kvm, openstack-dashboard and rabbitmq-server units

19:54:04 DEBUG juju.cmd.juju.application bundle.go:838 created new machine 10 for holding ceph-osd and nova-compute-kvm units

19:54:34 - add unit nova-compute-kvm/0 to new machine 6

19:54:34 DEBUG juju.cmd.juju.application bundle.go:921 added nova-compute-kvm/0 unit to new machine
- add unit nova-compute-kvm/1 to new machine 7

19:54:35 DEBUG juju.cmd.juju.application - add unit nova-compute-kvm/2 to new machine 8
bundle.go:921 added nova-compute-kvm/1 unit to new machine

- add unit nova-compute-kvm/3 to new machine 9
19:54:35 DEBUG juju.cmd.juju.application bundle.go:921 added nova-compute-kvm/2 unit to new machine

- add unit nova-compute-kvm/4 to new machine 10
19:54:36 DEBUG juju.cmd.juju.application bundle.go:921 added nova-compute-kvm/3 unit to new machine

- add unit nova-compute-kvm/5 to new machine 13
19:54:36 DEBUG juju.cmd.juju.application bundle.go:921 added nova-compute-kvm/4 unit to new machine

19:54:34 - add unit nova-compute-kvm/0 to new machine 6

19:54:34 DEBUG juju.cmd.juju.application bundle.go:921 added nova-compute-kvm/0 unit to new machine
- add unit nova-compute-kvm/1 to new machine 7