| 2019-07-08 18:26:42 |
Dmitrii Shcherbakov |
bug |
|
|
added bug |
| 2019-07-08 20:59:29 |
Dmitrii Shcherbakov |
description |
https://solutions.qa.canonical.com/#/qa/testRun/808239c1-7b06-4281-9267-0f09421604a1
https://oil-jenkins.canonical.com/artifacts/808239c1-7b06-4281-9267-0f09421604a1/index.html (deployment artifacts)
Juju tried 10 times to allocate a machine with tag "prometheus" but MAAS responded with 509s to all requests so Juju eventually marked the machine as failed. The machine is a pod VM (the host is in zone-3). Based on the DB dump it looks like it exists, is ready, has the right tag and interface with access to the right space.
8cf17ecd-809f-4c14-8220-09b6b66bbc74: machine-0 2019-07-05 19:55:24 WARNING juju.worker.provisioner provisioner_task.go:1157 failed to start machine 3 (failed to acquire node: No available machine matches constraints: [('agent_name', ['85f7eddb-f1bb-45fd-8eba-e9e50b532eda']), ('interfaces', ['internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3']), ('tags', ['prometheus']), ('zone', ['zone3'])] (resolved to "interfaces=internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3 tags=prometheus zone=zone3")), retrying in 10s (10 more attempts)
2019-07-05 19:55:23 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 200 OK (referrer: -; agent: Go-http-client/1.1)
2019-07-05 19:55:24 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 409 CONFLICT (referrer: -; agent: Go-http-client/1.1)
select x.hostname,x.status,x.zone_id,x.name from (select * from maasserver_node inner join maasserver_zone on maasserver_node.zone_id = maasserver_zone.id) as x where x.hostname LIKE '%prometheus%';
hostname | status | zone_id | name
--------------+--------+---------+-------
prometheus-3 | 4 | 3 | zone3
(1 row)
maasdb=# select id from maasserver_node where hostname = 'prometheus-3';
id
----
13
(1 row)
maasdb=# select * from maasserver_space;
id | created | updated | name | description
----+-------------------------------+-------------------------------+--------------------+-----------------------------------------------------
1 | 2019-07-05 19:02:02.797481+00 | 2019-07-05 19:02:02.797481+00 | ps45_routers | foundation-engine created space: ps45_routers
2 | 2019-07-05 19:02:03.603296+00 | 2019-07-05 19:02:03.603296+00 | internal-space | foundation-engine created space: internal-space
3 | 2019-07-05 19:02:05.074262+00 | 2019-07-05 19:02:05.074262+00 | oam-space | foundation-engine created space: oam-space
4 | 2019-07-05 19:02:06.108567+00 | 2019-07-05 19:02:06.108567+00 | external-space | foundation-engine created space: external-space
5 | 2019-07-05 19:02:06.922964+00 | 2019-07-05 19:02:06.922964+00 | ceph-replica-space | foundation-engine created space: ceph-replica-space
6 | 2019-07-05 19:02:07.80897+00 | 2019-07-05 19:02:07.80897+00 | ceph-access-space | foundation-engine created space: ceph-access-space
(6 rows)
maasdb=# select vlan_id from maasserver_interface where node_id = 13;
vlan_id
---------
5002
(1 row)
maasdb=# select id,space_id from maasserver_vlan where id = 5002;
id | space_id
------+----------
5002 | 3
(1 row)
maasdb=# select node_id,tag_id from maasserver_node inner join maasserver_node_tags on maasserver_node.id = maasserver_node_tags.node_id where maasserver_node.id = 13;
node_id | tag_id
---------+--------
13 | 2
13 | 1
13 | 9
(3 rows)
maasdb=# select * from maasserver_tag where name like '%prometheus%';
id | created | updated | name | definition | comment | kernel_opts
----+-------------------------------+-------------------------------+------------+------------+---------+-------------
9 | 2019-07-05 19:03:58.501961+00 | 2019-07-05 19:03:58.501961+00 | prometheus | | |
(1 row) |
https://solutions.qa.canonical.com/#/qa/testRun/808239c1-7b06-4281-9267-0f09421604a1
https://oil-jenkins.canonical.com/artifacts/808239c1-7b06-4281-9267-0f09421604a1/index.html (deployment artifacts)
Juju tried 10 times to allocate a machine with tag "prometheus" but MAAS responded with 509s to all requests so Juju eventually marked the machine as failed. The machine is a pod VM (the host is in zone-3). Based on the DB dump it looks like it exists, is ready, has the right tag and interface with access to the right space.
8cf17ecd-809f-4c14-8220-09b6b66bbc74: machine-0 2019-07-05 19:55:24 WARNING juju.worker.provisioner provisioner_task.go:1157 failed to start machine 3 (failed to acquire node: No available machine matches constraints: [('agent_name', ['85f7eddb-f1bb-45fd-8eba-e9e50b532eda']), ('interfaces', ['internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3']), ('tags', ['prometheus']), ('zone', ['zone3'])] (resolved to "interfaces=internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3 tags=prometheus zone=zone3")), retrying in 10s (10 more attempts)
2019-07-05 19:55:23 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 200 OK (referrer: -; agent: Go-http-client/1.1)
2019-07-05 19:55:24 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 409 CONFLICT (referrer: -; agent: Go-http-client/1.1)
select x.hostname,x.status,x.zone_id,x.name from (select * from maasserver_node inner join maasserver_zone on maasserver_node.zone_id = maasserver_zone.id) as x where x.hostname LIKE '%prometheus%';
hostname | status | zone_id | name
--------------+--------+---------+-------
prometheus-3 | 4 | 3 | zone3
(1 row)
src/maasserver/enum.py
class NODE_STATUS:
# ...
READY = 4
#: The node is ready for named deployment.
maasdb=# select id from maasserver_node where hostname = 'prometheus-3';
id
----
13
(1 row)
maasdb=# select * from maasserver_space;
id | created | updated | name | description
----+-------------------------------+-------------------------------+--------------------+-----------------------------------------------------
1 | 2019-07-05 19:02:02.797481+00 | 2019-07-05 19:02:02.797481+00 | ps45_routers | foundation-engine created space: ps45_routers
2 | 2019-07-05 19:02:03.603296+00 | 2019-07-05 19:02:03.603296+00 | internal-space | foundation-engine created space: internal-space
3 | 2019-07-05 19:02:05.074262+00 | 2019-07-05 19:02:05.074262+00 | oam-space | foundation-engine created space: oam-space
4 | 2019-07-05 19:02:06.108567+00 | 2019-07-05 19:02:06.108567+00 | external-space | foundation-engine created space: external-space
5 | 2019-07-05 19:02:06.922964+00 | 2019-07-05 19:02:06.922964+00 | ceph-replica-space | foundation-engine created space: ceph-replica-space
6 | 2019-07-05 19:02:07.80897+00 | 2019-07-05 19:02:07.80897+00 | ceph-access-space | foundation-engine created space: ceph-access-space
(6 rows)
maasdb=# select vlan_id from maasserver_interface where node_id = 13;
vlan_id
---------
5002
(1 row)
maasdb=# select id,space_id from maasserver_vlan where id = 5002;
id | space_id
------+----------
5002 | 3
(1 row)
maasdb=# select node_id,tag_id from maasserver_node inner join maasserver_node_tags on maasserver_node.id = maasserver_node_tags.node_id where maasserver_node.id = 13;
node_id | tag_id
---------+--------
13 | 2
13 | 1
13 | 9
(3 rows)
maasdb=# select * from maasserver_tag where name like '%prometheus%';
id | created | updated | name | definition | comment | kernel_opts
----+-------------------------------+-------------------------------+------------+------------+---------+-------------
9 | 2019-07-05 19:03:58.501961+00 | 2019-07-05 19:03:58.501961+00 | prometheus | | |
(1 row) |
|
| 2019-08-19 16:00:34 |
Adam Collard |
maas: assignee |
|
Newell Jensen (newell-jensen) |
|
| 2019-08-22 16:40:21 |
Adam Collard |
maas: status |
New |
Incomplete |
|
| 2019-08-22 16:40:23 |
Adam Collard |
maas: status |
Incomplete |
Triaged |
|
| 2019-08-22 16:40:28 |
Adam Collard |
maas: importance |
Undecided |
High |
|
| 2019-08-22 16:40:32 |
Newell Jensen |
maas: status |
Triaged |
In Progress |
|
| 2019-08-22 16:40:41 |
Newell Jensen |
maas: milestone |
|
2.7.0alpha1 |
|
| 2019-08-27 17:38:32 |
Newell Jensen |
maas: status |
In Progress |
Incomplete |
|
| 2019-08-28 17:36:08 |
Dmitrii Shcherbakov |
description |
https://solutions.qa.canonical.com/#/qa/testRun/808239c1-7b06-4281-9267-0f09421604a1
https://oil-jenkins.canonical.com/artifacts/808239c1-7b06-4281-9267-0f09421604a1/index.html (deployment artifacts)
Juju tried 10 times to allocate a machine with tag "prometheus" but MAAS responded with 509s to all requests so Juju eventually marked the machine as failed. The machine is a pod VM (the host is in zone-3). Based on the DB dump it looks like it exists, is ready, has the right tag and interface with access to the right space.
8cf17ecd-809f-4c14-8220-09b6b66bbc74: machine-0 2019-07-05 19:55:24 WARNING juju.worker.provisioner provisioner_task.go:1157 failed to start machine 3 (failed to acquire node: No available machine matches constraints: [('agent_name', ['85f7eddb-f1bb-45fd-8eba-e9e50b532eda']), ('interfaces', ['internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3']), ('tags', ['prometheus']), ('zone', ['zone3'])] (resolved to "interfaces=internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3 tags=prometheus zone=zone3")), retrying in 10s (10 more attempts)
2019-07-05 19:55:23 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 200 OK (referrer: -; agent: Go-http-client/1.1)
2019-07-05 19:55:24 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 409 CONFLICT (referrer: -; agent: Go-http-client/1.1)
select x.hostname,x.status,x.zone_id,x.name from (select * from maasserver_node inner join maasserver_zone on maasserver_node.zone_id = maasserver_zone.id) as x where x.hostname LIKE '%prometheus%';
hostname | status | zone_id | name
--------------+--------+---------+-------
prometheus-3 | 4 | 3 | zone3
(1 row)
src/maasserver/enum.py
class NODE_STATUS:
# ...
READY = 4
#: The node is ready for named deployment.
maasdb=# select id from maasserver_node where hostname = 'prometheus-3';
id
----
13
(1 row)
maasdb=# select * from maasserver_space;
id | created | updated | name | description
----+-------------------------------+-------------------------------+--------------------+-----------------------------------------------------
1 | 2019-07-05 19:02:02.797481+00 | 2019-07-05 19:02:02.797481+00 | ps45_routers | foundation-engine created space: ps45_routers
2 | 2019-07-05 19:02:03.603296+00 | 2019-07-05 19:02:03.603296+00 | internal-space | foundation-engine created space: internal-space
3 | 2019-07-05 19:02:05.074262+00 | 2019-07-05 19:02:05.074262+00 | oam-space | foundation-engine created space: oam-space
4 | 2019-07-05 19:02:06.108567+00 | 2019-07-05 19:02:06.108567+00 | external-space | foundation-engine created space: external-space
5 | 2019-07-05 19:02:06.922964+00 | 2019-07-05 19:02:06.922964+00 | ceph-replica-space | foundation-engine created space: ceph-replica-space
6 | 2019-07-05 19:02:07.80897+00 | 2019-07-05 19:02:07.80897+00 | ceph-access-space | foundation-engine created space: ceph-access-space
(6 rows)
maasdb=# select vlan_id from maasserver_interface where node_id = 13;
vlan_id
---------
5002
(1 row)
maasdb=# select id,space_id from maasserver_vlan where id = 5002;
id | space_id
------+----------
5002 | 3
(1 row)
maasdb=# select node_id,tag_id from maasserver_node inner join maasserver_node_tags on maasserver_node.id = maasserver_node_tags.node_id where maasserver_node.id = 13;
node_id | tag_id
---------+--------
13 | 2
13 | 1
13 | 9
(3 rows)
maasdb=# select * from maasserver_tag where name like '%prometheus%';
id | created | updated | name | definition | comment | kernel_opts
----+-------------------------------+-------------------------------+------------+------------+---------+-------------
9 | 2019-07-05 19:03:58.501961+00 | 2019-07-05 19:03:58.501961+00 | prometheus | | |
(1 row) |
Re-targeted to Juju based on https://bugs.launchpad.net/maas/+bug/1835823/comments/2
Old title: "maas reported 409 CONFLICT from "allocate" while the node matching constraints was available"
https://solutions.qa.canonical.com/#/qa/testRun/808239c1-7b06-4281-9267-0f09421604a1
https://oil-jenkins.canonical.com/artifacts/808239c1-7b06-4281-9267-0f09421604a1/index.html (deployment artifacts)
Juju tried 10 times to allocate a machine with tag "prometheus" but MAAS responded with 509s to all requests so Juju eventually marked the machine as failed. The machine is a pod VM (the host is in zone-3). Based on the DB dump it looks like it exists, is ready, has the right tag and interface with access to the right space.
8cf17ecd-809f-4c14-8220-09b6b66bbc74: machine-0 2019-07-05 19:55:24 WARNING juju.worker.provisioner provisioner_task.go:1157 failed to start machine 3 (failed to acquire node: No available machine matches constraints: [('agent_name', ['85f7eddb-f1bb-45fd-8eba-e9e50b532eda']), ('interfaces', ['internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3']), ('tags', ['prometheus']), ('zone', ['zone3'])] (resolved to "interfaces=internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3 tags=prometheus zone=zone3")), retrying in 10s (10 more attempts)
2019-07-05 19:55:23 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 200 OK (referrer: -; agent: Go-http-client/1.1)
2019-07-05 19:55:24 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 409 CONFLICT (referrer: -; agent: Go-http-client/1.1)
select x.hostname,x.status,x.zone_id,x.name from (select * from maasserver_node inner join maasserver_zone on maasserver_node.zone_id = maasserver_zone.id) as x where x.hostname LIKE '%prometheus%';
hostname | status | zone_id | name
--------------+--------+---------+-------
prometheus-3 | 4 | 3 | zone3
(1 row)
src/maasserver/enum.py
class NODE_STATUS:
# ...
READY = 4
#: The node is ready for named deployment.
maasdb=# select id from maasserver_node where hostname = 'prometheus-3';
id
----
13
(1 row)
maasdb=# select * from maasserver_space;
id | created | updated | name | description
----+-------------------------------+-------------------------------+--------------------+-----------------------------------------------------
1 | 2019-07-05 19:02:02.797481+00 | 2019-07-05 19:02:02.797481+00 | ps45_routers | foundation-engine created space: ps45_routers
2 | 2019-07-05 19:02:03.603296+00 | 2019-07-05 19:02:03.603296+00 | internal-space | foundation-engine created space: internal-space
3 | 2019-07-05 19:02:05.074262+00 | 2019-07-05 19:02:05.074262+00 | oam-space | foundation-engine created space: oam-space
4 | 2019-07-05 19:02:06.108567+00 | 2019-07-05 19:02:06.108567+00 | external-space | foundation-engine created space: external-space
5 | 2019-07-05 19:02:06.922964+00 | 2019-07-05 19:02:06.922964+00 | ceph-replica-space | foundation-engine created space: ceph-replica-space
6 | 2019-07-05 19:02:07.80897+00 | 2019-07-05 19:02:07.80897+00 | ceph-access-space | foundation-engine created space: ceph-access-space
(6 rows)
maasdb=# select vlan_id from maasserver_interface where node_id = 13;
vlan_id
---------
5002
(1 row)
maasdb=# select id,space_id from maasserver_vlan where id = 5002;
id | space_id
------+----------
5002 | 3
(1 row)
maasdb=# select node_id,tag_id from maasserver_node inner join maasserver_node_tags on maasserver_node.id = maasserver_node_tags.node_id where maasserver_node.id = 13;
node_id | tag_id
---------+--------
13 | 2
13 | 1
13 | 9
(3 rows)
maasdb=# select * from maasserver_tag where name like '%prometheus%';
id | created | updated | name | definition | comment | kernel_opts
----+-------------------------------+-------------------------------+------------+------------+---------+-------------
9 | 2019-07-05 19:03:58.501961+00 | 2019-07-05 19:03:58.501961+00 | prometheus | | |
(1 row) |
|
| 2019-08-28 17:36:48 |
Dmitrii Shcherbakov |
summary |
maas reported 409 CONFLICT from "allocate" while the node matching constraints was available |
Juju incorrectly placed a unit onto an existing machine |
|
| 2019-08-28 17:37:03 |
Dmitrii Shcherbakov |
bug task added |
|
juju |
|
| 2019-08-28 17:38:30 |
Dmitrii Shcherbakov |
description |
Re-targeted to Juju based on https://bugs.launchpad.net/maas/+bug/1835823/comments/2
Old title: "maas reported 409 CONFLICT from "allocate" while the node matching constraints was available"
https://solutions.qa.canonical.com/#/qa/testRun/808239c1-7b06-4281-9267-0f09421604a1
https://oil-jenkins.canonical.com/artifacts/808239c1-7b06-4281-9267-0f09421604a1/index.html (deployment artifacts)
Juju tried 10 times to allocate a machine with tag "prometheus" but MAAS responded with 509s to all requests so Juju eventually marked the machine as failed. The machine is a pod VM (the host is in zone-3). Based on the DB dump it looks like it exists, is ready, has the right tag and interface with access to the right space.
8cf17ecd-809f-4c14-8220-09b6b66bbc74: machine-0 2019-07-05 19:55:24 WARNING juju.worker.provisioner provisioner_task.go:1157 failed to start machine 3 (failed to acquire node: No available machine matches constraints: [('agent_name', ['85f7eddb-f1bb-45fd-8eba-e9e50b532eda']), ('interfaces', ['internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3']), ('tags', ['prometheus']), ('zone', ['zone3'])] (resolved to "interfaces=internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3 tags=prometheus zone=zone3")), retrying in 10s (10 more attempts)
2019-07-05 19:55:23 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 200 OK (referrer: -; agent: Go-http-client/1.1)
2019-07-05 19:55:24 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 409 CONFLICT (referrer: -; agent: Go-http-client/1.1)
select x.hostname,x.status,x.zone_id,x.name from (select * from maasserver_node inner join maasserver_zone on maasserver_node.zone_id = maasserver_zone.id) as x where x.hostname LIKE '%prometheus%';
hostname | status | zone_id | name
--------------+--------+---------+-------
prometheus-3 | 4 | 3 | zone3
(1 row)
src/maasserver/enum.py
class NODE_STATUS:
# ...
READY = 4
#: The node is ready for named deployment.
maasdb=# select id from maasserver_node where hostname = 'prometheus-3';
id
----
13
(1 row)
maasdb=# select * from maasserver_space;
id | created | updated | name | description
----+-------------------------------+-------------------------------+--------------------+-----------------------------------------------------
1 | 2019-07-05 19:02:02.797481+00 | 2019-07-05 19:02:02.797481+00 | ps45_routers | foundation-engine created space: ps45_routers
2 | 2019-07-05 19:02:03.603296+00 | 2019-07-05 19:02:03.603296+00 | internal-space | foundation-engine created space: internal-space
3 | 2019-07-05 19:02:05.074262+00 | 2019-07-05 19:02:05.074262+00 | oam-space | foundation-engine created space: oam-space
4 | 2019-07-05 19:02:06.108567+00 | 2019-07-05 19:02:06.108567+00 | external-space | foundation-engine created space: external-space
5 | 2019-07-05 19:02:06.922964+00 | 2019-07-05 19:02:06.922964+00 | ceph-replica-space | foundation-engine created space: ceph-replica-space
6 | 2019-07-05 19:02:07.80897+00 | 2019-07-05 19:02:07.80897+00 | ceph-access-space | foundation-engine created space: ceph-access-space
(6 rows)
maasdb=# select vlan_id from maasserver_interface where node_id = 13;
vlan_id
---------
5002
(1 row)
maasdb=# select id,space_id from maasserver_vlan where id = 5002;
id | space_id
------+----------
5002 | 3
(1 row)
maasdb=# select node_id,tag_id from maasserver_node inner join maasserver_node_tags on maasserver_node.id = maasserver_node_tags.node_id where maasserver_node.id = 13;
node_id | tag_id
---------+--------
13 | 2
13 | 1
13 | 9
(3 rows)
maasdb=# select * from maasserver_tag where name like '%prometheus%';
id | created | updated | name | definition | comment | kernel_opts
----+-------------------------------+-------------------------------+------------+------------+---------+-------------
9 | 2019-07-05 19:03:58.501961+00 | 2019-07-05 19:03:58.501961+00 | prometheus | | |
(1 row) |
Re-targeted to Juju (2.6.5) based on https://bugs.launchpad.net/maas/+bug/1835823/comments/2
Old title: "maas reported 409 CONFLICT from "allocate" while the node matching constraints was available"
https://solutions.qa.canonical.com/#/qa/testRun/808239c1-7b06-4281-9267-0f09421604a1
https://oil-jenkins.canonical.com/artifacts/808239c1-7b06-4281-9267-0f09421604a1/index.html (deployment artifacts)
Juju tried 10 times to allocate a machine with tag "prometheus" but MAAS responded with 509s to all requests so Juju eventually marked the machine as failed. The machine is a pod VM (the host is in zone-3). Based on the DB dump it looks like it exists, is ready, has the right tag and interface with access to the right space.
8cf17ecd-809f-4c14-8220-09b6b66bbc74: machine-0 2019-07-05 19:55:24 WARNING juju.worker.provisioner provisioner_task.go:1157 failed to start machine 3 (failed to acquire node: No available machine matches constraints: [('agent_name', ['85f7eddb-f1bb-45fd-8eba-e9e50b532eda']), ('interfaces', ['internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3']), ('tags', ['prometheus']), ('zone', ['zone3'])] (resolved to "interfaces=internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3 tags=prometheus zone=zone3")), retrying in 10s (10 more attempts)
2019-07-05 19:55:23 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 200 OK (referrer: -; agent: Go-http-client/1.1)
2019-07-05 19:55:24 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 409 CONFLICT (referrer: -; agent: Go-http-client/1.1)
select x.hostname,x.status,x.zone_id,x.name from (select * from maasserver_node inner join maasserver_zone on maasserver_node.zone_id = maasserver_zone.id) as x where x.hostname LIKE '%prometheus%';
hostname | status | zone_id | name
--------------+--------+---------+-------
prometheus-3 | 4 | 3 | zone3
(1 row)
src/maasserver/enum.py
class NODE_STATUS:
# ...
READY = 4
#: The node is ready for named deployment.
maasdb=# select id from maasserver_node where hostname = 'prometheus-3';
id
----
13
(1 row)
maasdb=# select * from maasserver_space;
id | created | updated | name | description
----+-------------------------------+-------------------------------+--------------------+-----------------------------------------------------
1 | 2019-07-05 19:02:02.797481+00 | 2019-07-05 19:02:02.797481+00 | ps45_routers | foundation-engine created space: ps45_routers
2 | 2019-07-05 19:02:03.603296+00 | 2019-07-05 19:02:03.603296+00 | internal-space | foundation-engine created space: internal-space
3 | 2019-07-05 19:02:05.074262+00 | 2019-07-05 19:02:05.074262+00 | oam-space | foundation-engine created space: oam-space
4 | 2019-07-05 19:02:06.108567+00 | 2019-07-05 19:02:06.108567+00 | external-space | foundation-engine created space: external-space
5 | 2019-07-05 19:02:06.922964+00 | 2019-07-05 19:02:06.922964+00 | ceph-replica-space | foundation-engine created space: ceph-replica-space
6 | 2019-07-05 19:02:07.80897+00 | 2019-07-05 19:02:07.80897+00 | ceph-access-space | foundation-engine created space: ceph-access-space
(6 rows)
maasdb=# select vlan_id from maasserver_interface where node_id = 13;
vlan_id
---------
5002
(1 row)
maasdb=# select id,space_id from maasserver_vlan where id = 5002;
id | space_id
------+----------
5002 | 3
(1 row)
maasdb=# select node_id,tag_id from maasserver_node inner join maasserver_node_tags on maasserver_node.id = maasserver_node_tags.node_id where maasserver_node.id = 13;
node_id | tag_id
---------+--------
13 | 2
13 | 1
13 | 9
(3 rows)
maasdb=# select * from maasserver_tag where name like '%prometheus%';
id | created | updated | name | definition | comment | kernel_opts
----+-------------------------------+-------------------------------+------------+------------+---------+-------------
9 | 2019-07-05 19:03:58.501961+00 | 2019-07-05 19:03:58.501961+00 | prometheus | | |
(1 row) |
|
| 2019-08-28 19:16:06 |
Dmitrii Shcherbakov |
description |
Re-targeted to Juju (2.6.5) based on https://bugs.launchpad.net/maas/+bug/1835823/comments/2
Old title: "maas reported 409 CONFLICT from "allocate" while the node matching constraints was available"
https://solutions.qa.canonical.com/#/qa/testRun/808239c1-7b06-4281-9267-0f09421604a1
https://oil-jenkins.canonical.com/artifacts/808239c1-7b06-4281-9267-0f09421604a1/index.html (deployment artifacts)
Juju tried 10 times to allocate a machine with tag "prometheus" but MAAS responded with 509s to all requests so Juju eventually marked the machine as failed. The machine is a pod VM (the host is in zone-3). Based on the DB dump it looks like it exists, is ready, has the right tag and interface with access to the right space.
8cf17ecd-809f-4c14-8220-09b6b66bbc74: machine-0 2019-07-05 19:55:24 WARNING juju.worker.provisioner provisioner_task.go:1157 failed to start machine 3 (failed to acquire node: No available machine matches constraints: [('agent_name', ['85f7eddb-f1bb-45fd-8eba-e9e50b532eda']), ('interfaces', ['internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3']), ('tags', ['prometheus']), ('zone', ['zone3'])] (resolved to "interfaces=internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3 tags=prometheus zone=zone3")), retrying in 10s (10 more attempts)
2019-07-05 19:55:23 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 200 OK (referrer: -; agent: Go-http-client/1.1)
2019-07-05 19:55:24 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 409 CONFLICT (referrer: -; agent: Go-http-client/1.1)
select x.hostname,x.status,x.zone_id,x.name from (select * from maasserver_node inner join maasserver_zone on maasserver_node.zone_id = maasserver_zone.id) as x where x.hostname LIKE '%prometheus%';
hostname | status | zone_id | name
--------------+--------+---------+-------
prometheus-3 | 4 | 3 | zone3
(1 row)
src/maasserver/enum.py
class NODE_STATUS:
# ...
READY = 4
#: The node is ready for named deployment.
maasdb=# select id from maasserver_node where hostname = 'prometheus-3';
id
----
13
(1 row)
maasdb=# select * from maasserver_space;
id | created | updated | name | description
----+-------------------------------+-------------------------------+--------------------+-----------------------------------------------------
1 | 2019-07-05 19:02:02.797481+00 | 2019-07-05 19:02:02.797481+00 | ps45_routers | foundation-engine created space: ps45_routers
2 | 2019-07-05 19:02:03.603296+00 | 2019-07-05 19:02:03.603296+00 | internal-space | foundation-engine created space: internal-space
3 | 2019-07-05 19:02:05.074262+00 | 2019-07-05 19:02:05.074262+00 | oam-space | foundation-engine created space: oam-space
4 | 2019-07-05 19:02:06.108567+00 | 2019-07-05 19:02:06.108567+00 | external-space | foundation-engine created space: external-space
5 | 2019-07-05 19:02:06.922964+00 | 2019-07-05 19:02:06.922964+00 | ceph-replica-space | foundation-engine created space: ceph-replica-space
6 | 2019-07-05 19:02:07.80897+00 | 2019-07-05 19:02:07.80897+00 | ceph-access-space | foundation-engine created space: ceph-access-space
(6 rows)
maasdb=# select vlan_id from maasserver_interface where node_id = 13;
vlan_id
---------
5002
(1 row)
maasdb=# select id,space_id from maasserver_vlan where id = 5002;
id | space_id
------+----------
5002 | 3
(1 row)
maasdb=# select node_id,tag_id from maasserver_node inner join maasserver_node_tags on maasserver_node.id = maasserver_node_tags.node_id where maasserver_node.id = 13;
node_id | tag_id
---------+--------
13 | 2
13 | 1
13 | 9
(3 rows)
maasdb=# select * from maasserver_tag where name like '%prometheus%';
id | created | updated | name | definition | comment | kernel_opts
----+-------------------------------+-------------------------------+------------+------------+---------+-------------
9 | 2019-07-05 19:03:58.501961+00 | 2019-07-05 19:03:58.501961+00 | prometheus | | |
(1 row) |
Re-targeted to Juju (2.6.5) based on https://bugs.launchpad.net/maas/+bug/1835823/comments/2 (see this comment for details)
Old title: "maas reported 409 CONFLICT from "allocate" while the node matching constraints was available"
https://solutions.qa.canonical.com/#/qa/testRun/808239c1-7b06-4281-9267-0f09421604a1
https://oil-jenkins.canonical.com/artifacts/808239c1-7b06-4281-9267-0f09421604a1/index.html (deployment artifacts)
Juju tried 10 times to allocate a machine with tag "prometheus" but MAAS responded with 509s to all requests so Juju eventually marked the machine as failed. The machine is a pod VM (the host is in zone-3). Based on the DB dump it looks like it exists, is ready, has the right tag and interface with access to the right space.
8cf17ecd-809f-4c14-8220-09b6b66bbc74: machine-0 2019-07-05 19:55:24 WARNING juju.worker.provisioner provisioner_task.go:1157 failed to start machine 3 (failed to acquire node: No available machine matches constraints: [('agent_name', ['85f7eddb-f1bb-45fd-8eba-e9e50b532eda']), ('interfaces', ['internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3']), ('tags', ['prometheus']), ('zone', ['zone3'])] (resolved to "interfaces=internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3 tags=prometheus zone=zone3")), retrying in 10s (10 more attempts)
2019-07-05 19:55:23 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 200 OK (referrer: -; agent: Go-http-client/1.1)
2019-07-05 19:55:24 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 409 CONFLICT (referrer: -; agent: Go-http-client/1.1)
select x.hostname,x.status,x.zone_id,x.name from (select * from maasserver_node inner join maasserver_zone on maasserver_node.zone_id = maasserver_zone.id) as x where x.hostname LIKE '%prometheus%';
hostname | status | zone_id | name
--------------+--------+---------+-------
prometheus-3 | 4 | 3 | zone3
(1 row)
src/maasserver/enum.py
class NODE_STATUS:
# ...
READY = 4
#: The node is ready for named deployment.
maasdb=# select id from maasserver_node where hostname = 'prometheus-3';
id
----
13
(1 row)
maasdb=# select * from maasserver_space;
id | created | updated | name | description
----+-------------------------------+-------------------------------+--------------------+-----------------------------------------------------
1 | 2019-07-05 19:02:02.797481+00 | 2019-07-05 19:02:02.797481+00 | ps45_routers | foundation-engine created space: ps45_routers
2 | 2019-07-05 19:02:03.603296+00 | 2019-07-05 19:02:03.603296+00 | internal-space | foundation-engine created space: internal-space
3 | 2019-07-05 19:02:05.074262+00 | 2019-07-05 19:02:05.074262+00 | oam-space | foundation-engine created space: oam-space
4 | 2019-07-05 19:02:06.108567+00 | 2019-07-05 19:02:06.108567+00 | external-space | foundation-engine created space: external-space
5 | 2019-07-05 19:02:06.922964+00 | 2019-07-05 19:02:06.922964+00 | ceph-replica-space | foundation-engine created space: ceph-replica-space
6 | 2019-07-05 19:02:07.80897+00 | 2019-07-05 19:02:07.80897+00 | ceph-access-space | foundation-engine created space: ceph-access-space
(6 rows)
maasdb=# select vlan_id from maasserver_interface where node_id = 13;
vlan_id
---------
5002
(1 row)
maasdb=# select id,space_id from maasserver_vlan where id = 5002;
id | space_id
------+----------
5002 | 3
(1 row)
maasdb=# select node_id,tag_id from maasserver_node inner join maasserver_node_tags on maasserver_node.id = maasserver_node_tags.node_id where maasserver_node.id = 13;
node_id | tag_id
---------+--------
13 | 2
13 | 1
13 | 9
(3 rows)
maasdb=# select * from maasserver_tag where name like '%prometheus%';
id | created | updated | name | definition | comment | kernel_opts
----+-------------------------------+-------------------------------+------------+------------+---------+-------------
9 | 2019-07-05 19:03:58.501961+00 | 2019-07-05 19:03:58.501961+00 | prometheus | | |
(1 row) |
|
| 2019-08-28 20:36:45 |
Newell Jensen |
maas: status |
Incomplete |
Invalid |
|
| 2019-10-01 09:25:39 |
Alberto Donato |
maas: milestone |
2.7.0alpha1 |
|
|
| 2019-10-14 02:14:01 |
Tim Penhey |
juju: status |
New |
Incomplete |
|
| 2019-10-14 05:55:00 |
Dmitrii Shcherbakov |
juju: status |
Incomplete |
New |
|
| 2020-02-03 18:53:54 |
Richard Harding |
juju: status |
New |
Triaged |
|
| 2020-02-03 18:53:56 |
Richard Harding |
juju: importance |
Undecided |
Medium |
|
| 2022-11-03 15:45:59 |
Canonical Juju QA Bot |
juju: importance |
Medium |
Low |
|
| 2022-11-03 15:46:01 |
Canonical Juju QA Bot |
tags |
cdo-qa |
cdo-qa expirebugs-bot |
|
