[2.2] Deleting machines from Nodes listing page tries decomposing multiple times.

[Newell Jensen]

Deleting machines from an RSD Pod in the UI Nodes listing page causes errors. From what I could gather this was caused because MAAS is trying to delete the machines twice which causes the decompose signal to try and decompose the machine twice. Since the machine has already been decomposed we are getting a 404 error back from the RSD Redfish API. With some print statements we can see in the logs below that the first node to try and be deleted twice is node_id 302:

==> /var/log/maas/maas.log <==
Feb 23 22:11:04 maas-dev maas.node: [info] driven-toad: Deleting node
Feb 23 22:11:04 maas-dev maas.node: [info] fancy-cub: Deleting node
Feb 23 22:11:04 maas-dev maas.node: [info] deep-piglet: Deleting node
Feb 23 22:11:04 maas-dev maas.node: [info] driven-marten: Deleting node

==> /var/log/maas/regiond.log <==
2017-02-23 22:11:04 stdout: [info] instance.creation_typ: 2
2017-02-23 22:11:04 stdout: [info] instance.power_parameters.node_id: 304

==> /var/log/maas/rackd.log <==
2017-02-23 22:11:04 stdout: [info] decompose node_id: b'304'

==> /var/log/maas/regiond.log <==
2017-02-23 22:11:04 stdout: [info] instance.creation_typ: 2
2017-02-23 22:11:04 stdout: [info] instance.power_parameters.node_id: 302
2017-02-23 22:11:04 stdout: [info] instance.creation_typ: 2
2017-02-23 22:11:04 stdout: [info] instance.power_parameters.node_id: 303

==> /var/log/maas/rackd.log <==
2017-02-23 22:11:04 stdout: [info] decompose node_id: b'302'
2017-02-23 22:11:04 stdout: [info] decompose node_id: b'303'

==> /var/log/maas/maas.log <==
Feb 23 22:11:04 maas-dev maas.node: [info] fluent-molly: Deleting node
Feb 23 22:11:04 maas-dev maas.node: [info] full-kiwi: Deleting node
Feb 23 22:11:04 maas-dev maas.node: [info] many-dory: Deleting node

==> /var/log/maas/regiond.log <==
2017-02-23 22:11:05 stdout: [info] instance.creation_typ: 2
2017-02-23 22:11:05 stdout: [info] instance.power_parameters.node_id: 301

==> /var/log/maas/rackd.log <==
2017-02-23 22:11:05 stdout: [info] decompose node_id: b'301'

==> /var/log/maas/maas.log <==
Feb 23 22:11:13 maas-dev maas.node: [info] driven-marten: Deleting node

==> /var/log/maas/regiond.log <==
2017-02-23 22:11:13 stdout: [info] instance.creation_typ: 2
2017-02-23 22:11:13 stdout: [info] instance.power_parameters.node_id: 302

==> /var/log/maas/rackd.log <==
2017-02-23 22:11:13 stdout: [info] decompose node_id: b'302'

==> /var/log/maas/maas.log <==
Feb 23 22:11:13 maas-dev maas.node: [info] deep-piglet: Deleting node

==> /var/log/maas/rackd.log <==
2017-02-23 22:11:13 provisioningserver.rpc.pods: [critical] Failed to decompose machine.

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 434, in errback
    self._startRunCallbacks(fail)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python3/dist-packages/provisioningserver/drivers/pod/rsd.py", line 686, in decompose

[Newell Jensen (newell-jensen)]

Additionally, this seems to only happen if we attempt to delete more than one machine at the same time. Deleting one machine at a time does not cause this.

[Brendan Donegan (brendan-donegan)]

Note that the RSD test suite deletes multiple (2) nodes separately in quick succession using the CLI. Also I have seen this when attempting to delete as few as two nodes from the UI simultaneously.