[2.2] Deleting machines from Nodes listing page tries decomposing multiple times.

Bug #1667754 reported by Newell Jensen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Newell Jensen
MAAS RSD
Fix Released
Critical
Newell Jensen

Bug Description

Deleting machines from an RSD Pod in the UI Nodes listing page causes errors. From what I could gather this was caused because MAAS is trying to delete the machines twice which causes the decompose signal to try and decompose the machine twice. Since the machine has already been decomposed we are getting a 404 error back from the RSD Redfish API. With some print statements we can see in the logs below that the first node to try and be deleted twice is node_id 302:

==> /var/log/maas/maas.log <==
Feb 23 22:11:04 maas-dev maas.node: [info] driven-toad: Deleting node
Feb 23 22:11:04 maas-dev maas.node: [info] fancy-cub: Deleting node
Feb 23 22:11:04 maas-dev maas.node: [info] deep-piglet: Deleting node
Feb 23 22:11:04 maas-dev maas.node: [info] driven-marten: Deleting node

==> /var/log/maas/regiond.log <==
2017-02-23 22:11:04 stdout: [info] instance.creation_typ: 2
2017-02-23 22:11:04 stdout: [info] instance.power_parameters.node_id: 304

==> /var/log/maas/rackd.log <==
2017-02-23 22:11:04 stdout: [info] decompose node_id: b'304'

==> /var/log/maas/regiond.log <==
2017-02-23 22:11:04 stdout: [info] instance.creation_typ: 2
2017-02-23 22:11:04 stdout: [info] instance.power_parameters.node_id: 302
2017-02-23 22:11:04 stdout: [info] instance.creation_typ: 2
2017-02-23 22:11:04 stdout: [info] instance.power_parameters.node_id: 303

==> /var/log/maas/rackd.log <==
2017-02-23 22:11:04 stdout: [info] decompose node_id: b'302'
2017-02-23 22:11:04 stdout: [info] decompose node_id: b'303'

==> /var/log/maas/maas.log <==
Feb 23 22:11:04 maas-dev maas.node: [info] fluent-molly: Deleting node
Feb 23 22:11:04 maas-dev maas.node: [info] full-kiwi: Deleting node
Feb 23 22:11:04 maas-dev maas.node: [info] many-dory: Deleting node

==> /var/log/maas/regiond.log <==
2017-02-23 22:11:05 stdout: [info] instance.creation_typ: 2
2017-02-23 22:11:05 stdout: [info] instance.power_parameters.node_id: 301

==> /var/log/maas/rackd.log <==
2017-02-23 22:11:05 stdout: [info] decompose node_id: b'301'

==> /var/log/maas/maas.log <==
Feb 23 22:11:13 maas-dev maas.node: [info] driven-marten: Deleting node

==> /var/log/maas/regiond.log <==
2017-02-23 22:11:13 stdout: [info] instance.creation_typ: 2
2017-02-23 22:11:13 stdout: [info] instance.power_parameters.node_id: 302

==> /var/log/maas/rackd.log <==
2017-02-23 22:11:13 stdout: [info] decompose node_id: b'302'

==> /var/log/maas/maas.log <==
Feb 23 22:11:13 maas-dev maas.node: [info] deep-piglet: Deleting node

==> /var/log/maas/rackd.log <==
2017-02-23 22:11:13 provisioningserver.rpc.pods: [critical] Failed to decompose machine.

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 434, in errback
    self._startRunCallbacks(fail)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python3/dist-packages/provisioningserver/drivers/pod/rsd.py", line 686, in decompose

Related branches

Changed in maas:
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
Newell Jensen (newell-jensen) wrote :

Additionally, this seems to only happen if we attempt to delete more than one machine at the same time. Deleting one machine at a time does not cause this.

Revision history for this message
Brendan Donegan (brendan-donegan) wrote :

Note that the RSD test suite deletes multiple (2) nodes separately in quick succession using the CLI. Also I have seen this when attempting to delete as few as two nodes from the UI simultaneously.

Changed in maas:
milestone: none → 2.2.0
Changed in maasrsd:
milestone: none → alpha
milestone: alpha → beta
summary: - Deleting machines from Nodes listing page tries decomposing multiple
- times.
+ [2.2] Deleting machines from Nodes listing page tries decomposing
+ multiple times.
Changed in maas:
assignee: nobody → Newell Jensen (newell-jensen)
Changed in maasrsd:
assignee: nobody → Newell Jensen (newell-jensen)
Changed in maas:
status: Triaged → In Progress
Changed in maasrsd:
status: Triaged → In Progress
Changed in maas:
status: In Progress → Fix Committed
Changed in maasrsd:
status: In Progress → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
Changed in maasrsd:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.