Machine details stuck at "Loading" for machines with no disks

[Björn Tillenius]

I removed all disks from a machine, and when I go to the machine details page (e.g. /MAAS/r/machine/74ewg4/summary), all I see is the "Loading" spinner.

[Björn Tillenius (bjornt)]

Seems that it's not enough to only remove the disks from the machine. You also have to deploy the machine. If you do that from the machine details page, the machine status stays "Ready" on that page. The deploy is happening, though, and if you reload the page, it gets stuck in "Loading".

The machine listing still works.

[Björn Tillenius (bjornt)]

Looks to be a backend bug. I see this in the logs:

2022-11-09 15:32:52 maasserver.websockets.protocol: [critical] Error on request (4) machine.set_active: ScriptSet matching query does not exist.
 Traceback (most recent call last):
   File "/usr/lib/python3.10/threading.py", line 946, in run
     self._target(*self._args, **self._kwargs)
   File "/snap/maas/24793/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 822, in worker
     return target()
   File "/snap/maas/24793/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 47, in work
     task()
   File "/snap/maas/24793/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 182, in doWork
     task()
 --- <exception caught here> ---
   File "/snap/maas/24793/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
     result = inContext.theWork() # type: ignore[attr-defined]
   File "/snap/maas/24793/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
     inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
   File "/snap/maas/24793/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/snap/maas/24793/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
     return func(*args, **kw)
   File "/snap/maas/24793/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 857, in callInContext
     return func(*args, **kwargs)
   File "/snap/maas/24793/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
     result = func(*args, **kwargs)
   File "/snap/maas/24793/lib/python3.10/site-packages/maasserver/utils/orm.py", line 771, in call_within_transaction
     return func_outside_txn(*args, **kwargs)
   File "/snap/maas/24793/lib/python3.10/site-packages/maasserver/utils/orm.py", line 574, in retrier
     return func(*args, **kwargs)
   File "/usr/lib/python3.10/contextlib.py", line 79, in inner
     return func(*args, **kwds)
   File "/snap/maas/24793/lib/python3.10/site-packages/maasserver/websockets/base.py", line 437, in prep_user_execute
     return self._call_method_track_queries(
   File "/snap/maas/24793/lib/python3.10/site-packages/maasserver/websockets/base.py", line 461, in _call_method_track_queries
     result = method(params)
   File "/snap/maas/24793/lib/python3.10/site-packages/maasserver/websockets/base.py", line 809, in set_active
     obj_data = self.get(params)
   File "/snap/maas/24793/lib/python3.10/site-packages/maasserver/websockets/base.py", line 700, in get
     return self.full_dehydrate(obj)
   File "/snap/maas/24793/lib/python3.10/site-packages/maasserver/websockets/base.py", line 219, in full_dehydrate
     field_obj = getattr(obj, field_name)
   File "/snap/maas/24793/usr/lib/python3/dist-packages/django/db/models/fields/related_descriptors.py", line 187, in __get__
     rel_obj = self.get_object(instance)
   File "/snap/maas/24793/usr/lib/python3/dist-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object
     retur...

[Björn Tillenius (bjornt)]

The 'current_commissioning_script_set', 'current_testing_script_set' and 'current_commissioning_script_set' are declared like this in Node:

    current_testing_script_set = ForeignKey(
        "metadataserver.ScriptSet",
        blank=True,
        null=True,
        on_delete=SET_NULL,
        related_name="+",
    )

But if you look in the database, there's no FOREIGN KEY constraint for that column.

Further more, it seems like the 'on_delete=SET_NULL' also doesn't work properly. Node.start() eventually triggers ScriptResult.create_installation_script_set(), which triggers ScriptResult._clean_old(), which deletes all empty script sets. If one of those are the 'current' scriptset for a node, the Node attribute doesn't get set to None. Instead it leaves a dangling reference to a non-existing script set.

I believe the missing foreign key constraint is due to an old bug in Django: https://github.com/django/django/commit/50bf567675

They fixed it only for new table, though, and existing tables stayed broken. Not sure how that bug affects on_delete=SET_NULL, though, since that's done all in Python.