Updating pod fails if a VM interface has no MAC

[Huw Wilkins]

If you try and update a pod you get an error about the mac_address field being blank, but as far as I can see there is no mac_address associated with a pod.

You can see this by updating a pod on bolla (change any field and save the form):

http://bolla.internal:5240/MAAS/r/kvm/336/edit

You can also reproduce this via the API:

maas $PROFILE vmhost update $VM_HOST_ID tags=abc

Responds with:

{"mac_address": ["This field cannot be blank."]}

[Alberto Donato (ack)]

When trying this on bolla I get this traceback in logs:

2021-10-19 07:14:04 maasserver: [error] ################################ Exception: {'mac_address': ['This field cannot be blank.']} ################################
2021-10-19 07:14:04 maasserver: [error] Traceback (most recent call last):
  File "/snap/maas/16969/usr/lib/python3/dist-packages/django/db/models/query.py", line 538, in get_or_create
    return self.get(**kwargs), False
  File "/snap/maas/16969/usr/lib/python3/dist-packages/django/db/models/query.py", line 406, in get
    raise self.model.DoesNotExist(
maasserver.models.interface.PhysicalInterface.DoesNotExist: PhysicalInterface matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/maas/16969/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/utils/views.py", line 284, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/api/support.py", line 56, in __call__
    response = super().__call__(request, *args, **kwargs)
  File "/snap/maas/16969/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 20, in inner_func
    response = func(*args, **kwargs)
  File "/snap/maas/16969/usr/lib/python3.8/dist-packages/piston3/resource.py", line 197, in __call__
    result = self.error_handler(e, request, meth, em_format)
  File "/snap/maas/16969/usr/lib/python3.8/dist-packages/piston3/resource.py", line 195, in __call__
    result = meth(request, *args, **kwargs)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/api/support.py", line 308, in dispatch
    return function(self, request, *args, **kwargs)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/api/support.py", line 158, in w...

[Alberto Donato (ack)]

This is caused by one of the VMs in the LXD host missing the MAC address for one interface (no volatile.eth0.hwaddr key). Not sure how it got there.

Setting a random mac on the machine made it go further, but it now fails with this:

2021-10-19 07:23:08 maasserver: [error] ################################ Exception: ("Cannot delete some instances of model 'VLAN' because they are referenced through a protected foreign key: 'Subnet.vlan'", <QuerySet [<Subnet: test:10.0.6.0/24(vid=0)>, <Subnet: 123.124.1.3/24:123.124.1.0/24(vid=0)>, <Subnet: fd42:2c39:4a4e:99a6::/64:fd42:2c39:4a4e:99a6::/64(vid=0)>, <Subnet: zzzzz:192.168.2.3/32(vid=0)>, <Subnet: 192.168.123.0/24:192.168.123.0/24(vid=0)>, <Subnet: 10.249.0.0/21:10.249.0.0/21(vid=0)>, <Subnet: 172.16.99.0/24:172.16.99.0/24(vid=0)>, <Subnet: tenbus:10.1.238.0/24(vid=0)>]>) ################################
2021-10-19 07:23:08 maasserver: [error] Traceback (most recent call last):
  File "/snap/maas/16969/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/utils/views.py", line 284, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/api/support.py", line 56, in __call__
    response = super().__call__(request, *args, **kwargs)
  File "/snap/maas/16969/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 20, in inner_func
    response = func(*args, **kwargs)
  File "/snap/maas/16969/usr/lib/python3.8/dist-packages/piston3/resource.py", line 197, in __call__
    result = self.error_handler(e, request, meth, em_format)
  File "/snap/maas/16969/usr/lib/python3.8/dist-packages/piston3/resource.py", line 195, in __call__
    result = meth(request, *args, **kwargs)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/api/support.py", line 308, in dispatch
    return function(self, request, *args, **kwargs)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/api/support.py", line 158, in wrapper
    return func(self, request, *args, **kwargs)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/api/pods.py", line 162, in update
    _try_sync_and_save(pod, request.user)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/api/pods.py", line 552, in _try_sync_and_save
    discover_and_sync_vmhost(pod, user)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/vmhost.py", line 86, in discover_and_sync_vmhost
    vmhost = _update_db(discovered_pod, discovered, vmhost, user)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/vmhost.py", line 255, in _update_db
    vmhost.sync(discovered_pod, user, cluster=cluster)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/models/bmc.py", line 1535, in sync
    self.sync_machines(discovered_pod.machines, commissioning_user)
  File "/snap/maas/16969/lib/python3.8/site-packages/maasserver/models/bmc.py", line 1442, in s...

[Christian Grabowski (cgrabowski)]

Interestingly, VMs without mac addresses only seem to be created on Polong, Karura works fine. It appears MAAS composes the VM, but the returned value does not contain a MAC address. This then also becomes an issue on refresh or any update as MAAS tries to create the machine again, and LXD still returns it without a MAC address.

[Christian Grabowski (cgrabowski)]

Found the issue, being a cluster, LXD shares a profile across machines, the given profile uses br1, which exists on karura, but not polong, causing LXD to be able to create the machine, but not start, nor report a MAC address for the machine.

[Christian Grabowski (cgrabowski)]

Marking as invalid as this was an issue with the LXD host's network configuration.

[Huw Wilkins (huwshimi)]

This still appears to be an issue (on Bolla). If it's related to the network setup maybe the API should provide a more helpful error message.

[Huw Wilkins (huwshimi)]

On second thoughts, I don't think this should be marked as invalid as I don't think the pod.update call should return an error in this case. The error appears to be unrelated and the update actually succeeds.

[Christian Grabowski (cgrabowski)]

Actually, this was being discussed today, and now there's an open MP regarding this, will link it.