StaticIPAddress matching query does not exist.

[Chris Johnston]

I am deploying MAAS using FCE. During one of the FCE steps it is configuring the nodes. The first time that I configure the nodes things work correctly. If I realize that I need to change something and then reconfigure the nodes I run into this error when it runs the command `maas <user> machine restore-networking-configuration <system_id>`. I am seeing this with 3.3/edge, but if I deploy with 3.2 I don't see the same error.

2023-04-05 20:15:21 maasserver: [error] ################################ Exception: StaticIPAddress matching query does not exist. ################################
2023-04-05 20:15:21 maasserver: [error] Traceback (most recent call last):
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/db/models/fields/related_descriptors.py", line 173, in __get__
    rel_obj = self.field.get_cached_value(instance)
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/db/models/fields/mixins.py", line 15, in get_cached_value
    return instance._state.fields_cache[cache_name]
KeyError: 'gateway_link_ipv4'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/snap/maas/27109/lib/python3.10/site-packages/maasserver/utils/views.py", line 293, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/snap/maas/27109/lib/python3.10/site-packages/maasserver/api/support.py", line 62, in __call__
    response = super().__call__(request, *args, **kwargs)
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 20, in inner_func
    response = func(*args, **kwargs)
  File "/snap/maas/27109/usr/lib/python3.10/dist-packages/piston3/resource.py", line 238, in __call__
    stream = srl.render(request)
  File "/snap/maas/27109/usr/lib/python3.10/dist-packages/piston3/emitters.py", line 448, in render
    self.construct(), cls=DjangoJSONEncoder, ensure_ascii=False, indent=4
  File "/snap/maas/27109/usr/lib/python3.10/dist-packages/piston3/emitters.py", line 352, in construct
    return _any(self.data, self.fields)
  File "/snap/maas/27109/usr/lib/python3.10/dist-packages/piston3/emitters.py", line 122, in _any
    ret = _model(thing, fields)
  File "/snap/maas/27109/usr/lib/python3.10/dist-packages/piston3/emitters.py", line 269, in _model
    ret[maybe_field] = _any(met_fields[maybe_field](data))
  File "/snap/maas/27109/lib/python3.10/site-packages/maasserver/api/machines.py", line 564, in default_gateways
    gateways = machine.get_default_gateways()
  File "/snap/maas/27109/lib/python3.10/site-packages/maasserver/models/node.py", line 5133, in get_default_gateways
    if self.gateway_link_ipv4 is not None:
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/db/models/fields/related_descriptors.py", line 187, in __get__
    rel_obj = self.get_object(instance)
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object
    return qs.get(self.field.get_reverse_related_filter(instance))
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
maasserver.models.staticipaddress.StaticIPAddress.DoesNotExist: StaticIPAddress matching query does not exist.

I found this commit which looks like it could possibly be related:

https://git.launchpad.net/maas/commit/?id=3ac7d21924bb25c0181320c95bc52f787920fa5b

[Andre Ruiz (andre-ruiz)]

This is happening on 3.3.2 during an Openstack deployment.

I reduced my entire bucektsconfig to 1 line, and I still get this error, even when cleaning all machines and starting from scratch without any prior config:

disks: []
nics:
  - { id: ens1f0, name: ens1f0, nic: 4, subnets: [], type: physical, fabric: openstack, vlan_id: 0, mtu: 9000 }

This is enough to give me:

ubuntu@brtlvmrs0763co:~/2023-01-30-Telefonica-OCS5-PedroGusso-OP-237579$ fce --debug build --layer maas --steps maas:configure_nodes
2023-04-12-15:16:27 root DEBUG fce --debug build --layer maas --steps maas:configure_nodes
2023-04-12-15:16:27 root DEBUG FCE version: 2.18.2
2023-04-12-15:16:27 root DEBUG [localhost]: juju export-bundle -m foundations-maas:lma-maas
2023-04-12-15:16:27 root ERROR [localhost] Command failed: juju export-bundle -m foundations-maas:lma-maas
2023-04-12-15:16:27 root ERROR [localhost] STDOUT follows:
b''
2023-04-12-15:16:27 root ERROR [localhost] STDERR follows:
ERROR controller foundations-maas not found

2023-04-12-15:16:27 root DEBUG [localhost]: juju export-bundle -m foundations-maas:openstack
2023-04-12-15:16:27 root ERROR [localhost] Command failed: juju export-bundle -m foundations-maas:openstack
2023-04-12-15:16:27 root ERROR [localhost] STDOUT follows:
b''
2023-04-12-15:16:27 root ERROR [localhost] STDERR follows:
ERROR controller foundations-maas not found

2023-04-12-15:16:27 root DEBUG Running 'zone' project check
2023-04-12-15:16:27 fce.build INFO Started building layer: maas
2023-04-12-15:16:30 fce.maas INFO Starting step: maas:configure_nodes
2023-04-12-15:16:30 root DEBUG [localhost]: maas root machines read hostname=brtlvmrs0754co domain=maas
2023-04-12-15:16:33 root DEBUG [localhost]: maas root tags create name=control-nodes
2023-04-12-15:16:35 root ERROR [localhost] Command failed: maas root tags create name=control-nodes
2023-04-12-15:16:35 root ERROR [localhost] STDOUT follows:
{"name": ["Tag with this Name already exists."]}
2023-04-12-15:16:35 root ERROR [localhost] STDERR follows:
b''
2023-04-12-15:16:35 root DEBUG [localhost]: maas root tag update-nodes control-nodes add=aws7he
2023-04-12-15:16:36 root INFO Applying config 'control-nodes' to 'brtlvmrs0754co.maas'
2023-04-12-15:16:36 root DEBUG [localhost]: maas root machines read hostname=brtlvmrs0754co domain=maas
2023-04-12-15:16:39 root INFO Skipping storage configuration.
2023-04-12-15:16:39 root INFO Setting up networking on 'aws7he'
2023-04-12-15:16:39 root INFO Clearing previous network configuration
2023-04-12-15:16:39 root DEBUG [localhost]: maas root machine restore-networking-configuration aws7he
2023-04-12-15:16:43 root ERROR [localhost] Command failed: maas root machine restore-networking-configuration aws7he
2023-04-12-15:16:43 root ERROR [localhost] STDOUT follows:
StaticIPAddress matching query does not exist.
2023-04-12-15:16:43 root ERROR [localhost] STDERR follows:
b''
2023-04-12-15:16:43 root ERROR ('brtlvmrs0754co.maas', 'control-nodes'):Command '['maas', 'root', 'machine', 'restore-networking-configuration', 'aws7he']' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/usr/local/bin/fce", line 33, in <module>
...

[Andre Ruiz (andre-ruiz)]

Subscribed field-critical on this one since deployment just got blocked.

[Igor Brovtsin (igor-brovtsin)]

For some reason, Django attempts to fetch the record by the foreign key, even though it is nullable. Probably because there *was* a value in the first place, although I cannot seem to reproduce that exact failure locally. A hotfix would be to handle the error in `default_gateways`.

[Andre Ruiz (andre-ruiz)]

Removing field-critical since I have a workaround (re-install with maas 3.2 for now, upgrade again later).

[Navdeep (navdeep-bjn)]

I just moved to MaaS 3.3 on Support's request and now facing the same issue. The issue for me happens when I have static IP's assigned to the node and run maas:configure_nodes steps. The command it fails on maas', 'admin', 'machine', 'restore-networking-configuration'<node-id>. If I go into the maas UI and delete the statically assigned IP then run the step again it works..

[Vern Hart (vern)]

I've encountered this on MAAS 3.3.4.

When I configure nodes (with fce) and it executes the restore-networking-configuration sub-command, I get the StaticIPAddress error. I can recreate it at the command-line:

  $ maas root machine restore-networking-configuration fk7ncm
  StaticIPAddress matching query does not exist.

If I recommission that machine and re-run the restore-networking-configuration sub-command, the error goes away but seems to pop up on a different node. I've gone through a few cycles of this whack-a-mole.

I'm not using any static IP addresses.

I have 42 machines. I just recommissioned them all at once and, when done ran the following and had positive results for every machine:

  $ for id in $(maas root machines read | jq -r '.[]|.system_id'); do
      maas root machine restore-networking-configuration $id | grep fqdn;
    done

Then, after running `fce build --layer maas --steps generate_buckets,configure_nodes` I saw no errors.

Why would this give different results than recommissioning individual nodes at a time? Maybe I got lucky?

[Vern Hart (vern)]

A less drastic work-around than recommissioning all nodes (or rolling back to 3.2) is the following db update:

  sudo su - maas -c "psql maasdb -c 'update maasserver_node set gateway_link_ipv4_id=null where system_id='\''cpg88q'\'';' "

Where cpg88q is the system id of the node where the restore-networking-configuration command fails.

[Nobuto Murata (nobuto)]

Subscribing ~field-high.

Basically `maas <PROFILE> machine restore-networking-configuration` is not working. And there is no known workaround unless doing a database surgery or completely removing and re-adding a machine.

[Chris Johnston (cjohnston)]

This should be available in 3.3/edge: 3.3.5-13208-g.beee034cf for anyone who needs it.

[Bartosz Woronicz (mastier1)]

Thanks Chris.

So far I checked also re-commisioning the nodes also help, so that at least cuts the time on enlisting nodes. Looking forward to that change landing in stable.

[DUFOUR Olivier (odufourc)]

It happened to me today as well on a customer's deployment.
I can confirm that using MAAS 3.3.5 from 3.3/edge fixes the issue.