MAAS fails on determining default route

[Rostyslav-fridman]

MAAS version: 2.9.2. Installed via snap.

MAAS fails with the following stack trace when trying to determine the default route.
/var/snap/maas/common/log/regiond.log:
```
2021-05-04 13:29:58 twisted.internet.defer: [critical] Unhandled error in Deferred:
2021-05-04 13:29:58 twisted.internet.defer: [critical]
 Traceback (most recent call last):
   File "/usr/lib/python3.8/threading.py", line 870, in run
     self._target(*self._args, **self._kwargs)
   File "/snap/maas/12555/lib/python3.8/site-packages/provisioningserver/utils/twisted.py", line 825, in worker
     return target()
   File "/snap/maas/12555/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
     task()
   File "/snap/maas/12555/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
     task()
 --- <exception caught here> ---
   File "/snap/maas/12555/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
     result = inContext.theWork()
   File "/snap/maas/12555/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
   File "/snap/maas/12555/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/snap/maas/12555/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
     return func(*args,**kw)
   File "/snap/maas/12555/lib/python3.8/site-packages/provisioningserver/utils/twisted.py", line 860, in callInContext
     return func(*args, **kwargs)
   File "/snap/maas/12555/lib/python3.8/site-packages/provisioningserver/utils/twisted.py", line 192, in wrapper
     result = func(*args, **kwargs)
   File "/snap/maas/12555/lib/python3.8/site-packages/maasserver/utils/orm.py", line 737, in call_within_transaction
     return func_outside_txn(*args, **kwargs)
   File "/snap/maas/12555/lib/python3.8/site-packages/maasserver/utils/orm.py", line 540, in retrier
     return func(*args, **kwargs)
   File "/usr/lib/python3.8/contextlib.py", line 75, in inner
     return func(*args, **kwds)
   File "/snap/maas/12555/lib/python3.8/site-packages/maasserver/ipc.py", line 361, in create_endpoints
     self._updateEndpoints(process, self._getListenAddresses(port))
   File "/snap/maas/12555/lib/python3.8/site-packages/maasserver/ipc.py", line 303, in _getListenAddresses
     addresses = get_all_interface_source_addresses()
   File "/snap/maas/12555/lib/python3.8/site-packages/provisioningserver/utils/network.py", line 1223, in get_all_interface_source_addresses
     for network in get_all_interface_subnets():
   File "/snap/maas/12555/lib/python3.8/site-packages/provisioningserver/utils/network.py", line 1208, in get_all_interface_subnets
     for interface in get_all_interfaces_definition().values()
   File "/snap/maas/12555/lib/python3.8/site-packages/provisioningserver/utils/network.py", line 1188, in get_all_interfaces_definition
     fix_link_gateways(interface["links"], iproute_info)
   File "/snap/maas/12555/lib/python3.8/site-packages/provisioningserver/utils/network.py", line 1001, in fix_link_gateways
     and IPAddress(iproute_info["default"]["via"]) in ip_addr
 builtins.KeyError: 'via'
```

This happens due to the fact that default route is being handled by the user-space BGP routing daemon and the default route is actually multi-path.
```
default proto bgp metric 20
        nexthop via 169.254.0.1 dev external_1 weight 1 onlink
        nexthop via 169.254.0.1 dev external_2 weight 1 onlink
```

[Björn Tillenius (bjornt)]

Thanks for your bug report! Did you mean to propose a fix for it? I see that you have an MP, but the diff is empty.

I also wonder if this affects master/3.0 as well? The code path there has changed slightly.

[Rostyslav-fridman (rostyslav-fridman)]

Resubmitted: https://code.launchpad.net/~rostyslav-fridman/maas/+git/maas/+merge/405302

[Björn Tillenius (bjornt)]

Thanks for submitting the MP. But, before it can be fixed in 2.9, it has to be fixed in master. After that, the fix can be backported to 3.0 and 2.9.

[Alberto Donato (ack)]

The reason I mentioned target to 2.9 is that that code is gone entirely in 3.0 since we don't parse iproute directly anymore, we use the json output.

I'm not sure if the issue happens in 3.0 too, we need the output of `ip -json route list scope global` to confirm.

[Jerzy Husakowski (jhusakowski)]

Is this still an issue in MAAS 3.1 or later? BGP handling has changed since MAAS 2.9.