additional gateways use wrong source value

[Kevin Reeuwijk]

When deploying a node with multiple NICs on different subnets that each have their own default gateway set, MaaS incorrectly sets the "from:" parameter on the additional routes. This results in broken network connectivity for those routes. If the user manually correct the "from:" parameter and restarts systemd-networkd, the problem resolves.

Example:

Server with 2 NICs:
- NIC 1 on Subnet 1 with 33.44.55.64/29, default gateway is 33.44.55.65 and IPs are auto-assigned
- NIC 2 on Subnet 2 with 192.168.0.0/24, default gateway is 192.168.0.1 and IPs are auto-assigned

Resulting netplan config:
* NIC1 has IP 33.44.55.66 and "gateway4: 33.44.55.65" parameter set
* NIC2 has IP 192.168.0.2 and no gateway4 parameter set, but has a routing-policy:
    routes:
    - table: 1
        to: 0.0.0.0/0
        via: 192.168.0.1
    routing-policy:
    - from: 192.168.0.0/24
        priority: 100
        table: 1
    - from: 192.168.0.0/24
        table: 254
        to: 192.168.0.0/24

This results in not being able to ping any other address on the 192.168.0.0/24 network.

However the official documentation (https://netplan.io/reference#routing) notes that the "from:" parameter should be an IP address, not a CIDR. If I change the "from:" parameter in both entries of the routing-policy to "192.168.0.2", apply the netplan and restart systemd-networkd, everything works correctly.

I currently have to resort to the following cloud-init script to fix this upon deployment:
    #!/bin/bash
    ADDR_PRIV=$(networkctl status bond0.1234 | grep " Address: " | awk '{print $2}')
    sed -i "s/- from: 192.168.0.0/24/- from: $ADDR_PRIV/g" /etc/netplan/50-cloud-init.yaml

Can this be fixed in MaaS please?

[Kevin Reeuwijk (kreeuwijk)]

Besides this, it would also be nice if there was a more automatic way of telling MaaS which subnet has the preferred default gateway.

If the NIC order was reversed in the above example, I currently have to run the "maas <profile> interface set-default-gateway" command to ensure the "gateway4:" parameter gets set for the NIC in the desired network. While this works, it's impossible to see from the GUI which setting is active.

If instead, a priority could be given to the gateway in each MaaS subnet configuration, then MaaS can auto-select the gateway with the best priority value and admins don't have to manually set the right gateway on a per-node basis.

[Anton Troyanov (troyanov)]

Hello Kevin,

> However the official documentation (https://netplan.io/reference#routing) notes that the "from:" parameter should be an IP address, not a CIDR.

I don't think that MAAS is doing anything wrong here by specifying CIDR and not IP address.
netplan docs for "routing-policy (mapping)" [0] says:
"For from, to and via, both IPv4 and IPv6 addresses are recognized, and must be in the form *addr/prefixlen* or addr."

I believe this issue happens because of wrong priority and how RPDB works:

"Each policy routing rule consists of a selector and an action predicate. The RPDB is
scanned in order of decreasing priority. The selector of each rule is applied to {source
address, destination address, incoming interface, tos, fwmark} and, if the selector
matches the packet, the action is performed." [1]

I know that some users in addition to MAAS are using "advanced-routing" charm [2] to get such configuration.

[0] - https://netplan.io/reference
[1] - https://manpages.ubuntu.com/manpages/xenial/man8/ip-rule.8.html
[2] - https://charmhub.io/advanced-routing

[Kevin Reeuwijk (kreeuwijk)]

Hi Anton,

I actually wonder if the documentation is wrong here, and if it should say that the "to:" parameter supports both "addr/prefixlen" or "addr" forms, but the "from:" parameter only supports "addr".

At least that is my experience, as at least local network access is working correctly when the "from" parameter is changed to the IP address of the interface instead of the CIDR.

It's interesting to see that the advanced-routing charm also uses "from" CIDRs though. If CIDRs are supported here, I cannot explain why basic connectivity is not working correctly directly after MaaS deploys a machine.

Are there any plans to give more control over network config from MaaS to remediate this issue?

[Igor Brovtsin (igor-brovtsin)]

I am marking this bug as `Invalid` since the thread looks like a feature request/question at this point.

Kevin, if you want to continue with this, please make a topic with a feature request on our Discourse: https://discourse.maas.io/. If you do so, please describe your expectations of that feature in detail.

[Igor Brovtsin (igor-brovtsin)]

Also, you can find feature request guidelines here: https://maas.io/docs/how-to-request-new-features

[Kevin Reeuwijk (kreeuwijk)]

Ok I'll do some more testing and may open an item on Discourse to revisit this.