multi-homed host static routes formating

[Sean Wheller]

MAAS Version 2.1.1+bzr5544-0ubuntu1 (16.04.1)

Instruction to replicate:
2 Subnets
192.168.21.0
192.168.22.0

Set default GW
192.168.21.254
192.168.22.254

route add both to 192.168.1.0

Deploy host then see /etc/network/interfaces

auto eno1
iface eno1 inet static
    gateway 192.168.21.254
    dns-nameservers 192.168.1.1 192.168.3.1 8.8.8.8
    address 192.168.21.44/24
    mtu 1500

auto eno2
iface eno2 inet static
    dns-nameservers 192.168.1.1 192.168.3.1 8.8.8.8
    address 192.168.22.44/24
    mtu 1500
post-up route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.21.254 metric 0 || true
pre-down route del -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.21.254 metric 0 || truepost-up route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.22.254 metric 0 || true
pre-down route del -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.22.254 metric 0 || true

The result:

ubuntu@ops-lab-vod-04:~$ route -v
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.21.254 0.0.0.0 UG 0 0 0 eno1
192.168.1.0 192.168.21.254 255.255.255.0 UG 0 0 0 eno1
192.168.21.0 * 255.255.255.0 U 0 0 0 eno1
192.168.22.0 * 255.255.255.0 U 0 0 0 eno2
ubuntu@ops-lab-vod-04:~$

Problem:
Example above Line 14 reads in single string and should be two lines

pre-down route del -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.21.254 metric 0 || true
post-up route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.22.254 metric 0 || true

[Mike Pontillo (mpontillo)]

This looks like a problem in curtin's rendering of /etc/network/interfaces, but I don't know if it's a problem that has already been addressed. (You are using a MAAS release a couple of minor revisions behind; there is a newer one in xenial-proposed.)

So that we can confirm if this is a new issue or not, can you post the output of `apt-cache policy curtin` on the MAAS server?

Thanks in advance.

[Sean Wheller (seanwhe)]

Hi, thanks for the feed back and for looking into this problem.

See below (eof) is the apt policy. Just one more thing on this subject. I think, in addition to the formatting, that the actual implementation method for a multi-homed system may not work.

I think a correct implementation will require the addition of tables in /etc/iproute2/rt_tables
then;
It may then be a better implementation if the routes and rules are added to /etc/rc.local
This would result in the content of /etc/rc.local containing something like
(snipped for brevity just showing a single interface):
      ip route add 192.168.21.0/24 dev enp7s6 src 192.168.21.17 table lab
      ip route add default via 192.168.21.254 dev enp7s6 table lab
      ip rule add from 192.168.21.17/24 table lab
      ip rule add to 192.168.21.17/24 table lab
...
For each interface.

sean@ops-jhb-infra-06:~$ apt-cache policy curtin
curtin:
  Installed: (none)
  Candidate: 0.1.0~bzr425-0ubuntu1~16.04.1
  Version table:
     0.1.0~bzr425-0ubuntu1~16.04.1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages
     0.1.0~bzr365-0ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages
sean@ops-jhb-infra-06:~$

[Sean Wheller (seanwhe)]

Hello again,

I can confirm in my lan setup (see reference diagram), that addition of pre and post up stanza does not work.
[ https://www.lucidchart.com/documents/view/24043097-9fc5-4d88-a598-a825e5bd2217 ]

I have the following working implementation:

root@ops-lab-ter-01:/home/ubuntu# cat /etc/iproute2/rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
1 LABPUB
root@ops-lab-ter-01:/home/ubuntu#

root@ops-lab-ter-01:/home/ubuntu# cat /etc/rc.local
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

ip route add 192.168.22.0/24 dev eno2 src 192.168.22.1 table LABPUB
ip route add default via 192.168.22.254 dev eno2 table LABPUB
ip rule add from 192.168.22.1/24 table LABPUB
ip rule add to 192.168.22.1/24 table LABPUB

exit 0
root@ops-lab-ter-01:/home/ubuntu#

root@ops-lab-ter-01:/home/ubuntu# cat /etc/network/interfaces
auto lo
iface lo inet loopback
    dns-nameservers 192.168.1.1 192.168.3.1 8.8.8.8
    dns-search prepaidmeters.local maas prepaidmeters.example prepaidmeters.test

auto eno1
iface eno1 inet static
    dns-nameservers 192.168.1.1 192.168.3.1 8.8.8.8
    gateway 192.168.21.254
    address 192.168.21.1
    netmask 255.255.255.0
    broadcast 192.168.21.254
    mtu 1500

auto eno2
iface eno2 inet static
    dns-nameservers 192.168.1.1 192.168.3.1 8.8.8.8
    address 192.168.22.1
    netmask 255.255.255.0
    broadcast 192.168.22.254
    network 192.168.22.0
    mtu 1500

source /etc/network/interfaces.d/*.cfg
root@ops-lab-ter-01:/home/ubuntu#

Please note that I have manually edited these files to obtain this implementation.
I have also used the more traditional stanza format for iface definitions.

I am not sure how to make curtin or cloud-init produce these results.

I don't know if others would agree, but I find this to be a clearer, more flexible and more persistent method to implement the routing. However, this implementation is a test case to meet the requirements of OpenStack so I would assume that the implementation suggested above is of greater benefit than just MAAS.

[Mike Pontillo (mpontillo)]

Right now I'm not clear on if we would need to model any additional information about your setup in MAAS to make this work for you, so I'm marking this bug 'Opinion' there.

We may still want to investigate if curtin can do a better job rendering the routes.

Can you explain why you are using a separate 'table' in this setup? I'm not clear on why that is necessary. It seems like you simply want a default route via one interface, and you want a static to one specific subnet via the secondary interface. (That is, you don't have any weird situations like an overlapping subnet, which would require a custom metric to be set, etc.)

[Mike Pontillo (mpontillo)]

I just ran across this blog post, which was helpful for me to understand why you might be doing what you're doing:

http://blog.scottlowe.org/2013/05/29/a-quick-introduction-to-linux-policy-routing/

I suppose you don't want to mix traffic on the two networks, so you are trying to ensure that traffic sourced from the 'lab' network interface will only use lookups from the 'lab' table, and not be routed via the other interface toward the internet. Is that correct?

[Sean Wheller (seanwhe)]

Hello Mike,

Thanks for the update.

Yes you are correct. Again, I caution that my intent here is focused on a network architecture suitable for deploying OpenStack so may not be applicable in every circumstance. However, I do believe MAAS should be capable of handling any network architecture.

Majority of the reference models I have seen, stipulate all hosts are multi-homed so that they may support a management network and a public network. By multi-homed I mean a network host with 2 or more NICs each connected to a different subnet and where there is no IP Forwarding between these NICs. Each NIC should have its own Gateway.

Since it is not possible to define /etc/networks/interfaces with a gateway for each iface, one must resort to ip routes and rules. If one does not do so, ifaces without gateway will not be routed.

There are a number of use cases. However, our use case is using MAAS + Juju to deploy OpenStack. MAAS makes the bare metal ready. When Juju is used to deploy on a host in state Ready, the definitions provided by MAAS are deployed and then Juju goes to work. The present result of the deployed OS is a multi-homed device with only one routable interface. This then requires manual intervention. Of course that is less than desirable since each time you deploy to that metal you must manually intervene.

Here is a result.

If you are on host with network 192.168.1.0 and you want to communicate with a multi-homed host with one nic in 192.168.21.0 and another nic in 192.168.22.0.

traceroute 192.168.21.1
traceroute to 192.168.21.1 (192.168.21.1), 30 hops max, 60 byte packets
 1 192.168.1.254 (192.168.1.254) 1.178 ms 1.159 ms 1.393 ms
 2 192.168.1.253 (192.168.1.253) 1.381 ms 1.648 ms 1.914 ms
 3 192.168.21.1 (192.168.21.1) 1.109 ms 1.113 ms 1.111 ms

traceroute 192.168.22.1
traceroute to 192.168.22.1 (192.168.22.1), 30 hops max, 60 byte packets
 1 192.168.1.254 (192.168.1.254) 1.312 ms 1.299 ms 1.605 ms
 2 192.168.1.253 (192.168.1.253) 2.814 ms 1.277 ms 1.584 ms
 3 * * *
 .. snip
30 * * *

Despite the localhost having a route path to both subnets it will not be able to route to the interface that does not have a gateway defined. The traffic can reach the interface but there is no route back.

Hope this helps

[Sean Wheller (seanwhe)]

Sorry I should add, that IMHO, the most flexible way of implementation would be as described in Scot's blog post [ http://blog.scottlowe.org/2013/05/29/a-quick-introduction-to-linux-policy-routing/ ] as this method provides the widest and most flexible number of options.

[Mike Pontillo (mpontillo)]

All right; you've convinced me. ;-) I think we should consider adding source routing as a feature. I'll put it on the wishlist.

I would also point out that MAAS 2.1 introduced static routes, which may do mostly what you want. (If you add the lab network's supernet(s) as subnet(s) in MAAS, you should be able to define the lab gateway as the gateway for its managed network(s). But I can see how this would be less than ideal.)

[Sean Wheller (seanwhe)]

Hi Mike,

Yes, /MAAS/#/networks?by=fabric will list the networks automatically detected during the MAAS install. In this case, MAAS can have no clue as to the public subnet even after the commissioning of a node and despite detecting that there is another NIC. So, the ability for MAAS to allow manual addition and config of a subnet is spot on and works if you define a "static routes" to the manually created subnet with the exception of the initial rendering problem, which was the initial intent with this bug.

What prompted me to go further in this was when I manually corrected the rendering problem I found that the manually corrected solution which entailed simply adding in a linefeed at the correct position, did not work.

Hence, I implemented the policy based solution I know and trust and it worked.

So ... +1 for policy-based routing and enabling customers to implement policies that selectively cause packets to take different paths.

... and thanks to you for taking it under consideration.