[ubuntu 18.04][netplan] Machine network goes down when second gateway is configured

[bugproxy]

---Problem Description---
Machine network goes down (not sshable/pingable) when gateway is configured for second interface.

---uname output---
Linux ltciofvtr-bostonlc2 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = BostonLC bare-metal

---System Hang---
 Remove the gateway from second interface and machine is accessible again

---Steps to Reproduce---
I have 2 interfaces configured on my machine,
1. enP2p1s0f0 having public IP 9.40.xx.xx with default gateway 9.40.xx.1
2. second enP2p1s0f2 having private connections IP 40.1.xx.x with out gateway

# cat /etc/netplan/01-netcfg.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    enP2p1s0f0:
      addresses: [ 9.40.xxx.xx/24 ]
      gateway4: 9.xx.xx.1
      nameservers:
        search: [ xx.xx.ibm.com ]
        addresses:
                - 9.x.1.x

    enP2p1s0f2:
            dhcp4: no
            dhcp6: no
            addresses: [ 40.1.1.12/24 ]

# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 9.40.xx.1 0.0.0.0 UG 0 0 0 enP2p1s0f0
9.40.xx.0 0.0.0.0 255.255.255.0 U 0 0 0 enP2p1s0f0
40.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 enP2p1s0f2
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0

I see both the interfaces are pinging works fine.

But when I add gateway to the second interface i.e

# cat /etc/netplan/01-netcfg.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    enP2p1s0f0:
      addresses: [ 9.40.xxx.xx/24 ]
      gateway4: 9.xx.xx.1
      nameservers:
        search: [ xx.xx.ibm.com ]
        addresses:
                - 9.x.1.x

   enP2p1s0f2:
            dhcp4: no
            dhcp6: no
            addresses: [ 40.1.1.12/24 ]
            gateway4: 40.1.1.1

# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 40.1.1.1 0.0.0.0 UG 0 0 0 enP2p1s0f2 <<<<<< second gateway
0.0.0.0 9.40.xx.1 0.0.0.0 UG 0 0 0 enP2p1s0f0
9.40.xx.0 0.0.0.0 255.255.255.0 U 0 0 0 enP2p1s0f0
40.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 enP2p1s0f2
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0

run 'netplan apply'

After this I see both the interfaces are not pingable, machine is not sshable.

I see a similar issue and workaround being reported here : https://bugs.launchpad.net/netplan/+bug/1724666

== Comment: #2 - Pradeep Satyanarayana <email address hidden> - 2018-04-23 14:10:46 ==
One requires what is known as "Policy Based Routing" to address this issue. The workaround that is described in the bug that you mention:

https://bugs.launchpad.net/netplan/+bug/1724666

limits your options.

The complete solution to a setup that you have is well illustrated in:

http://www.microhowto.info/howto/ensure_symmetric_routing_on_a_server_with_multiple_default_gateways.html

If that does not work, then that would be a Linux Networking bug.

This appears to be a netplan limitation and you should consider mirroring the bug to Ubuntu and asking them to fix it. Googling around it appears that Netplan may not yet support "Policy Based Routing". Hence folks are suggesting workarounds to this problem.

[bugproxy (bugproxy)]

Default Comment by Bridge

[Mathieu Trudel-Lapierre (cyphermox)]

Sounds like this is "expected behavior". While gateway4 is available for all interfaces, it simply adds a new default route for the interface, always at a metric of 100 (if I'm not mistaken, but definitely always at the same metric value). In any case, the metric value is up to the default for networkd.

The solution around this is to set up your own default gateway via an explicit route:

route:
  - to: 0.0.0.0/0
    via: x.x.x.x
    metric: 50

Specifying the exact metric value such that it gains the right priority, if the interfaces are expected to behave with different priorities (ie. you usually want to use eth0, but if it happens to be down we'll use eth1).

For more complex scenarios, you will indeed want policy routing, which is also supported in 18.04 using netplan > 0.34 (18.04 released with netplan 0.37.1). This requires the use of "routes" along with "routing-policy". We've put an example on the netplan.io website on how to do this: https://netplan.io/examples#source-routing

[Manoj Iyer (manjo)]

Based on analysis on Comment #2, marking this bug as invalid, please re-open if this is still something you would like to discuss.

[Mathieu Trudel-Lapierre (cyphermox)]

There is now also another feature available in newer netplan versions (0.96 and above, in progress SRU for bionic and above, so it should be available soon on all supported releases...).

With this, you'd be able to use "dhcp4-overrides" to specify what metric to use for routes, and whether or not to install routes from DHCP. See https://netplan.io/reference#dhcp-overrides