Bridges without an address fail to come online with netplan+networkd

[Mike Pontillo]

I configured a container as follows, using some bridges (virbr0 and virbr1, my NAT network and test network, respectively) that I had previously configured in libvirt:

lxc init ubuntu-daily:b bionic-maas2 -p '' -s default
lxc network attach virbr0 bionic-maas2 eth0 eth0
lxc network attach virbr1 bionic-maas2 eth1 eth1
lxc config set bionic-maas2 user.network-config "version: 2
ethernets:
  eth0:
    match:
      name: eth0
    dhcp4: true
  eth1:
    match:
      name: eth1
bridges:
  br0:
    interfaces: [eth1]
"
lxc start bionic-maas2
lxc exec bionic-maas2 bash

After executing the shell, I noticed that the bridge interface I specified had been created, but not brought online:

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether c6:0d:b9:b9:68:44 brd ff:ff:ff:ff:ff:ff
32: eth0@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:2d:b9:df brd ff:ff:ff:ff:ff:ff link-netnsid 0
34: eth1@if35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:ef:4a:5b brd ff:ff:ff:ff:ff:ff link-netnsid 0

# networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 br0 ether off unmanaged
 32 eth0 ether routable configured
 34 eth1 ether carrier configured

To work around the issue, I had to manually set the link up:

# ip link set dev br0 up

# networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 br0 ether degraded unmanaged
 32 eth0 ether routable configured
 34 eth1 ether carrier configured

4 links listed.

It doesn't really seem useful for netplan to configure a 'bridge interface but not set it to come online.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: cloud-init 17.2-30-gf7deaf15-0ubuntu1
ProcVersionSignature: Ubuntu 4.13.0-32.35-generic 4.13.13
Uname: Linux 4.13.0-32-generic x86_64
ApportVersion: 2.20.8-0ubuntu8
Architecture: amd64
Date: Fri Feb 9 01:29:50 2018
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)
user_data.txt:
 #cloud-config
 {}

[Ryan Harper (raharper)]

I suspect that since you provided no network configuration to the bridge, then it is considered unconfigured. networkctl seems to confirm.

Now, it looks like we could use:

BindCarrier=
A link name or a list of link names. When set, controls the behavior of the current link. When all links in the list are in an operational down state, the current link is brought down. When at least one link has carrier, the current interface is brought up.

I tested this, and it appears to work:

# cat /run/systemd/network/10-netplan-br0.network
[Match]
Name=br0

[Network]
BindCarrier=eth1
root@b1:~# networkctl status br0
● 2: br0
       Link File: n/a
    Network File: /run/systemd/network/10-netplan-br0.network
            Type: ether
           State: degraded (configured)
      HW Address: ee:17:6d:12:fe:c5
         Address: fe80::ec17:6dff:fe12:fec5
Carrier Bound To: eth1

root@b1:~# ip link show br0
2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether ee:17:6d:12:fe:c5 brd ff:ff:ff:ff:ff:ff

root@b1:~# ip link show eth1
94: eth1@if95: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:02:88:ea brd ff:ff:ff:ff:ff:ff link-netnsid 0

However, it's not clear to me that we want to bring down br0 if one or more of the interfaces were to go down, so this may be something that needs a better upstream fix.

This issue seemed to match:
https://github.com/systemd/systemd/issues/425

And some users seem to be working around by emitting units which are effectively a network-online hook, but that doesn't look very clean IMO.

[Mike Pontillo (mpontillo)]

Actually it might be nice in this case to, in fact, bind the carrier to eth1.

In this scenario, I was planning on hanging other nested VMs and/or containers on the virtual bridge inside the LXC. If the underlying link was down for whatever reason, it would be more honest for that state to propagate all the way to the bridge, so that the internal interfaces on the nested VMs/bridges also see the link as down.

Another approach I have used in the past is to simply monitor the link state of the interface that provides connectivity to the bridge, so that if it becomes link-down, the VMs on the bridge can still communicate with each other, but we're able to act on the link-down event by dropping a route from the bridge, so that the proper ICMP errors occur when the nested VMs try to route outside the network via that interface.

[Mike Pontillo (mpontillo)]

I can confirm that this is a workaround (specifying an address to configure the bridge with):

lxc config set bionic-maas2 user.network-config "version: 2
ethernets:
  eth0:
    match:
      name: eth0
    dhcp4: true
  eth1:
    match:
      name: eth1
bridges:
  br0:
    interfaces: [eth1]
    addresses:
     - 172.16.99.0/24
"

But since a bridge is a layer 2 concept, it's arguably a bug in systemd-networkd, since it should bring up the bridge automatically regardless of whether or not a L3 address exists on the interface.

[Mathieu Trudel-Lapierre (cyphermox)]

This also affects VLANs, I think I've seen cases in which they would remain down if there is no configured address.

[E. MAS (erwan-mas)]

Do we have another workaround ?

A bridge without IP is a very common settings when you are doing virtualization , or firewalling , vpn , between multiple interface .

[Nagy Ákos (e-contact-pecska-ro)]

As E. MAS (erwan-mas) wrote, it is very common setting, and in case of virtualization, container, etc if we need to define IP address on host, it is a security leak, because with that IP address from the virtual OS can access the host, that is a big problem.

[Jolrael (bas)]

I've ran into this issue too and have been looking around for a workaround.

I have the feeling this bug is the same as bug #1736975. I've tested the workaround mentioned there, which works for me.

I'm not 100% if this is indeed a duplicate, and if so, how to mark this bug as a duplicate.