Ubuntu
systemd package

Systemd fails to configure bridged network in LXC container

Bug #1863873 reported by xavier
28
This bug affects 5 people
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

In all our LXC containers running Bionic Beaver, Eoan Ermine or Focal Fossa, installing the latest systemd package results in losing network configuration.

It is still possible to configure the network "by hand" with /usr/sbin/ip, but of course, the configuration is lost at reboot.

An example is provided, followed by a complete procedure to reproduce the issue.

Affected container distributions
================================

Xenial Xerus systemd 229-4ubuntu21.27: OK, not affected
Bionic Beaver systemd 237-3ubuntu10.38: OK, not affected
Bionic Beaver systemd 237-3ubuntu10.39: BUGGY
Disco Dingo systemd 240-6ubuntu5.8: OK, not affected
Eoan Ermine systemd 242-7ubuntu3.6: OK, not affected
Eoan Ermine systemd 242-7ubuntu3.7: BUGGY
Focal Fossa systemd 244.2-1ubuntu1: BUGGY

Affected hosts
==============

Debian Buster with default 4.19.0-6-amd64, custom 5.3.9, 5.4.8 or 5.4.13 kernel
Ubuntu 16.04 lxc 2.0.8-0ubuntu1~16.04.2 (https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1863873/comments/7)

Example
=======

Example host bridge configuration
---------------------------------

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 00:25:90:2b:f1:60 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br1 state DOWN group default qlen 1000
    link/ether 00:25:90:2b:f1:61 brd ff:ff:ff:ff:ff:ff
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:25:90:2b:f1:60 brd ff:ff:ff:ff:ff:ff
    inet 192.168.252.24/24 brd 192.168.252.255 scope global br0
       valid_lft forever preferred_lft forever
    inet 192.168.193.203/24 brd 192.168.193.255 scope global br0:1
       valid_lft forever preferred_lft forever
    inet6 fe80::225:90ff:fe2b:f160/64 scope link
       valid_lft forever preferred_lft forever

Example container network configuration
---------------------------------------

lxc.net.0.type = veth
lxc.net.0.veth.pair = vps525389
lxc.net.0.flags = up
lxc.net.0.link = br0
lxc.net.0.hwaddr = 02:00:00:52:53:89
lxc.net.0.name = eth0
lxc.net.0.ipv4.gateway = 192.168.252.1
lxc.net.0.ipv4.address = 192.168.252.177/32

Example steps to reproduce, inside the container
------------------------------------------------

root@vps525389:~# lsb_release -rd
Description: Ubuntu 18.04.4 LTS
Release: 18.04
root@vps525389:~# apt-cache policy systemd
systemd:
  Installed: 237-3ubuntu10.38
  Candidate: 237-3ubuntu10.39
  Version table:
     237-3ubuntu10.39 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
 *** 237-3ubuntu10.38 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
        100 /var/lib/dpkg/status
     237-3ubuntu10 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
root@vps525389:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
1958: eth0@if1959: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:00:00:52:53:89 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.252.177/32 brd 255.255.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 xxxx:xxxx:x:xx::x:xxxx/128 scope global
       valid_lft forever preferred_lft forever
    inet6 xxxx::xx:xxxx:xxxx/64 scope link
       valid_lft forever preferred_lft forever
root@vps525389:~# apt install systemd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libnss-systemd libpam-systemd libsystemd0
Suggested packages:
  systemd-container policykit-1
The following packages will be upgraded:
  libnss-systemd libpam-systemd libsystemd0 systemd
4 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 3330 kB of archives.
After this operation, 7168 B of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libnss-systemd amd64 237-3ubuntu10.39 [104 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpam-systemd amd64 237-3ubuntu10.39 [107 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 systemd amd64 237-3ubuntu10.39 [2912 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libsystemd0 amd64 237-3ubuntu10.39 [206 kB]
Fetched 3330 kB in 3s (1274 kB/s)
(Reading database ... 18195 files and directories currently installed.)
Preparing to unpack .../libnss-systemd_237-3ubuntu10.39_amd64.deb ...
Unpacking libnss-systemd:amd64 (237-3ubuntu10.39) over (237-3ubuntu10.38) ...
Preparing to unpack .../libpam-systemd_237-3ubuntu10.39_amd64.deb ...
Unpacking libpam-systemd:amd64 (237-3ubuntu10.39) over (237-3ubuntu10.38) ...
Preparing to unpack .../systemd_237-3ubuntu10.39_amd64.deb ...
Unpacking systemd (237-3ubuntu10.39) over (237-3ubuntu10.38) ...
Preparing to unpack .../libsystemd0_237-3ubuntu10.39_amd64.deb ...
Unpacking libsystemd0:amd64 (237-3ubuntu10.39) over (237-3ubuntu10.38) ...
Setting up libsystemd0:amd64 (237-3ubuntu10.39) ...
Setting up systemd (237-3ubuntu10.39) ...
Setting up libnss-systemd:amd64 (237-3ubuntu10.39) ...
Setting up libpam-systemd:amd64 (237-3ubuntu10.39) ...
Processing triggers for dbus (1.12.2-1ubuntu1.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
root@vps525389:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
1958: eth0@if1959: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:00:00:52:53:89 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::ff:fe52:5389/64 scope link
       valid_lft forever preferred_lft forever

Complete procedure to reproduce the issue
=========================================

It is here assumed that there is a DHCP server available elsewhere on the network.

Set-up
------

1. Install an amd64 Debian Buster (default network install),

2. create a bridge on the host with a static IP and deactivate DHCP, in `/etc/network/interfaces`,
```
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

## The primary network interface
#allow-hotplug ens18
#iface ens18 inet dhcp
## This is an autoconfigured IPv6 interface
#iface ens18 inet6 auto

iface ens18 inet manual

auto br0
iface br0 inet static
    address 192.168.1.168
    netmask 255.255.255.0
    gateway 192.168.1.220
    bridge_ports ens18
```

3. reboot the host,
```bash
reboot
```

4. install lxc and create a bionic amd64 container,
```bash
apt install lxc
lxc-create -t download -n bionic
```

5. on the host, modify the network configuration of the container to use the bridge with a static IP in `/var/lib/lxc/bionic/config`,
```
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template:
# Template script checksum (SHA-1): 273c51343604eb85f7e294c8da0a5eb769d648f3
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf

# For Ubuntu 14.04
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none bind,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0
lxc.arch = linux64

# Container specific configuration
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1
lxc.rootfs.path = dir:/var/lib/lxc/bionic/rootfs
lxc.uts.name = bionic

## Network configuration
#lxc.net.0.type = empty

# Network configuration
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = br0
lxc.net.0.name = eth0
lxc.net.0.ipv4.gateway = 192.168.1.220
lxc.net.0.ipv4.address = 192.168.1.169/32
```

6. inside the container, install the systemd packages without the bug, and deactivate DHCP in `/etc/netplan/10-lxc.yaml`,
```bash
lxc-start -n bionic
lxc-attach -n bionic
apt install systemd=237-3ubuntu10.38 libsystemd0=237-3ubuntu10.38 libnss-systemd=237-3ubuntu10.38 libpam-systemd=237-3ubuntu10.38
sed -i 's/true/false/' /etc/netplan/10-lxc.yaml
exit
```

7. stop the container.
```bash
lxc-stop -n bionic
```

Let’s do it
-----------

1. Start the container and check the IP config, which should be ok,
```bash
lxc-start -n bionic
lxc-attach -n bionic
ip a
```

2. upgrade the system and check the IP config, the static IP is gone.
```bash
apt upgrade
ip a
exit
```

If systemd is downgraded again to 237-3ubuntu10.38, the IP is back at the next reboot of the container.

See original description
xavier (xavier2)
description: updated
description: updated
description: updated
description: updated
xavier (xavier2)
description: updated
xavier (xavier2)
description: updated
description: updated
xavier (xavier2)
description: updated
Revision history for this message
Dan Streetman (ddstreet) wrote :
Download full text (4.7 KiB)

I can't reproduce your error:

ubuntu@lp1863873-b:~$ dpkg -l systemd|grep ii
ii systemd 237-3ubuntu10.38 amd64 system and service manager
ubuntu@lp1863873-b:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
50: eth0@if51: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:40:f5:e0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.202.51.98/24 brd 10.202.51.255 scope global dynamic eth0
       valid_lft 3584sec preferred_lft 3584sec
    inet6 fe80::216:3eff:fe40:f5e0/64 scope link
       valid_lft forever preferred_lft forever
ubuntu@lp1863873-b:~$ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  libnss-systemd libpam-systemd libsystemd0 libudev1 systemd systemd-sysv udev
7 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 4502 kB of archives.
After this operation, 10.2 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libnss-systemd amd64 237-3ubuntu10.39 [104 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libsystemd0 amd64 237-3ubuntu10.39 [206 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpam-systemd amd64 237-3ubuntu10.39 [107 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 systemd amd64 237-3ubuntu10.39 [2912 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 udev amd64 237-3ubuntu10.39 [1102 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libudev1 amd64 237-3ubuntu10.39 [56.1 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 systemd-sysv amd64 237-3ubuntu10.39 [13.9 kB]
Fetched 4502 kB in 1s (3046 kB/s)
(Reading database ... 28660 files and directories currently installed.)
Preparing to unpack .../libnss-systemd_237-3ubuntu10.39_amd64.deb ...
Unpacking libnss-systemd:amd64 (237-3ubuntu10.39) over (237-3ubuntu10.38) ...
Preparing to unpack .../libsystemd0_237-3ubuntu10.39_amd64.deb ...
Unpacking libsystemd0:amd64 (237-3ubuntu10.39) over (237-3ubuntu10.38) ...
Setting up libsystemd0:amd64 (237-3ubuntu10.39) ...
(Reading database ... 28660 files and directories currently installed.)
Preparing to unpack .../libpam-systemd_237-3ubuntu10.39_amd64.deb ...
Unpacking libpam-systemd:amd64 (237-3ubuntu10.39) over (237-3ubuntu10.38) ...
Preparing to unpack .../systemd_237-3ubuntu10.39_amd64.deb ...
Unpacking systemd (237-3ubuntu10.39) over (237-3ubuntu10.38) ...
Preparing to unpack .../udev_237-3ubuntu10.39_amd64.deb ...
Unpacking udev (237-3ubuntu10.39) over (237-3ubuntu10.38) ...
Preparing to unpack .....

Read more...

Changed in systemd (Ubuntu):
status: New → Incomplete
Revision history for this message
xavier (xavier2) wrote :

Did you do your test in an unprivileged LXC container?

Revision history for this message
xavier (xavier2) wrote :

The problem is also present in privileged LXC containers. Example with full LXC configuration below, on a host with a custom 5.4.13 kernel.

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf

# For Ubuntu 14.04
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none bind,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0
lxc.include = /usr/share/lxc/config/userns.conf

# For Ubuntu 14.04
lxc.mount.entry = /sys/firmware/efi/efivars sys/firmware/efi/efivars none bind,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none bind,optional 0 0
lxc.arch = linux64
lxc.hook.pre-start = /usr/local/share/lxc/hooks/pre-start.sh vps526706
lxc.hook.post-stop = /usr/local/share/lxc/hooks/post-stop.sh vps526706
lxc.hook.destroy = /usr/local/share/lxc/hooks/destroy.sh vps526706
lxc.mount.fstab = /lxc/vps526706/fstab
lxc.rootfs.path = dir:/lxc/vps526706/rootfs
lxc.uts.name = vps526706

# Network configuration
lxc.net.0.type = veth
lxc.net.0.veth.pair = vps526706
lxc.net.0.flags = up
lxc.net.0.link = br0
lxc.net.0.hwaddr = 02:00:00:52:67:06
lxc.net.0.name = eth0
lxc.net.0.ipv4.gateway = 192.168.252.1
lxc.net.0.ipv4.address = 192.168.252.171/32

Revision history for this message
xavier (xavier2) wrote :

The problem is also present in eoan and focal, but not disco (unprivileged LXC containers, 5.4.13 kernel).

Revision history for this message
xavier (xavier2) wrote :

Just in case, I also removed any arptables or iptables on the host. The problem is still present.
Here is an even simpler LXC configuration file to reproduce (privileged LXC container on Debian Buster, 5.4.13 kernel):

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = linux64

# Container specific configuration
lxc.hook.pre-start = /usr/local/share/lxc/hooks/pre-start.sh vps526706
lxc.hook.post-stop = /usr/local/share/lxc/hooks/post-stop.sh vps526706
lxc.hook.destroy = /usr/local/share/lxc/hooks/destroy.sh vps526706
lxc.mount.fstab = /lxc/vps526706/fstab
lxc.rootfs.path = dir:/lxc/vps526706/rootfs
lxc.uts.name = vps526706

# Network configuration
lxc.net.0.type = veth
lxc.net.0.veth.pair = vps526706
lxc.net.0.flags = up
lxc.net.0.link = br0
lxc.net.0.hwaddr = 02:00:00:52:67:06
lxc.net.0.name = eth0
lxc.net.0.ipv4.gateway = 192.168.252.1
lxc.net.0.ipv4.address = 192.168.252.171/32

Revision history for this message
xavier (xavier2) wrote :

Here is a full procedure to reproduce the issue.

Set-up
======

1. Install an amd64 Debian Buster (default network install),

2. install lxc and create a bionic amd64 container,
```bash
apt install lxc
lxc-create -t download -n bionic
lxc-start -n bionic
```

3. inside the container, deactivate dhcp (dhcp4: false )in `/etc/netplan/10-lxc.yaml`, and install the systemd packages without the bug,
```bash
lxc-attach -n bionic
sed -i 's/true/false/' /etc/netplan/10-lxc.yaml
apt install systemd=237-3ubuntu10.38 libsystemd0=237-3ubuntu10.38 libnss-systemd=237-3ubuntu10.38 libpam-systemd=237-3ubuntu10.38
exit
```

4. create a bridge on the host with a static IP and deactivate dhcp, in `/etc/network/interfaces`,
```
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

## The primary network interface
#allow-hotplug ens18
#iface ens18 inet dhcp
## This is an autoconfigured IPv6 interface
#iface ens18 inet6 auto

iface ens18 inet manual

auto br0
iface br0 inet static
    address 192.168.1.168
    netmask 255.255.255.0
    gateway 192.168.1.220
    bridge_ports ens18
    bridge_stp off
    bridge_waitport 0
    bridge_fd 0
```

5. on the host, modify the network configuration of the container to use the bridge with a static IP in `/var/lib/lxc/bionic/config`,
```
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template:
# Template script checksum (SHA-1): 273c51343604eb85f7e294c8da0a5eb769d648f3
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf

# For Ubuntu 14.04
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none bind,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0
lxc.arch = linux64

# Container specific configuration
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1
lxc.rootfs.path = dir:/var/lib/lxc/bionic/rootfs
lxc.uts.name = bionic

## Network configuration
#lxc.net.0.type = empty

# Network configuration
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = br0
lxc.net.0.name = eth0
lxc.net.0.ipv4.gateway = 192.168.1.220
lxc.net.0.ipv4.address = 192.168.1.169/32
```

6. reboot the host.
```bash
reboot
```

Let’s do it
===========

1. Start the container and check the IP config, which should be ok,
```bash
lxc-start -n bionic
lxc-attach -n bionic
ip a
```

2. upgrade the system and check the IP config, the static is gone.
```bash
apt upgrade
ip a
exit
```

If systemd is downgraded again to 237-3ubuntu10.38, the IP is back at the next reboot of the container.

xavier (xavier2)
description: updated
Revision history for this message
Matthias Hellinghausen (hellimat) wrote :

same behavior here!

systemd 237-3ubuntu10.38 -> IPV4 Adress after start of lxc-Container, everything OK
systemd 237-3ubuntu10.39 -> NO IPV4 Adress in Container after lxc-start

Host: Ubuntu 16.04 lxc 2.0.8-0ubuntu1~16.04.2
Container: Ubuntu 18.04

xavier (xavier2)
description: updated
xavier (xavier2)
description: updated
xavier (xavier2)
description: updated
description: updated
xavier (xavier2)
description: updated
description: updated
xavier (xavier2)
description: updated
description: updated
xavier (xavier2)
description: updated
description: updated
description: updated
description: updated
xavier (xavier2)
description: updated
Revision history for this message
Thomas Wien (thwien) wrote :

I can confirm this behavior after update to systemd 237-3ubuntu10.39 on privileged and unprivileged containers. We classify this bug as critical because in the next monthly update cycle via ansible orchestration we would expect failing all Bionic LXC containers. The only workaround seems to manually define a static network configuration in /etc/netplan/10-lxc.yaml via lxc-attach. But this is not an acceptable solution for 180 servers we run.

Some our Plesk servers already failed completely due to automatic upgrades Plesk triggered daily. Websites were down for hours until we figured out the reason.

I would appreciate that someone solves this problem due to urgency.

Thank you in advance.

Revision history for this message
Matthias Hellinghausen (hellimat) wrote :

Possible workaround:

switch back to ifupdown and don't use netplan

sudo apt-get update
sudo apt-get install ifupdown

cat << EOF > /etc/network/interfaces
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
EOF

systemctl stop systemd-networkd.socket systemd-networkd \
networkd-dispatcher systemd-networkd-wait-online
systemctl disable systemd-networkd.socket systemd-networkd \
networkd-dispatcher systemd-networkd-wait-online

sudo apt-get purge netplan.io

xavier (xavier2)
description: updated
Revision history for this message
Cyril (mistercyrilubuntu) wrote :

I have the same problem.

Revision history for this message
Dan Streetman (ddstreet) wrote :

> lxc.net.0.ipv4.gateway = 192.168.252.1
> lxc.net.0.ipv4.address = 192.168.252.171/32

> 3. inside the container, deactivate dhcp (dhcp4: false )in `/etc/netplan/10-lxc.yaml`,

ok, this is your problem; you're misconfiguring the instance. Unfortunately, you were just 'lucky' that it worked before, but this is absolutely intended behavior of systemd: when you tell it to manage an interface, it expects to control 100% of the interface's addresses and routing (with the minor exception that newer systemd has added 'KeepConfiguration' config for specific situations where externally-configured networking should be kept on the interface, such as some HA setups).

Your problem here is that you're configuring the interface networking *outside* networkd, but then also telling networkd it should manage the interface. It's fine to use lxc to externally configure the interface networking, but then inside the container you must make sure networkd does not think it's managing eth0, which means you must completely remove the 'eth0' section inside the netplan config yaml - if you leave any 'eth0' config there at all, it will create a network file telling networkd that it should manage eth0, even though it's not assigning any ipv4 addresses (it does setup ipv6 link-local by default, however). The file that netplan creates for the eth0 interface will be /run/systemd/network/10-netplan-eth0.network.

You can see in the output of networkctl:

root@bionic:/# networkctl list
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
 29 eth0 ether degraded configured

since in your use case you do not use (or want!) networkd to manage eth0, you want the 'SETUP' column to show 'unmanaged', not 'configured'.

What you need to do is, instead of simply changing the netplan eth0 dhcp4 configuration to false, you need to remove the eth0 section completely. If that's the only section you have, you'll need to remove the entire 'ethernets' section, or just remove the yaml file entirely (or rename it to anything that doesn't end in '.yaml').

Then, when you 'netplan apply' or reboot, you should notice that the /run/systemd/network/ file for eth0 is no longer created, and networkd no longer thinks it manages eth0, and your externally-configured eth0 networking should work again.

I hope this works for you, and very sorry for the disruption of your production environment.

Changed in systemd (Ubuntu):
status: Incomplete → Invalid
Revision history for this message
xavier (xavier2) wrote :

Thank you so much for your very detailed answer! I understand my mistake and fixed our set-up.

Revision history for this message
Thomas Wien (thwien) wrote :

Thanks a lot for your explanations and workaround examples.

I expanded our ansible update playbooks to check if /etc/netplan/10-lxc.yaml is installed, then to remove that file, runs "netplan apply" and then remove the package "netplan.io". It seems to work in our environment, without loosing the network configuration while updating and also after reboot.

Here is the ansible section we tested. Maybe it is helpful to anyone else. This script comes with no warranty and should be tested in your environment thoroughly.

- name: Is Netplan configuring LXC?
  stat:
    path: /etc/netplan/10-lxc.yaml
  register: netplan
- name: Remove networkd/netplan configuration
  file:
    path: /etc/netplan/10-lxc.yaml
    state: absent
  when: "netplan.stat.exists == true and ansible_virtualization_type == 'lxc'"
- name: Reload networkd/netplan
  command: netplan apply
  when: "netplan.stat.exists == true and ansible_virtualization_type == 'lxc'"
- name: Remove netplan.io
  apt:
    name: netplan.io
    state: absent
  when: "netplan.stat.exists == true and ansible_virtualization_type == 'lxc'"

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.