After unattended-upgrade of systemd IP configuration removed from interfaces

[Domonkos Tomcsanyi]

Info:
Ubuntu Server 18.04 LTS, 4.15.0-43-generic, fresh installation (December, 2018).
HP DL380 server

I had this issue already twice in the past couple of months. After unattended upgrade of systemd my server's public IP address is removed from the interface.
I statically configure the IP address using the interfaces file. I have opted out of the YML based network configuration. This is my apt history log:

Start-Date: 2019-04-09 06:15:01
Commandline: /usr/bin/unattended-upgrade
Upgrade: systemd-sysv:amd64 (237-3ubuntu10.13, 237-3ubuntu10.19)
End-Date: 2019-04-09 06:15:07

Start-Date: 2019-04-09 06:15:12
Commandline: /usr/bin/unattended-upgrade
Upgrade: udev:amd64 (237-3ubuntu10.13, 237-3ubuntu10.19), libudev1:amd64 (237-3ubuntu10.13, 237-3ubuntu10.19)
End-Date: 2019-04-09 06:15:56

Start-Date: 2019-04-09 06:15:59
Commandline: /usr/bin/unattended-upgrade
Upgrade: libsystemd0:amd64 (237-3ubuntu10.13, 237-3ubuntu10.19), libpam-systemd:amd64 (237-3ubuntu10.13, 237-3ubuntu10.19), systemd:amd64 (237-3ubuntu10.13, 237-3ubuntu10.19), libnss-systemd:amd64 (237-3ubuntu10.13, 237-3ubuntu10.19)
End-Date: 2019-04-09 06:16:15

After that my server was unavailable from the Internet. When I logged in via out of band management and simply brought down and up the interface everything went back to normal. As said above this is the second time I had this issue. Last time it was unattended upgrades and it was systemd again.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Joi Owen (jlellis)]

Two of our Ubuntu 18.04 hosts applied this update this morning and both dropped offline. They're both using netplan and have bonded interfaces. The bond layer was offline. I had to reboot to get the network stack running again. I left one host offline so the coworker who configured them can look it over later this morning. Two identical hosts did not apply this systemd update this morning and they're both still online.

[Dan Streetman (ddstreet)]

> I statically configure the IP address using the interfaces file.

you mention the interfaces file, but this bug is against systemd - what mechanism are you using to configure your network? For a fresh install of 18.04, you will get netplan by default, using systemd-networkd as its backend; is that what you're using, or something else?

> I have opted out of the YML based network configuration.

you mean you replaced netplan with something else? ifupdown?

[Domonkos Tomcsanyi (tomcsanyid)]

I have followed the instructions in the /etc/network/interfaces file to ditch netplan and use ifupdown:

# To re-enable ifupdown on this system, you can run:
# sudo apt install ifupdown

As an extra measure I have removed netplan packages completely:

~# grep netplan .bash_history
apt remove nplan netplan.io

I have opened the bug against systemd, because I think it is the one responsible for the issue, but I could be wrong. It could be the unattended-upgrades package with the combination of systemd as well, or something different. I have written down the way my network configuration works so that enough information of my system is available - especially in this case where networking itself seems to have a hiccup.
Weirdly enough my virtual bridge (configured via the same /etc/network/interfaces file) supporting my VMs running on the server has not lost its IP configuration. This + Joi's experience indicate to me that this is not an issue of how the network configuration is applied on the server, it is an issue of something on a lower level possibly, and the loss of network connection is just an inconvenient result, and not the issue itself.

[Joi Owen (jlellis)]

I followed up with my coworker who originally built our two misbehaving boxes. He also ended up having to reboot the second box I left for him, to get the br01 bridge configuration back in working order. Simply running 'netplan apply' and restarting the systemd network service did not resolve it, but a full reboot did. We are using the default 18.04 netplan setup, we haven't reversed back to ifupdown like the other reporter.

Both hosts lost their networking immediately after the systemd update was applied this morning, which installed version 237-3ubuntu10.19. I can delve through them to find any logs or config files you want. These are Supermicro rack mounts with dual 10G interfaces, built as headless servers, so no ubuntu-desktop or X11 is installed on them.

We have sssd installed on these with the required pam customizations to configure sssd to live in our Active Directory environment. pam-auth-updated printed about local mods to common-* so it wasn't updating. Those are the only unusual outputs from this mornings unattended-upgrades-dpkg.log file.

I can attach any logs or other files you care to view.

[Joi Owen (jlellis)]

I've done some more digging into the systemd package update issue. We have a number of Ubuntu 18.04 hosts, most of them are virtual machines running simple eth0 configurations under Hyper-V. All of those VMs updated the systemd package without issue. The only hosts which had issues after the systemd package upgrade were the two SuperMicros running bonded 10G interfaces. On both of those hosts, the bond defined in netplan seems to have lost its configuration during the systemd upgrade. There were multiple kern.log messages regarding the bond leaving one of its ports in a disabled state after the update. Both hosts logged this on the same second that dpkg.log shows the systemd package itself entering 'status installed', so I'm confident it was that systemd package rather than pam-systemd that followed several seconds later.

I am still willing to provides logs and configurations if you tell me what you need to see.

[Domonkos Tomcsanyi (tomcsanyid)]

Is the information provided already enough to change this report's status? I think we have provided details, as well as offered to give even more (thank you Joi!) if required, so it would be great to follow up on this.
Thanks!

[Unishop (unishop)]

Got same thing, but not on all virtual machines.

[Domonkos Tomcsanyi (tomcsanyid)]

Today the same issue happened again. After systemd update my server became unavailable...

[Domonkos Tomcsanyi (tomcsanyid)]

Same again today. I will need to create some countermeasure, e.g. after every unattended upgrade a script that brings down and up the interfaces, otherwise I will not be able to keep my services availability. Kind of disappointed in this, but at least it will be a solution.

[gschoenberger (schoeni-georg)]

I can confirm this issue on 2 of our VMs running Ubuntu 18.04.
Today I lost a secondary IP on one of the interfaces, after unattended upgrades installed a new systemd package.

Any updates on this issue?

[Dan Streetman (ddstreet)]

please reopen if this is still an issue

[Todor Dragnev (t0d0r)]

I just hit it, after system upgrades of the following packages:

Start-Date: 2022-08-30 06:13:35
Commandline: /usr/bin/unattended-upgrade
Upgrade: systemd-sysv:i386 (237-3ubuntu10.50, 237-3ubuntu10.54)
End-Date: 2022-08-30 06:13:36

Start-Date: 2022-08-30 06:13:42
Commandline: /usr/bin/unattended-upgrade
Upgrade: udev:i386 (237-3ubuntu10.50, 237-3ubuntu10.54), libudev1:i386 (237-3ubuntu10.50, 237-3ubuntu10.54)
End-Date: 2022-08-30 06:13:48

Start-Date: 2022-08-30 06:13:54
Commandline: /usr/bin/unattended-upgrade
Upgrade: libsystemd0:i386 (237-3ubuntu10.50, 237-3ubuntu10.54), libpam-systemd:i386 (237-3ubuntu10.50, 237-3ubuntu10.54), systemd:i386 (237-3ubuntu10.50, 237-3ubuntu10.54), libnss-systemd:i386 (237-3ubuntu10.50, 237-3ubuntu10.54)
End-Date: 2022-08-30 06:14:03