Updating systemd kills network on bionic

Bug #1782709 reported by Uli.Heller on 2018-07-20
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Medium
Unassigned
systemd (Ubuntu)
Undecided
Unassigned

Bug Description

I do have an ubuntu bionic server installation without a graphical interface. The server
has two active network adapters, both connected to the internet. One is used for outgoing
internet traffic, the other for incoming. The incoming adapter lives on a public network segment
(something like 88.236.133.104/29). I do have multiple servers within this segment.

Typically, I update the server regularly, meaning:

* ssh to the incoming adapter: ssh 88.236.133.108
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get clean
* reboot

It used to work ok.

BUT: Today, "apt-get upgrade" seem to take really long when updating systemd.
After a could of minutes, I realized that the ssh session became inactive. I couldn't type
into it, I had to abort it via ~. .

* ssh from the internet to the incoming adapter was not working anymore - ssh 88.236.133.108
* ssh via another server within the public network segment worked
    * ssh 88.236.133.106 -> ssh 88.236.133.108
* After running "netplan apply" everything was fine again

Here my netplan configuration (I changed the ip addresses):

network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: yes
    eno2:
      dhcp4: no
      addresses: [88.236.133.108/29]
      #gateway4: 88.236.133.105
      routes:
       - to: 0.0.0.0/0
         via: 88.236.133.105
         table: 120
      routing-policy:
       - from: 88.236.133.108/32
         table: 120
       - to: 88.236.133.108/32
         table: 120

Unfortunately, I don't have the console output avalable anymore.
Shall I provide additional info?

Best regards, Uli
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jul 20 08:10 seq
 crw-rw---- 1 root audio 116, 33 Jul 20 08:10 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-07-14 (5 days ago)
InstallationMedia: Ubuntu-Server 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: IBM IBM System x -[794582G]-
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
ProcFB: 0 mgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.15.0-23-generic root=UUID=a41b3020-8359-4fe2-8fc3-1c97011f71ec ro
ProcVersionSignature: Ubuntu 4.15.0-23.25-generic 4.15.18
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-23-generic N/A
 linux-backports-modules-4.15.0-23-generic N/A
 linux-firmware 1.173.1
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic
Uname: Linux 4.15.0-23-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 05/07/2014
dmi.bios.vendor: IBM Corp.
dmi.bios.version: -[D6E162AUS-1.20]-
dmi.board.asset.tag: (none)
dmi.board.name: 00D4062
dmi.board.vendor: IBM
dmi.board.version: (none)
dmi.chassis.asset.tag: none
dmi.chassis.type: 23
dmi.chassis.vendor: IBM
dmi.chassis.version: none
dmi.modalias: dmi:bvnIBMCorp.:bvr-[D6E162AUS-1.20]-:bd05/07/2014:svnIBM:pnIBMSystemx-[794582G]-:pvr00:rvnIBM:rn00D4062:rvr(none):cvnIBM:ct23:cvrnone:
dmi.product.family: System x
dmi.product.name: IBM System x -[794582G]-
dmi.product.version: 00
dmi.sys.vendor: IBM

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1782709

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: bionic
Uli.Heller (uli-heller) wrote :

root@samson:~# apport-collect 1782709
ERROR: The python3-launchpadlib package is not installed. This functionality is not available.

Do I really have to install this package?

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
tags: added: apport-collected
description: updated

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

Uli.Heller (uli-heller) wrote :

I examined /var/log/dpkg.log. There is no pause in there. The complete upgrade process looks pretty normal to my. So I guess it is just a network configuration issue during the upgrade that prevented my terminal session from getting updated.

Joseph Salisbury (jsalisbury) wrote :

Do you have a way to reproduce this issue, or was it a one time event?

Changed in linux (Ubuntu):
status: Confirmed → Incomplete
importance: Undecided → Medium
Uli.Heller (uli-heller) wrote :
Download full text (7.2 KiB)

Today, just another "apt-get upgrade":

root@samson:~# apt-get upgrade
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
Paketaktualisierung (Upgrade) wird berechnet... Fertig
Die folgenden Pakete werden aktualisiert (Upgrade):
  base-files console-setup console-setup-linux grub-common grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed grub2-common keyboard-configuration
  libnss-systemd libpam-systemd libperl5.26 libsystemd0 libudev1 mdadm perl perl-base perl-modules-5.26 python3-cryptography python3-software-properties
  python3-update-manager screen snapd software-properties-common systemd systemd-sysv udev unattended-upgrades update-manager-core
29 aktualisiert, 0 neu installiert, 0 zu entfernen und 0 nicht aktualisiert.
Es müssen 30,9 MB an Archiven heruntergeladen werden.
Nach dieser Operation werden 176 kB Plattenplatz zusätzlich benutzt.
Möchten Sie fortfahren? [J/n]
Holen:1 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 base-files amd64 10.1ubuntu2.1 [58,0 kB]
Holen:2 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libperl5.26 amd64 5.26.1-6ubuntu0.2 [3.531 kB]
Holen:3 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 perl amd64 5.26.1-6ubuntu0.2 [201 kB]
Holen:4 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 perl-base amd64 5.26.1-6ubuntu0.2 [1.390 kB]
Holen:5 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 perl-modules-5.26 all 5.26.1-6ubuntu0.2 [2.762 kB]
Holen:6 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libnss-systemd amd64 237-3ubuntu10.3 [105 kB]
Holen:7 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libsystemd0 amd64 237-3ubuntu10.3 [204 kB]
Holen:8 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpam-systemd amd64 237-3ubuntu10.3 [108 kB]
Holen:9 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 systemd amd64 237-3ubuntu10.3 [2.896 kB]
Holen:10 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 udev amd64 237-3ubuntu10.3 [1.100 kB]
Holen:11 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libudev1 amd64 237-3ubuntu10.3 [53,8 kB]
Holen:12 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 systemd-sysv amd64 237-3ubuntu10.3 [11,7 kB]
Holen:13 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 console-setup-linux all 1.178ubuntu2.3 [982 kB]
Holen:14 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 console-setup all 1.178ubuntu2.3 [105 kB]
Holen:15 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 keyboard-configuration all 1.178ubuntu2.3 [372 kB]
Holen:16 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-update-manager all 1:18.04.11.4 [34,1 kB]
Holen:17 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 update-manager-core all 1:18.04.11.4 [8.500 B]
Holen:18 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 grub-efi-amd64 amd64 2.02-2ubuntu8.2 [46,8 kB]
Holen:19 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 grub2-common amd64 2.02-2ubuntu8.2 [531 kB]
Holen:20 http://de.archive.ubuntu.com/...

Read more...

Shelby Merrick (forkineye) wrote :

I recently installed a fresh system with 18.04 as well and have experienced this exact issue when attempting to "apt dist-upgrade" from ssh. In the two instances where systemd was being updated (a few weeks ago and today), my ssh connection was dropped and I'm unable to re-connect to the server. I started the upgrade from a screen session this time and will investigate further when I can get to the console this evening.

DEXTER (mydexterid) wrote :
Download full text (7.6 KiB)

Just happened to me too:
root@gw:/etc# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  eatmydata libeatmydata1 python3-blinker python3-jinja2 python3-json-pointer python3-jsonpatch python3-jsonschema python3-jwt python3-markupsafe python3-oauthlib
Use 'apt autoremove' to remove them.
The following packages will be upgraded:
  apport busybox-initramfs busybox-static docker.io initramfs-tools initramfs-tools-bin initramfs-tools-core landscape-common libidn11 libnss-systemd libpam-modules libpam-modules-bin libpam-runtime libpam-systemd libpam0g libpci3 libseccomp2 libsystemd0
  libudev1 libx11-6 libx11-data libxcb1 pciutils python3-apport python3-gi python3-problem-report systemd systemd-sysv udev unattended-upgrades
30 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 54.0 MB of archives.
After this operation, 37.2 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpam0g amd64 1.1.8-3.6ubuntu2.18.04.1 [55.1 kB]
Get:2 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpam-modules-bin amd64 1.1.8-3.6ubuntu2.18.04.1 [36.7 kB]
Get:3 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpam-modules amd64 1.1.8-3.6ubuntu2.18.04.1 [242 kB]
Get:4 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libnss-systemd amd64 237-3ubuntu10.15 [105 kB]
Get:5 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libsystemd0 amd64 237-3ubuntu10.15 [205 kB]
Get:6 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpam-systemd amd64 237-3ubuntu10.15 [108 kB]
Get:7 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 systemd amd64 237-3ubuntu10.15 [2,899 kB]
Get:8 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 udev amd64 237-3ubuntu10.15 [1,100 kB]
Get:9 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libudev1 amd64 237-3ubuntu10.15 [54.2 kB]
Get:10 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpam-runtime all 1.1.8-3.6ubuntu2.18.04.1 [37.1 kB]
Get:11 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 initramfs-tools all 0.130ubuntu3.7 [9,592 B]
Get:12 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 initramfs-tools-core all 0.130ubuntu3.7 [48.1 kB]
Get:13 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 initramfs-tools-bin amd64 0.130ubuntu3.7 [12.8 kB]
Get:14 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 busybox-initramfs amd64 1:1.27.2-2ubuntu3.1 [166 kB]
Get:15 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 systemd-sysv amd64 237-3ubuntu10.15 [12.0 kB]
Get:16 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libseccomp2 amd64 2.3.1-2.1ubuntu4.1 [39.1 kB]
Get:17 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libidn11 amd64 1.33-2.1ubuntu1.1 [45.9 kB]
Get:18 http://hu.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-gi amd64 3.26.1-2ubuntu1 [153 kB]
Get:19 http://hu.archiv...

Read more...

DEXTER (mydexterid) wrote :

And logging in on the console the apt process was waiting on for some answers:
ps aux | grep apt
root 13335 0.1 6.6 116500 67388 pts/1 S+ 19:30 0:00 apt upgrade
root 15048 0.0 0.3 11856 3628 pts/3 S+ 19:31 0:00 /bin/bash /usr/bin/ucf --three-way --debconf-ok /usr/share/unattended-upgrades/20auto-upgrades-disabled /etc/apt/apt.conf.d/20auto-upgrades
root 15090 0.0 0.4 24460 4976 pts/3 S+ 19:31 0:00 whiptail --backtitle Package configuration --title Configuring unattended-upgrades --output-fd 11 --nocancel --default-item keep the local version currently installed --menu A new version (/usr/share/unattended-upgrades/20auto-upgrades-disabled) of configuration file /etc/apt/apt.conf.d/20auto-upgrades is available, but the version installed currently has been locally modified. What do you want to do about modified configuration file 20auto-upgrades? 17 211 7 -- install the package maintainer's version keep the local version currently installed show the differences between the versions show a side-by-side difference between the versions show a 3-way difference between available versions do a 3-way merge between available versions start a new shell to examine the situation
root 16771 0.0 0.1 13136 1148 tty1 S+ 19:41 0:00 grep --color=auto apt

DEXTER (mydexterid) wrote :

I could only resolve the ssh issue with killing the apt process, and restarted networking.
This is a pretty bad issue imho. Upgrading packages through ssh should not disconnect the session, but even if it is, upgrading the systemd package should not result in not being able to log back in ever again!

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status: New → Confirmed
DEXTER (mydexterid) wrote :

Also I see this in the logs, but maybe not related (why can't I edit my comments?)

Mar 17 19:31:55 localhost systemd-networkd-wait-online[14189]: Event loop failed: Connection timed out
Mar 17 19:31:55 localhost apt-helper[14171]: E: Sub-process /lib/systemd/systemd-networkd-wait-online returned an error code (1)
Mar 17 19:31:55 localhost systemd[1]: apt-daily.service: Control process exited, code=exited status=100
Mar 17 19:31:55 localhost systemd[1]: apt-daily.service: Failed with result 'exit-code'.
Mar 17 19:31:55 localhost systemd[1]: Failed to start Daily apt download activities.

DEXTER (mydexterid) wrote :

Some more info:
 - The machine is a Virtualbox ubuntu bionic, and I have a snapshot before the upgrade, so I can reproduce it
 - I'm using ifconfig instead of netplan, because I need a ppp interface (for pppoe)
 - I have an /etc/network/interfaces file
 - it seems that the interface through which I'm using ssh loses it's IP address (defined in the interfaces file as static)
 - I also did these after installing bionic:
"systemctl disable systemd-networkd-wait-online.service
systemctl mask systemd-networkd-wait-online.service" because it was holding up the boot process for a long period

rjr162 (ronrossman) wrote :

I'd like to also confirm this bug, which we've been experiencing for the past few months since we first setup an 18.04 image for our deployment portal.

Initially I thought the cause MAY have been from us using ifupdown (we have to as our vCenter instance hasn't been upgraded to 6.5.x or 6.7 where Ubuntu 18.04 with NetPlan is officially supported).

I've checked logs and it seems that *every time* apt upgrade is ran and one of the systemd components are upgraded, the network dies.

You can do "ip addr" and see the interface has no addresses assigned.
A simple restart of networking and the IPs come back. This literally just happened this morning to two of our 18.04 LTS systems and the systemd packages that were updated were:

libsystemd0
systemd-sysv
libpam-systemd
systemd
libnss-systemd

The rest of the packages that were updated:
grub-common
grub2-common
udev
grub-pc
apache2-data
libudev1
grub-pc-bin
apache2-bin
libunistring2
apache2
apache2-utils

I also had another report on Feb 20th, and it also involved the following updates:
Start-Date: 2019-02-20 04:41:09
Commandline: /usr/bin/apt-get -o quiet=1 upgrade -y -o APT::Get::Show-Upgraded=true
Upgrade: libsystemd0:amd64 (237-3ubuntu10.12, 237-3ubuntu10.13), udev:amd64 (237-3ubuntu10.12, 237-3ubuntu10.13), libudev1:amd64 (237-3ubuntu10.12, 237-3ubuntu10.13), systemd-sysv:amd64 (237-3ubuntu10.12, 237-3ubuntu10.13), libpam-systemd:amd64 (237-3ubuntu10.12, 237-3ubuntu10.13), systemd:amd64 (237-3ubuntu10.12, 237-3ubuntu10.13), libnss-systemd:amd64 (237-3ubuntu10.12, 237-3ubuntu10.13), pciutils:amd64 (1:3.5.2-1ubuntu1, 1:3.5.2-1ubuntu1.1), libpci3:amd64 (1:3.5.2-1ubuntu1, 1:3.5.2-1ubuntu1.1)
End-Date: 2019-02-20 04:42:19

The interesting thing is, looking at my notes, in my case it MAY BE from the ifupdown which should "just work", as this 2nd VM also has that setup. I looked and it updated this morning the same as the first machine but I didn't have a report of it being offline. Now on this 2nd machine, I did also do an "apt remove netplan.io" as a "hope this solves the issue and it's some conflict or issue between netplan.io and ifupdown OR a conflict between systemd and netplan.io and since there's no report of it being down and I just verified networking is fine.... is it something between systemd and netplan?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers