systemd breaks due to old libsecomp libs left on the system

[Greg]

Upgraded Ubuntu 18.04 to 20.04. Following the upgrade, booting was not possible. The error messages is:
/sbin/init: symbol lookup error: /lib/systemd/libsystemd-shared-245.so: undefined symbol: seccomp_api_get
[ 4.608900] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
See also attached photograph of screen during boot.

Upgrade followed steps from here: https://help.ubuntu.com/community/FocalUpgrades/Kubuntu
With the excpetion that The -d flag was used for the do-release-upgrade:
sudo do-release-upgrade -d -m desktop

1) The release of Ubuntu you are using, via 'lsb_release -rd' or System -> About Ubuntu
Prior to upgrade: Ubuntu 18.04.4
After upgrade (but never booted): Ubuntu (Kubuntu) 20.04
Note that Ubuntu had originally be installed, but kubuntu-desktop was recently installed to change to Kubuntu, but no booting problems were experienced before updating to 20.04.

2) The version of the package you are using, via 'apt-cache policy pkgname' or by checking in
Unknown -- Package version may have changed when upgrading to 20.04.

3) What you expected to happen
Boot without kernel panic.

4) What happened instead
Could not boot. Even selecting safe mode from grub could not boot. Had to restore system from backups. Will not attempt upgrade again.

[warrenc5 (warren-crossing-mofokom)]

This is something to do with linux 5 kernel api

You need to manually downgrade systemd - well - that's what I did.

when you boot with grub or refind - edit default entry and change the init=/bin/init to init=/bin/bash

That will bypass systemd starting up and drop you straight into a shell.

mount -o remount /
mount /usr
mount /var
mount /dev/sdc1 /mnt
dpkg -i /mnt/systemd_241-7~deb10u3_amd64.deb

On a system that works or download a previous version somehow.

dpkg -l systemd
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii systemd 245.5-2 amd64 system and service manager
root@leet:/home/wozza# apt-cache policy systemd
systemd:
  Installed: 245.5-2
  Candidate: 245.5-2
  Version table:
 *** 245.5-2 500
        500 http://ftp.iinet.net.au/debian/debian bullseye/main amd64 Packages
        100 /var/lib/dpkg/status
     241-7~deb10u3 500
        500 http://ftp.iinet.net.au/debian/debian stable/main amd64 Packages
root@leet:/home/wozza# apt-get download systemd=241-7~deb10u3
Get:1 http://ftp.iinet.net.au/debian/debian stable/main amd64 systemd amd64 241-7~deb10u3 [3,495 kB]
Fetched 3,495 kB in 11s (321 kB/s)
W: Download is performed unsandboxed as root as file '/home/wozza/systemd_241-7~deb10u3_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
root@leet:/home/wozza# ll systemd_241-7~deb10u3_amd64.deb
-rw-r--r-- 1 root root 3495428 Jan 31 15:11 systemd_241-7~deb10u3_amd64.deb
root@leet:/home/wozza# mount /dev/sdc1 /mnt
root@leet:/home/wozza# cp systemd_241-7~deb10u3_amd64.deb /mnt/
root@leet:/home/wozza# cd /mnt/
root@leet:/mnt# mkdir unpack
root@leet:/mnt# cd unpack/
root@leet:/mnt/unpack# dpkg-deb -R ../systemd_241-7~deb10u3_amd64.deb .

worst comes to worse can copy the files across from unpack directory - good luck.

[warrenc5 (warren-crossing-mofokom)]

After trying to roll forward again I saw the output from ldconfig

/lib/x86_64-linux-gnu/libseccomp.so.2.2.0 is not a symbolic link

Looks like I had duplicate versions of libseccomp2 lying around on my LD_LIBRARY_PATH

dpkg -L libseccomp2

/usr/lib/x86_64-linux-gnu/libseccomp.so.2.4.3

So I deleted the ones in /lib/ and then upgraded systemd to the latest version.

[Christian Ehrhardt  (paelzer)]

Hi,
even Bionic which you upgraded from would have been
  /lib/x86_64-linux-gnu/libseccomp.so.2.4.1

2.2 was last available in
 libseccomp2 | 2.2.3-2ubuntu1~ubuntu14.04.1 | trusty-backports | amd64, arm64, armhf, i386, powerpc, ppc64el
 libseccomp2 | 2.2.3-3ubuntu3 | xenial | amd64, arm64, armhf, i386, powerpc, ppc64el, s390x

And it would have been removed in Bionic a long time ago.
Unless this was a file not out of the Ubuntu Archive in which case I'm not sure what we could do to help.

I'll mark it incomplete and I'm glad that you were able to resolve it for yourself.
Is there anything out of this you'd think that Ubuntu as an OS could do better - I see no opportunity as it seems not to be caused by the packages provided.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Rafael David Tinoco (rafaeldtinoco)]

Moving back to incomplete as stated by @paelzer.

[Joi Owen (jlellis)]

I've just encountered this issue myself, on a VM that started life as Ubuntu 16.04, do-release-upgrade to 18.04, then to 20.04, and now it fails to boot with a similar error: /lib/systemd/libsystemd-shared-245.so: undefined symbol: seccomp_api_get

The only non-Ubuntu-standard stuff on this VM is some perl modules installed via CPAN, and none of those touch anything to do with Systemd, so I'm not sure why this is happening. I did see a post-install upgrade issue with the rkhunter package failing to find /usr/bin/egrep, but I ignored that issue and continued.

I'm going to rollback, look for the lib involved, and see if I can remove it before retrying the do-release-upgrade -d to 20.04.

[Christian Ehrhardt  (paelzer)]

Hi Joi,
thanks for the extra data, I still can't think of a case to create this.
Looking forward to what you find checking this in more detail.

When in the error case could you look around which seccomp the system still has around?
For example via:
$ ls -laF /lib/x86_64-linux-gnu/libseccomp* /usr/lib/x86_64-linux-gnu/libseccomp*
$ dpkg -S /lib/x86_64-linux-gnu/libseccomp* /usr/lib/x86_64-linux-gnu/libseccomp*
$ apt-cache policy '*libseccomp*'

[Raphaël Jakse (raphael-jakse)]

I'm in the same case as Joi: I went from Ubuntu 16.04 to Ubuntu 18.04 to Ubuntu 20.04. Removing libsecomp libraries in /lib and the old one in /usr/lib (version 2.3) allowed the system to boot.

[Raphaël Jakse (raphael-jakse)]

These files didn't belong to any package (anymore?). Here are the files I removed:
- /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
- /lib/x86_64-linux-gnu/libseccomp.so.2 (link to the previous file)
- /usr/lib/x86_64-linux-gnu/libseccomp.so.2.3.3

[Christian Ehrhardt  (paelzer)]

Hmm thanks Raphaël.
So files were still around that were unowned, that really should not happen.
Thanks for the exact file names.

The files you had in your case were part of libseccomp2 2.3.1-2.1ubuntu4 which is what Ubuntu had in Bionic-release.
 libseccomp2 | 2.3.1-2.1ubuntu4 | bionic |

But before you'd be able to upgrade to Focal (do-release-upgrade will otherwise block) that would even in Bionic be upgraded to 2.4.3-1ubuntu3.18.04.3
 libseccomp2 | 2.4.3-1ubuntu3.18.04.3 | bionic-security |
 libseccomp2 | 2.4.3-1ubuntu3.18.04.3 | bionic-updates |

On that upgrade these files would vanish.

I've taken a Bionic system, downgraded to 2.3.1-2.1ubuntu4 and from there upgraded the package.
The files went away as one would expect:

root@b:~# ll /lib/x86_64-linux-gnu/libseccomp.so*
lrwxrwxrwx 1 root root 19 Mar 1 2018 /lib/x86_64-linux-gnu/libseccomp.so.2 -> libseccomp.so.2.3.1
-rw-r--r-- 1 root root 280776 Mar 1 2018 /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
root@b:~# apt install libseccomp2
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  libseccomp2
1 upgraded, 0 newly installed, 0 to remove and 28 not upgraded.
Need to get 42.0 kB of archives.
After this operation, 29.7 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libseccomp2 amd64 2.4.3-1ubuntu3.18.04.3 [42.0 kB]
Fetched 42.0 kB in 0s (312 kB/s)
(Reading database ... 43328 files and directories currently installed.)
Preparing to unpack .../libseccomp2_2.4.3-1ubuntu3.18.04.3_amd64.deb ...
Unpacking libseccomp2:amd64 (2.4.3-1ubuntu3.18.04.3) over (2.3.1-2.1ubuntu4) ...
Setting up libseccomp2:amd64 (2.4.3-1ubuntu3.18.04.3) ...
Processing triggers for libc-bin (2.27-3ubuntu1.2) ...
root@b:~# ll /lib/x86_64-linux-gnu/libseccomp.so*
lrwxrwxrwx 1 root root 19 Jun 29 12:52 /lib/x86_64-linux-gnu/libseccomp.so.2 -> libseccomp.so.2.4.3
-rw-r--r-- 1 root root 309456 Jun 29 12:52 /lib/x86_64-linux-gnu/libseccomp.so.2.4.3

Do you have any chance (backup, snapshot or anything) to check which package (if any) owned these files on your Bionic system before upgrade?

Furthermore from your system it might be interesting to get /var/log/dpkg.log* attached here.
Not sure one can find something in there, but we could track exactly what happened to your libseccomp2 in the past - maybe that reconstruction helps to find how/why these files got left on your disk.

[Launchpad Janitor (janitor)]

[Expired for libseccomp (Ubuntu) because there has been no activity for 60 days.]

[Jeremy Akers (irwinr12)]

I just ran into this same issue. I have a server that started out on 16.04. I just did "do-release-upgrade -m server" to upgrade to 18.04 and then again to 20.04. After the 20.04 upgrade finished the server would no longer boot with the exact same error described here:

/sbin/init: symbol lookup error: /lib/systemd/libsystemd-shared-245.so: undefined symbol: seccomp_api_get

I do not understand why bugs like this cannot get fixed even years after several people have reported the same issue and the repro steps are clear: Start with 16.04 upgrade to 18.04 and then 20.04. Clearly the upgrade tool is not properly cleaning up these files.

[Christian Ehrhardt  (paelzer)]

Hi Jeremy

> I do not understand why bugs like this cannot get fixed even years after
> several people have reported the same issue and the repro steps are clear

I understand this might seem frustrating, but the TL;DR is: Because it isn't as clear as it might seem

Detail:

As you see throughout the discussions many have tried to recreate it with those steps but it was not triggering for further debugging.

Just to be sure I did try to recreate again in a new clean system (this time direct upgrades, no do-release-upgrade) upgrading X-B-F => no issues. I also rechecked the libseccomp.so files - always had only those belonging to the current installed version.

As you can see the open question is either:
a) find the details to the steps to really recreate this
or
b) finding out where the older files came from as they have in none of the case been part of the system that was upgraded from but from somewhere further in the past.