Starting systemd Network Service fails with "Invalid argument"

[David Cunningham]

This server has a new install of Ubuntu 18.04 server. Sometimes (around 50% of the time) when it boots systemd-networkd.service initially fails to start with an error as below. It then tries again and succeeds, however other services which depend on systemd-networkd have already failed on the dependency by then.

From the output of "journalctl -u systemd-networkd":

-- Reboot --
May 17 19:44:28 server1 systemd[1]: Starting Network Service...
May 17 19:44:30 server1 systemd[2261]: systemd-networkd.service: Failed to set up mount namespacing: Invalid argument
May 17 19:44:30 server1 systemd[2261]: systemd-networkd.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-networkd: Invalid argument
May 17 19:44:30 server1 systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=226/NAMESPACE
May 17 19:44:30 server1 systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
May 17 19:44:30 server1 systemd[1]: Failed to start Network Service.
May 17 19:44:30 server1 systemd[1]: systemd-networkd.service: Service has no hold-off time, scheduling restart.
May 17 19:44:30 server1 systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 1.
May 17 19:44:30 server1 systemd[1]: Stopped Network Service.
May 17 19:44:30 server1 systemd[1]: Starting Network Service...
May 17 19:44:30 server1 systemd-networkd[2321]: Enumeration completed
May 17 19:44:30 server1 systemd[1]: Started Network Service.
May 17 19:44:30 server1 systemd-networkd[2321]: eno2: IPv6 successfully enabled
May 17 19:44:30 server1 systemd-networkd[2321]: eno1: Link is not managed by us
May 17 19:44:30 server1 systemd-networkd[2321]: enp94s0f0np0: Link is not managed by us
May 17 19:44:30 server1 systemd-networkd[2321]: lo: Link is not managed by us
May 17 19:44:30 server1 systemd-networkd[2321]: enp94s0f1np1: Link is not managed by us
May 17 19:44:30 server1 systemd-networkd[2321]: eno1: IPv6 successfully enabled
May 17 19:44:30 server1 systemd-networkd[2321]: enp94s0f0np0: Link is not managed by us
May 17 19:44:30 server1 systemd-networkd[2321]: lo: Link is not managed by us
May 17 19:44:30 server1 systemd-networkd[2321]: enp94s0f1np1: Link is not managed by us
May 17 19:44:31 server1 systemd-networkd[2321]: eno2: Link UP
May 17 19:44:31 server1 systemd-networkd[2321]: eno1: Link UP
May 17 19:44:34 server1 systemd-networkd[2321]: eno2: Gained carrier
May 17 19:44:34 server1 systemd-networkd[2321]: eno1: Gained carrier
May 17 19:44:35 server1 systemd-networkd[2321]: eno1: Gained IPv6LL
May 17 19:44:35 server1 systemd-networkd[2321]: eno1: Configured
May 17 19:44:36 server1 systemd-networkd[2321]: eno2: Gained IPv6LL
May 17 19:44:36 server1 systemd-networkd[2321]: eno2: Configured

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: systemd 237-3ubuntu10.46
ProcVersionSignature: Ubuntu 4.15.0-142.146-generic 4.15.18
Uname: Linux 4.15.0-142-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.23
Architecture: amd64
Date: Mon May 17 19:53:11 2021
InstallationDate: Installed on 2021-03-18 (60 days ago)
InstallationMedia: Ubuntu-Server 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 004: ID 1604:10c0 Tascam
 Bus 001 Device 003: ID 1604:10c0 Tascam
 Bus 001 Device 002: ID 1604:10c0 Tascam
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Dell Inc. PowerEdge R440
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-142-generic root=/dev/mapper/vg1-lv_root ro processor.max_cstate=1 intel_idle.max_cstate=0 transparent_hugepage=never
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 09/23/2020
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 2.9.3
dmi.board.name: 04JN2K
dmi.board.vendor: Dell Inc.
dmi.board.version: A06
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr2.9.3:bd09/23/2020:svnDellInc.:pnPowerEdgeR440:pvr:rvnDellInc.:rn04JN2K:rvrA06:cvnDellInc.:ct23:cvr:
dmi.product.family: PowerEdge
dmi.product.name: PowerEdge R440
dmi.sys.vendor: Dell Inc.

[Dan Streetman (ddstreet)]

can you turn on systemd debug (e.g. with systemd.log_level=debug boot parameter) and reproduce the error, then check the logs for additional messages indicating what the failure is

[David Cunningham (davidvoisonics)]

Hi Dan, thanks for that. Interestingly it was much harder to reproduce the problem with debugging enabled, but we got it on about the 15th try. The log is attached.

[David Cunningham (davidvoisonics)]

Would there be any update on this please?

[Dan Streetman (ddstreet)]

it looks like you're seeing this problem inside an lxc/lxd container, not on bare metal, is that correct?

[Dan Streetman (ddstreet)]

ok it looks like this *might* be upstream bug 16156, which unfortunately was just very recently fixed, so backporting that to bionic may be difficult.

I'll mark the bug and take a look but even if it is possible to backport the fix, it will likely take a while.

[Dan Streetman (ddstreet)]

marking wontfix for groovy, as a backport of this size is unlikely to happen before groovy reaches end of life this month.

[David Cunningham (davidvoisonics)]

Hi Dan,

Thank you for that. The server is bare metal, not in a container.

If I understand correctly the fix will go in the latest Bionic packages?

[Dan Streetman (ddstreet)]

> If I understand correctly the fix will go in the latest Bionic packages?

I think it's very unlikely that the very recent, and rather large, patchset will get backported back to the relatively old and much different codebase in bionic. I haven't looked at it in detail yet though.

[Brian Murray (brian-murray)]

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.

[Dan Streetman (ddstreet)]

per comment 6, this should be fixed in jammy, so marking fixed released there.

[Brian Murray (brian-murray)]

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.

[David Cunningham (davidvoisonics)]

What version is this fix in please? Just asking as i was reported back in May 2021. Thanks.

[Nick Rosbrook (enr0n)]

This issue is fixed in Jammy, but not earlier LTS releases.