networking.service races with kernel interface detection, ifup@.service is not being triggered

Bug #1374521 reported by James Hunt on 2014-09-26
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Undecided
Martin Pitt

Bug Description

/lib/systemd/system/ifup@.service specifies:

  ExecStart=/sbin/ifup --allow=hotplug %I

This really needs to be:

  ExecStart=/sbin/ifup --allow=auto %I

Since otherwise, for systems that don't use network-manager, the service will only bring up interfaces which have specified "allow-hotplug $if" in /etc/network/interfaces*.

ProblemType: Bug
DistroRelease: Ubuntu 14.10
Package: systemd 208-8ubuntu4
ProcVersionSignature: Ubuntu 3.16.0-17.23-generic 3.16.3
Uname: Linux 3.16.0-17-generic x86_64
ApportVersion: 2.14.7-0ubuntu2
Architecture: amd64
CurrentDesktop: Unity
Date: Fri Sep 26 16:54:12 2014
InstallationDate: Installed on 2014-04-11 (167 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Daily amd64 (20140409)
SourcePackage: systemd
UpgradeStatus: Upgraded to utopic on 2014-05-08 (141 days ago)

James Hunt (jamesodhunt) wrote :
Changed in systemd (Ubuntu):
assignee: nobody → Martin Pitt (pitti)
Martin Pitt (pitti) wrote :

I updated the package description to what you (most probably) mean. I need to check with the Debian systemd maintainers whether hotplug should additionally be there, too.

description: updated
Martin Pitt (pitti) wrote :

I have some trouble reproducing this. I installed systemd-sysv into a minimal VM without NetworkManager, and /etc/network/interfaces.d/eth0.cfg has

  # The primary network interface
  auth eth0
  iface eth0 inet dhcp

Indeed "ifquery -l --allow=hotplug" doesn't mention any interface, but eth0 comes up just fine anyway. When I comment out above file, it does not come up, so that file and ifup is definitively being used here.

But running "sudo ifup --verbose --allow=hotplug eth0" manually confirms that the stanza is being ignored. So the most probable explanation is that /etc/init.d/networking is being run during boot which takes care of the "auto" stanzas? Why isn't that being run for you? Can you please show "systemctl status networking"?

Changed in systemd (Ubuntu):
status: New → Incomplete
James Hunt (jamesodhunt) wrote :

$ systemctl status networking

networking.service - LSB: Raise network interfaces.
   Loaded: loaded (/etc/init.d/networking)
  Drop-In: /run/systemd/generator/networking.service.d
           `-50-insserv.conf-$network.conf
   Active: active (exited) since Mon 2014-09-29 09:22:44 UTC; 1min 51s ago
  Process: 332 ExecStart=/etc/init.d/networking start (code=exited, status=0/SUCCESS)

Martin Pitt (pitti) wrote :

Right, I get the same (on both current utopic and your Ubuntu Core environment). So /etc/init.d/networking exists and is being run -- what's missing? What does "ip a" show? It gets a proper DHCP assignment and routing here.

ifup@.service works as intended. The idea is that /etc/init.d/networking brings up all "auto" interfaces, and that unit takes care of the hotplugged interfaces only.

The problem is that /etc/init.d/networking (or something it calls) doesn't work if / is read-only. I'll investigate what it's trying to do.

summary: - ifup should specify '-allow=auto' not '-allow=hotplug'
+ auto interfaces do not come up with r/o images
affects: systemd (Ubuntu) → ifupdown (Ubuntu)
Changed in ifupdown (Ubuntu):
status: Incomplete → New
Martin Pitt (pitti) wrote :

Further debugging: /etc/init.d/networking just gets redirected through /lib/lsb/init-functions.d/40-systemd. This gets closer to the problem:

networking.service - LSB: Raise network interfaces.
   Loaded: loaded (/etc/init.d/networking)
  Drop-In: /run/systemd/generator/networking.service.d
           `-50-insserv.conf-$network.conf
   Active: active (exited) since Mon 2014-09-29 13:15:31 UTC; 41s ago
  Process: 405 ExecStart=/etc/init.d/networking start (code=exited, status=0/SUCCESS)

Sep 29 13:15:31 localhost.localdomain dhclient[486]: For info, please visit https://www.isc.org/software/dhcp/
Sep 29 13:15:31 localhost.localdomain networking[405]: For info, please visit https://www.isc.org/software/dhcp/
Sep 29 13:15:31 localhost.localdomain dhclient[486]:
Sep 29 13:15:31 localhost.localdomain networking[405]: Cannot find device "eth0"
Sep 29 13:15:31 localhost.localdomain networking[405]: Error getting hardware address for "eth0": No such device
Sep 29 13:15:31 localhost.localdomain networking[405]: Failed to bring up eth0.
Sep 29 13:15:31 localhost.localdomain networking[405]: ...done.
Sep 29 13:15:31 localhost.localdomain systemd[1]: Started LSB: Raise network interfaces..
Sep 29 13:15:31 localhost.localdomain ntpdate[566]: Can't find host ntp.ubuntu.com: Name or service not known (-2)
Sep 29 13:15:31 localhost.localdomain ntpdate[566]: no servers can be used, exiting

Running "sudo systemctl start networking.service" manually works, so somehow this is running at a time when there's no eth0 yet. This also explains why adding --allow=auto to the job works, as then it gets re-run when eth0 is detected.

Martin Pitt (pitti) wrote :

So for some reason your system configuration causes /etc/init.d/networking to be run before the kernel detects eth0 (journalctl shows this quite nicely).

affects: ifupdown (Ubuntu) → systemd (Ubuntu)
summary: - auto interfaces do not come up with r/o images
+ networking.service races with kernel interface detection

I talked to Michael Biebl, some notes:

 * I suppose due to keeping the root fs readonly the entire boot is a tad faster and thus exposes the race.
 * Due to that race it is recommended that physical interfaces use 'allow-hotplug', and virtual interfaces (bridges etc) use 'auto'.
 * debian-installer uses allow-hotplug for writing /etc/network/interfaces, to cope with slow detection on the kernel side

However, it also doesn't work with allow-hotplug, I'll investigate further.

Martin Pitt (pitti) wrote :

In my test system, ifup@.service actually never gets called, which is why the hotplugging fails. (I. e. changing to --allow=auto doesn't work either). "sudo systemctl start <email address hidden>" works (you then see that it moves to active(exited) and ran the commands), but the BindsTo=sys-subsystem-net-devices-%i.device does not seem to work.

Martin Pitt (pitti) wrote :

OK, I know what's wrong with ifup@.service. I'll fix it to actually get triggered.

James: To avoid the race between the kernel and /etc/init.d/networking, is it possible to fix your image scripts to mark physical interfaces (eth0) as "allow-hotplug" instead of "auto"? That would mirror what d-i does.

summary: - networking.service races with kernel interface detection
+ networking.service races with kernel interface detection, ifup@.service
+ is not being triggered
Changed in systemd (Ubuntu):
status: New → In Progress
Martin Pitt (pitti) wrote :

Uploaded fix.

Changed in systemd (Ubuntu):
status: In Progress → Fix Committed
Martin Pitt (pitti) on 2014-09-30
tags: added: systemd-boot
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 208-8ubuntu5

---------------
systemd (208-8ubuntu5) utopic; urgency=medium

  * Extend systemd's "net" device udev rule to trigger ifup@.service on
    network devices. In Debian this is done through net.agent, but we do not
    install this in Ubuntu. (LP: #1374521)
  * debian/ifup@.service: Additionally run for "auto" class. We don't really
    support "allow-hotplug" in Ubuntu at the moment, so we need to deal with
    "auto" devices appearing after "/etc/init.d/networking start" already ran.
    (LP: #1374521)
  * Pull latest keymaps from upstream trunk.
 -- Martin Pitt <email address hidden> Tue, 30 Sep 2014 07:55:12 +0200

Changed in systemd (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers