networking.service races with kernel interface detection, ifup@.service is not being triggered

[James Hunt]

/lib/systemd/system/ifup@.service specifies:

  ExecStart=/sbin/ifup --allow=hotplug %I

This really needs to be:

  ExecStart=/sbin/ifup --allow=auto %I

Since otherwise, for systems that don't use network-manager, the service will only bring up interfaces which have specified "allow-hotplug $if" in /etc/network/interfaces*.

ProblemType: Bug
DistroRelease: Ubuntu 14.10
Package: systemd 208-8ubuntu4
ProcVersionSignature: Ubuntu 3.16.0-17.23-generic 3.16.3
Uname: Linux 3.16.0-17-generic x86_64
ApportVersion: 2.14.7-0ubuntu2
Architecture: amd64
CurrentDesktop: Unity
Date: Fri Sep 26 16:54:12 2014
InstallationDate: Installed on 2014-04-11 (167 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Daily amd64 (20140409)
SourcePackage: systemd
UpgradeStatus: Upgraded to utopic on 2014-05-08 (141 days ago)

[Martin Pitt (pitti)]

I updated the package description to what you (most probably) mean. I need to check with the Debian systemd maintainers whether hotplug should additionally be there, too.

[Martin Pitt (pitti)]

I have some trouble reproducing this. I installed systemd-sysv into a minimal VM without NetworkManager, and /etc/network/interfaces.d/eth0.cfg has

  # The primary network interface
  auth eth0
  iface eth0 inet dhcp

Indeed "ifquery -l --allow=hotplug" doesn't mention any interface, but eth0 comes up just fine anyway. When I comment out above file, it does not come up, so that file and ifup is definitively being used here.

But running "sudo ifup --verbose --allow=hotplug eth0" manually confirms that the stanza is being ignored. So the most probable explanation is that /etc/init.d/networking is being run during boot which takes care of the "auto" stanzas? Why isn't that being run for you? Can you please show "systemctl status networking"?

[James Hunt (jamesodhunt)]

$ systemctl status networking

networking.service - LSB: Raise network interfaces.
   Loaded: loaded (/etc/init.d/networking)
  Drop-In: /run/systemd/generator/networking.service.d
           `-50-insserv.conf-$network.conf
   Active: active (exited) since Mon 2014-09-29 09:22:44 UTC; 1min 51s ago
  Process: 332 ExecStart=/etc/init.d/networking start (code=exited, status=0/SUCCESS)

[Martin Pitt (pitti)]

Right, I get the same (on both current utopic and your Ubuntu Core environment). So /etc/init.d/networking exists and is being run -- what's missing? What does "ip a" show? It gets a proper DHCP assignment and routing here.

[Martin Pitt (pitti)]

ifup@.service works as intended. The idea is that /etc/init.d/networking brings up all "auto" interfaces, and that unit takes care of the hotplugged interfaces only.

The problem is that /etc/init.d/networking (or something it calls) doesn't work if / is read-only. I'll investigate what it's trying to do.

[Martin Pitt (pitti)]

Further debugging: /etc/init.d/networking just gets redirected through /lib/lsb/init-functions.d/40-systemd. This gets closer to the problem:

networking.service - LSB: Raise network interfaces.
   Loaded: loaded (/etc/init.d/networking)
  Drop-In: /run/systemd/generator/networking.service.d
           `-50-insserv.conf-$network.conf
   Active: active (exited) since Mon 2014-09-29 13:15:31 UTC; 41s ago
  Process: 405 ExecStart=/etc/init.d/networking start (code=exited, status=0/SUCCESS)

Sep 29 13:15:31 localhost.localdomain dhclient[486]: For info, please visit https://www.isc.org/software/dhcp/
Sep 29 13:15:31 localhost.localdomain networking[405]: For info, please visit https://www.isc.org/software/dhcp/
Sep 29 13:15:31 localhost.localdomain dhclient[486]:
Sep 29 13:15:31 localhost.localdomain networking[405]: Cannot find device "eth0"
Sep 29 13:15:31 localhost.localdomain networking[405]: Error getting hardware address for "eth0": No such device
Sep 29 13:15:31 localhost.localdomain networking[405]: Failed to bring up eth0.
Sep 29 13:15:31 localhost.localdomain networking[405]: ...done.
Sep 29 13:15:31 localhost.localdomain systemd[1]: Started LSB: Raise network interfaces..
Sep 29 13:15:31 localhost.localdomain ntpdate[566]: Can't find host ntp.ubuntu.com: Name or service not known (-2)
Sep 29 13:15:31 localhost.localdomain ntpdate[566]: no servers can be used, exiting

Running "sudo systemctl start networking.service" manually works, so somehow this is running at a time when there's no eth0 yet. This also explains why adding --allow=auto to the job works, as then it gets re-run when eth0 is detected.

[Martin Pitt (pitti)]

So for some reason your system configuration causes /etc/init.d/networking to be run before the kernel detects eth0 (journalctl shows this quite nicely).

[Martin Pitt (pitti)]

I talked to Michael Biebl, some notes:

 * I suppose due to keeping the root fs readonly the entire boot is a tad faster and thus exposes the race.
 * Due to that race it is recommended that physical interfaces use 'allow-hotplug', and virtual interfaces (bridges etc) use 'auto'.
 * debian-installer uses allow-hotplug for writing /etc/network/interfaces, to cope with slow detection on the kernel side

However, it also doesn't work with allow-hotplug, I'll investigate further.

[Martin Pitt (pitti)]

In my test system, ifup@.service actually never gets called, which is why the hotplugging fails. (I. e. changing to --allow=auto doesn't work either). "sudo systemctl start <email address hidden>" works (you then see that it moves to active(exited) and ran the commands), but the BindsTo=sys-subsystem-net-devices-%i.device does not seem to work.

[Martin Pitt (pitti)]

OK, I know what's wrong with ifup@.service. I'll fix it to actually get triggered.

James: To avoid the race between the kernel and /etc/init.d/networking, is it possible to fix your image scripts to mark physical interfaces (eth0) as "allow-hotplug" instead of "auto"? That would mirror what d-i does.

[Martin Pitt (pitti)]

Uploaded fix.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 208-8ubuntu5

---------------
systemd (208-8ubuntu5) utopic; urgency=medium

  * Extend systemd's "net" device udev rule to trigger ifup@.service on
    network devices. In Debian this is done through net.agent, but we do not
    install this in Ubuntu. (LP: #1374521)
  * debian/ifup@.service: Additionally run for "auto" class. We don't really
    support "allow-hotplug" in Ubuntu at the moment, so we need to deal with
    "auto" devices appearing after "/etc/init.d/networking start" already ran.
    (LP: #1374521)
  * Pull latest keymaps from upstream trunk.
 -- Martin Pitt <email address hidden> Tue, 30 Sep 2014 07:55:12 +0200