[systemd] dh_installinit does not create /etc/rc*.d/S??foo if there is an /etc/init/foo.conf

[Iain Lane]

I had no networking when I booted up with systemd just now.

laney@raleigh> cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

laney@raleigh> ifconfig
lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:572 errors:0 dropped:0 overruns:0 frame:0
          TX packets:572 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:46364 (46.3 KB) TX bytes:46364 (46.3 KB)

laney@raleigh> more /etc/network/interfaces
# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto br0
iface br0 inet dhcp
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0

laney@raleigh> sudo ifup eth0 br0
ifup: failed to open lockfile /run/network/.ifstate.lock: No such file or directory

Then copying from /etc/init/networking.conf's pre-start

laney@raleigh> sudo mkdir -p /run/network
laney@raleigh> sudo ifup -a
Internet Systems Consortium DHCP Client 4.2.4
Copyright 2004-2012 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/br0/30:85:a9:9d:63:f6
Sending on LPF/br0/30:85:a9:9d:63:f6
Sending on Socket/fallback
DHCPDISCOVER on br0 to 255.255.255.255 port 67 interval 3 (xid=0x1b02ce25)
DHCPDISCOVER on br0 to 255.255.255.255 port 67 interval 7 (xid=0x1b02ce25)
DHCPREQUEST of 192.168.1.136 on br0 to 255.255.255.255 port 67 (xid=0x1b02ce25)
DHCPOFFER of 192.168.1.136 from 192.168.1.1
DHCPACK of 192.168.1.136 from 192.168.1.1
bound to 192.168.1.136 -- renewal in 39350 seconds.
laney@raleigh> cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.1.1

I feel like there's probably an existing systemd-ish way of doing this.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: ifupdown 0.7.47.2ubuntu4
ProcVersionSignature: Ubuntu 3.13.0-24.46-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
CurrentDesktop: Unity
Date: Fri Apr 25 18:01:25 2014
InstallationDate: Installed on 2012-10-07 (564 days ago)
InstallationMedia: Ubuntu 12.10 "Quantal Quetzal" - Beta amd64 (20121007)
SourcePackage: ifupdown
UpgradeStatus: Upgraded to trusty on 2013-05-07 (353 days ago)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin Pitt (pitti)]

Doing "sudo systemctl start networking" after booting works fine here. I suppose the difference is that Debian has /etc/rcS.d/S10networking -> ../init.d/networking but our ifupdown package doesn't call update-rc.d or otherwise removes the symlinks.

[Martin Pitt (pitti)]

In other words, we either need to bring back the symlink or add proper systemd units to ifupdown.

[Martin Pitt (pitti)]

There is no debian/rules diff in Ubuntu. But the autogenerated postinst diff between sid and utopic explains this, so I suppose our dh_installinit works in a way that an upstart job statically masks an init script:

@ -124,9 +143,14 @@
⎵
 # Automatically added by dh_installinit
 if [ -x "/etc/init.d/networking" ]; then
-…………………update-rc.d networking defaults >/dev/null || exit $?
+…………………if [ ! -e "/etc/init/networking.conf" ]; then
+………………………………………update-rc.d networking defaults >/dev/null
+…………………fi
 fi
 # End automatically added section
+# Automatically added by dh_installinit
+update-rc.d -f networking remove >/dev/null || exit $?
+# End automatically added section
 # Automatically added by dh_installdeb
 dpkg-maintscript-helper rm_conffile /etc/default/ifupdown 0.7~+ ifupdown -- "$@"
 # End automatically added section

That's in debhelper's merge log:

   - autoscripts/postinst-init*: Only call update-rc.d when there's no
      upstart job.

[Martin Pitt (pitti)]

Adding a debhelper task, to check if we can drop that "autoscripts/postinst-init*: Only call update-rc.d when there's no upstart job" delta. Whether we can just do that depends on this question: If there is an /etc/rc?.d/S??foo and an /etc/init/foo.conf, does upstart only run the upstart job and the init.d script gets "masked" by that automatically (1)? Or would upstart run both (2)?

If (1), we can drop the dh_installinit delta, get the rc?.d symlinks back, and will have init.d scripts as properly working fallbacks for packages which don't have a systemd unit. This will also unbreak insserv, as it will then be able to properly figure out dependencies (Debian stopped supporting the non-insserv "static priorities" mode, so sooner or later we'll have to do this).

If (2), how difficult is it to implement this masking dynamically in upstart instead of statically in dh_installinit? This would then need an upstart task.

[Steve Langasek (vorlon)]

On Mon, May 12, 2014 at 03:40:42PM -0000, Martin Pitt wrote:
> Adding a debhelper task, to check if we can drop that "autoscripts
> /postinst-init*: Only call update-rc.d when there's no upstart job"
> delta.

No, we cannot. This would cause the init script to be run on boot via
/etc/init/rc.conf, in addition to the upstart job being run, with undefined
behavior.

Before you can drop this debhelper delta, we need insserv enabled in Ubuntu,
which first requires repairing the delta of packages that have had their
init scripts dropped completely in Ubuntu (at this point, possibly mostly
just the initscripts package).

> Whether we can just do that depends on this question: If there is
> an /etc/rc?.d/S??foo and an /etc/init/foo.conf, does upstart only run
> the upstart job and the init.d script gets "masked" by that
> automatically (1)? Or would upstart run both (2)?

2. The masking support is done via startpar + startpar-upstart-bridge,
which is only available once we're insserv enabled.

So the steps are:

 1) fix all per-package deltas in Ubuntu that removed initscripts from the
    package in favor of upstart jobs (i.e.: bring the packages in line with
    current Debian policy wrt upstart jobs)
 2) undo the insserv/sysvinit/startpar changes that have disabled insserv in
    Ubuntu
 3) test insserv extensively in Ubuntu to make sure we're getting the
    expected result
 4) drop the debhelper change

[Martin Pitt (pitti)]

Hey Steve,

thanks for the explanations!

Steve Langasek [2014-05-12 16:09 -0000]:
> 2. The masking support is done via startpar + startpar-upstart-bridge,
> which is only available once we're insserv enabled.

Ah, good to know. So startpar knows about upstart jobs and won't
create static runlevel symlinks? Because /etc/init/rc.conf doesn't
seem to have any particular magic for ignoring rc?.d/ links which have
a corresponding upstart job.

> 1) fix all per-package deltas in Ubuntu that removed initscripts from the
> package in favor of upstart jobs (i.e.: bring the packages in line with
> current Debian policy wrt upstart jobs)

That's https://www.debian.org/doc/debian-policy/ch-opersys.html#s-upstart , right?
I. e. we'll also need to update the init.d scripts in these packages
to add init_is_upstart().

> 2) undo the insserv/sysvinit/startpar changes that have disabled insserv in
> Ubuntu
> 4) drop the debhelper change

Just to ensure that I understood this: For 1) we just put back the
init.d scripts but *without* rc?.d/ links, i. e. we build all those
packages with the current dh_installinit which does the "does this
have an upstart job?" test. Otherwise we'd create a dependency loop
between 1) and 2) as we would then actually run these init.d scripts.
Then we do 2) and for 4) we rebuild all these packages again with the
"new" dh_installinit so that these packages do get rc?.d symlinks.

Thanks,
Martin

[Steve Langasek (vorlon)]

On Tue, May 13, 2014 at 05:28:10AM -0000, Martin Pitt wrote:
> thanks for the explanations!

> Steve Langasek [2014-05-12 16:09 -0000]:
> > 2. The masking support is done via startpar + startpar-upstart-bridge,
> > which is only available once we're insserv enabled.

> Ah, good to know. So startpar knows about upstart jobs and won't
> create static runlevel symlinks?

insserv still creates the links. Startpar, which is the component that runs
as boot, recognizes that the init scripts should be masked by upstart jobs.

> Because /etc/init/rc.conf doesn't seem to have any particular magic for
> ignoring rc?.d/ links which have a corresponding upstart job.

Correct.

> > 1) fix all per-package deltas in Ubuntu that removed initscripts from the
> > package in favor of upstart jobs (i.e.: bring the packages in line with
> > current Debian policy wrt upstart jobs)

> That's https://www.debian.org/doc/debian-policy/ch-opersys.html#s-upstart
> , right? I.e. we'll also need to update the init.d scripts in these
> packages to add init_is_upstart().

Or to ensure the package is including /lib/lsb/init-functions and is used
with upstart >= 1.12.1-0ubuntu6, which includes Dimitri's change to make all
LSB init scripts DTRT automatically.

> > 2) undo the insserv/sysvinit/startpar changes that have disabled insserv in
> > Ubuntu
> > 4) drop the debhelper change

> Just to ensure that I understood this: For 1) we just put back the
> init.d scripts but *without* rc?.d/ links, i. e. we build all those
> packages with the current dh_installinit which does the "does this
> have an upstart job?" test.

Correct.

> Otherwise we'd create a dependency loop between 1) and 2) as we would then
> actually run these init.d scripts. Then we do 2) and for 4) we rebuild
> all these packages again with the "new" dh_installinit so that these
> packages do get rc?.d symlinks.

Yes, exactly.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Dimitri John Ledkov (xnox)]

Here is the list of init.d scripts that don't declare sysv-init compatible dependency based boot orderings:
fcoe
irda-setup
jabber-irc
nordugrid-arc-ldap-infosys
powerfail
refdb
showfsck

i will ignore them for now. If it becomes a problem, either add upstart/systemd units or add proper headers or remove these packages.

[Dimitri John Ledkov (xnox)]

With proposed branch insserv is converted, and i can shutdown/boot with it, however runlevel 2 never finishes and thus tty1 does not come up:
root 1498 0.0 0.0 4444 652 ? Ss 11:14 0:00 /bin/sh -e /proc/self/fd/9
root 1527 0.0 0.0 4444 776 ? S 11:14 0:00 \_ /bin/sh /etc/init.d/rc 2
root 1567 0.0 0.0 13304 3096 ? SL 11:14 0:00 \_ startpar -p 4 -t 20 -T 3 -M start -P N -R 2

xnox@sochi:~$ initctl --system status tty1
tty1 stop/waiting
xnox@sochi:~$ initctl --system status rc
rc start/running, process 1498

[Steve Langasek (vorlon)]

Dimitri, when startpar blocks, what do the contents of /etc/rc2.d look like on your system?

[Dimitri John Ledkov (xnox)]

Rebooted just now, and still same result. rc job is start/running, tty1 job is stop/waiting.
Here is the contents of rc2.d, all symlinks dereferenced & tarred up.

[Dimitri John Ledkov (xnox)]

I think this was self-induced harm cause by /etc/init/syslog-ng.conf (not from distro).
Also i somehow had scripts activated without their dependencies, which should not be possible. (e.g. pppd-dns without gdm).

I'll test this more, on more normal installations. But otherwise I think it should be ready for utopic.

[Steve Langasek (vorlon)]

No obvious issues with any of the scripts in /etc/rc2.d being shadowed by an upstart job that's not being started:

(base)vorlon@virgil:/tmp/1312836/rc2.d$ ls -1 S??* | sed -e's,^S..,/etc/init/,; s/$/\.conf/' | apt-file search -f
(base)vorlon@virgil:/tmp/1312836/rc2.d$

Can you attach /etc/init.d/.depend.start ?

[Dimitri John Ledkov (xnox)]

So, i had syslog-ng upstart job added ( from upstream https://github.com/balabit/syslog-ng-3.5/blob/master/contrib/upstart/syslog-ng.conf.upstart )

Then I had startpar-bridge.override set to manual (who needs that, right?! :)))))) )

and in .depend.start everything depending on syslog-ng. init.d script not started because, upstart job is present. no startpar notification about syslog-ng started, since that is set to manual. End result startpar hanging as expected.

I've re-enabled startpar-bridge upstart job and everything is fine now. rc finishes, tty1 starts, no hanging startpar processes.

[Launchpad Janitor (janitor)]

This bug was fixed in the package sysvinit - 2.88dsf-41ubuntu13

---------------
sysvinit (2.88dsf-41ubuntu13) utopic; urgency=medium

  * Revert previous upload, which caused regression due to UPSTARTDIR
    variable removal that was still in-use.
  * Reintroduce 11 rcS initd scripts:
    - all of them source /lib/lsb/init-functions and thus do nothing under
    upstart, and symlinked to /dev/null under systemd.
    - adjust initscripts conffiles, postinst and postrm to setup symlinks
    and clean up appropriately.
  * Enable insserv (LP: #1312836):
     - adjust path variable in postinst to find insserv under
    /usr/lib/insserv due to 200_hide_insserv_on_ubuntu.patch in insserv on
    ubuntu.
    - adjust legacy_boot check to examine both rc0.d and rc6.d and the
    stamp file.
    - perform conversion and remove flag file
    - cherrypick patch for update-rc.d to lookup insserv in
    /usr/lib/insserv and /sbin.
    - drop touching legacy boot ordering stamp in sysv-rc preinst.
 -- Dimitri John Ledkov <email address hidden> Tue, 27 May 2014 13:52:14 +0200

[Martin Pitt (pitti)]

Package got reverted for now, reopening.

[Launchpad Janitor (janitor)]

This bug was fixed in the package debhelper - 9.20140228ubuntu5

---------------
debhelper (9.20140228ubuntu5) utopic; urgency=medium

  * Enable calling update-rc.d, if an init.d script is installed,
    regardless presence of the upstart jobs. (LP: #1312836)
 -- Dimitri John Ledkov <email address hidden> Wed, 28 May 2014 11:00:44 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package sysvinit - 2.88dsf-41ubuntu14

---------------
sysvinit (2.88dsf-41ubuntu14) utopic; urgency=medium

  * Revert the dependency changes from the previous upload to avoid loops.
 -- Adam Conrad <email address hidden> Tue, 27 May 2014 08:45:53 -0600

[Martin Pitt (pitti)]

https://launchpad.net/ubuntu/+source/ifupdown/0.7.48.1ubuntu4

[Nazar Mokrynskyi (nazar-pc)]

File /sbin/insserv is absent, so I symlinked it to /usr/lib/insserv/insserv
I think this happened after todays update.

[Martin Pitt (pitti)]

Nazar Mokrynskyi [2014-05-28 20:06 -0000]:
> File /sbin/insserv is absent

Where did you see this path being mentioned? Current sysvinit (should)
call it from /usr/lib/insserv/. Did we miss a spot?

Thanks!

[Luis Mondesi (lemsx1)]

Maybe this should be reopened?

root@hp:~# ifup br0
ifup: failed to open lockfile /run/network/.ifstate.lock: No such file or directory
root@hp:~# mkdir -p /run/network
root@hp:~# ifup br0

Waiting for br0 to get ready (MAXWAIT is 32 seconds).
root@hp:~# ifconfig
br0 Link encap:Ethernet HWaddr 88:51:fb:3f:3e:26
...

$> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Vivid Vervet (development branch)
Release: 15.04
Codename: vivid

ii systemd-sysv 218-3ubuntu4 amd64 system and service manager - SysV
un sysvinit <none> <none> (no description available)
ii ifupdown 0.7.48.1ubun amd64 high level tools to configure net

I'm experimenting with systemd-sysv and got this on first boot after upgrading to 15.04.

[Martin Pitt (pitti)]

Luis, that seems to be a different bug/cause now. To confirm, what does that show right after a clean boot:

  sudo systemctl status -l networking

(Please confirm that you still don't have /run/network right after boot).

[Luis Mondesi (lemsx1)]

Martin, I thought this was fixed, but I'm still getting the same error and /run/network still does not exists on reboot:

$> sudo systemctl status -l networking
* networking.service - LSB: Raise network interfaces.
   Loaded: loaded (/etc/init.d/networking)
  Drop-In: /run/systemd/generator/networking.service.d
           `-50-insserv.conf-$network.conf
        /lib/systemd/system/networking.service.d
           `-network-pre.conf
   Active: inactive (dead)
     Docs: man:systemd-sysv-generator(8)

When the system boots, /run/network directory is not present.

$> sudo journalctl |grep run|grep network
Mar 09 10:42:13 my_hostname_here NetworkManager[927]: <info> monitoring ifupdown state file '/run/network/ifstate'.
Mar 09 10:44:16 my_hostname_here NetworkManager[927]: <warn> Error: failed to open /run/network/ifstate
Mar 09 10:44:16 my_hostname_here NetworkManager[927]: <warn> Error: failed to open /run/network/ifstate
Mar 09 10:49:39 my_hostname_here sudo[5273]: lmondesi : TTY=pts/0 ; PWD=/home/lmondesi ; USER=root ; COMMAND=/bin/mkdir /run/network
Mar 09 10:49:40 my_hostname_here NetworkManager[927]: (NetworkManager:927): GLib-GIO-WARNING **: Missing callback called fullpath = /run/network/ifstate
Mar 09 10:56:10 my_hostname_here sudo[5600]: lmondesi : TTY=pts/1 ; PWD=/etc/init.d ; USER=root ; COMMAND=/bin/chown root:netdev /run/network

[Luis Mondesi (lemsx1)]

I think I found the culprit...

my /etc/network/interfaces file contains lines like:

auto em0
iface em0 inet manual

auto eth0
iface eth0 inet manual

They refer to previous names of the same interface as it went from being called "eth0" to now "em0". When attempting to start networking via systemctl start networking, it times out with:

Mar 09 11:23:26 my_hostname_here systemd[1]: Starting LSB: Raise network interfaces....
Mar 09 11:23:26 my_hostname_here networking[7235]: * Configuring network interfaces...
Mar 09 11:23:26 my_hostname_here networking[7235]: Cannot find device "eth0"
Mar 09 11:23:26 my_hostname_here networking[7235]: Failed to bring up eth0.
Mar 09 11:28:26 my_hostname_here systemd[1]: networking.service start operation timed out. Terminating.
Mar 09 11:28:26 my_hostname_here systemd[1]: Failed to start LSB: Raise network interfaces..
Mar 09 11:28:26 my_hostname_here systemd[1]: Unit networking.service entered failed state.
Mar 09 11:28:26 my_hostname_here systemd[1]: networking.service failed.
Mar 09 11:28:26 my_hostname_here networking[7235]: Cannot find device "em0"
Mar 09 11:28:26 my_hostname_here networking[7235]: Failed to bring up em0.

Commenting out said lines fixed the problem. Although I have not restarted yet to actually verify, but systemctl start networking is now happy. I'll restart and report back if this didn't fix the problem.

[Luis Mondesi (lemsx1)]

Ok, no mater what I do, I cannot get this to work. I commented all but 1 interfaces needed for br0. The configuration looks like:

auto em1
iface em1 inet manual

auto br0
iface br0 inet static
 bridge_ports all
...

And I seem to be hitting this nag:

> sudo systemctl status -l ifup-wait-all-auto.service
* ifup-wait-all-auto.service - Wait for all "auto" /etc/network/interfaces to be up for network-online.target
   Loaded: loaded (/lib/systemd/system/ifup-wait-all-auto.service; static; vendor preset: enabled)
   Active: failed (Result: timeout) since Mon 2015-03-09 12:16:42 EDT; 9min ago
     Docs: man:interfaces(5)
           man:ifup(8)
  Process: 950 ExecStart=/bin/sh -ec for i in $(ifquery --list --exclude lo --allow auto); do INTERFACES="$INTERFACES$i "; done; [ -n "$INTERFACES" ] || exit 0; while ! ifquery --state $INTERFACES >/dev/null; do sleep 1; done (code=killed, signal=TERM)
 Main PID: 950 (code=killed, signal=TERM)

Mar 09 12:14:42 my_hostname_here systemd[1]: Starting Wait for all "auto" /etc/network/interfaces to be up for network-online.target...
Mar 09 12:16:42 my_hostname_here systemd[1]: ifup-wait-all-auto.service start operation timed out. Terminating.
Mar 09 12:16:42 my_hostname_here systemd[1]: Failed to start Wait for all "auto" /etc/network/interfaces to be up for network-online.target.
Mar 09 12:16:42 my_hostname_here systemd[1]: Unit ifup-wait-all-auto.service entered failed state.
Mar 09 12:16:42 my_hostname_here systemd[1]: ifup-wait-all-auto.service failed.

$> ifquery --state eth1 em1 br0
Segmentation fault (core dumped)

I'll attach the crash file.