rc.local runs earlier than cloud-init.service

[Scott Moser]

per https://www.debian.org/doc/manuals/debian-faq/ch-customizing.en.html
 | The rc.local script is executed at the end of each multiuser runlevel.

Generally speaking people expect rc.local to be run as "the last thing in boot".

currently it is being run by systemd much earlier.

Provided in the attached is user-data that tells cloud-init to write a file in /usr/local/bin/rc-local-message and edit set rc.local to execute it.

On current wily, under most scenarios the script will not be executed as rc.local will most likely run before the cloud-init service has a chance to update /etc/rc.local.

To reproduce:
 a.) launch an instance with provided user-data

Expected behavior would be:
 1. /run/rc-local-message exists with contents of /proc/uptime at the time it was run
 2. /rc-local-message.mark exists with the number of times this script has been run ('1' after first boot)
 3. /var/log/rc-local-message.log exists with a message like:
      [/var/log/rc-local-message.log] <date> === successful boot 1 ====
 4. console log should also have message like in 3 with '/dev/console'

4 is broken as rc.local output is not sent to console as reported separately at bug 1468102.

Related bugs:
 * bug 1468102: rc.local output does not go to console

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: systemd 220-7ubuntu1
ProcVersionSignature: User Name 3.19.0-22.22-generic 3.19.8-ckt1
Uname: Linux 3.19.0-22-generic x86_64
ApportVersion: 2.17.3-0ubuntu4
Architecture: amd64
Date: Tue Jun 23 20:21:50 2015
Ec2AMI: ami-00000434
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-22-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
SourcePackage: systemd
UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/01/2011
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnOpenStackFoundation:pnOpenStackNova:pvr2014.1.4:cvnBochs:ct1:cvr:
dmi.product.name: OpenStack Nova
dmi.product.version: 2014.1.4
dmi.sys.vendor: OpenStack Foundation

[Martin Pitt (pitti)]

> Generally speaking people expect rc.local to be run as "the last thing in boot".

This can't be taken literally, as services can and do come and go (stop and start) all the time, and as any service can declare dependencies to run after "your thing" this can't be enforced either.

For example, could-final.service has an "After=rc-local.service" which I assume is deliberate. So we can't do a thing like "run rc-local.service after multi-user.target" as that would be an unresolvable ordering with cloud-final.service.

Thus I suggest to add an After=rc-local.service to cloud-init.service. This is explicit and unambiguous.

[Scott Moser (smoser)]

Martin,

I dont think your suggestion is correct solution. Independent of cloud-init, things in rc.local should be run as "the last thing in boot". rc.local came from a simpler time. sysvinit ran rc.local as the last thing, and that was that last thing that would run.

Many people will do:
 a.) install Ubuntu
 b.) add some script to /usr/local/bin
 c.) add /usr/local/bin/myscript to /etc/rc.local
and expect that their script will run "at the end of boot"

It may be simplistic, but it is extremely useful and many users expect that functionality.

Breaking this expectation and functionality is not helpful. upstart successfully maintained a useful resemblance to the original rc.local functionality. I personally would expect systemd to also.

[Martin Pitt (pitti)]

> Independent of cloud-init, things in rc.local should be run as "the last thing in boot".

So now we have "last thing except cloud-init" :-) There are other services which order themselves after it, like getty@, plymouth-quit and others, to avoid disturbing init.d/rc.local scripts with allocated VTs or plymouth going away. This was more or less the same case with upstart.

But nevertheless it should run at least after the init.d scripts (and their unit counterparts), to maintain backwards compatibility with that. This still isn't robust (if you need to run after something it's always correct to actually declare it), but might help in some situations.

[Tiago Peralta (tperalta82)]

A simple fix is something like this:

Create the /etc/systemd/system/rc-local.service file with the following contents:

[Unit]
Description=/etc/rc.local Compatibility

[Service]
Type=idle
ExecStart=/etc/rc.systemd
TimeoutSec=0
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Create a file /etc/rc.systemd with whatever you want to run, set it to executable.
Delete everything you added from rc.local until this gets properly fixed

run systemctl daemon-reload

Reboot to test it.

Working for me at the moment

[Martin Pitt (pitti)]

Tiago, your service file has even weaker dependencies like the standard rc-local.service, so it'll run even earlier. It's not good advice to create blanket "wrappers" around shell scripts without specifying any dependency.

[Martin Pitt (pitti)]

As I explained, the notion of "the last thing during boot" is not well-defined with a partially ordered and parallel init system like systemd or upstart (or even sysvinit+startpar).

So as long as cloud-init.service declares itself to run after rc-local.service, then rc-local.service necessarily can't be "the" last thing even accidentally. This is something to decide in cloud-init.service.

[Martin Pitt (pitti)]

> So as long as cloud-init.service declares itself to run after rc-local.service

Sorry, I meant cloud-final.service.

[sles (slesru)]

>the notion of "the last thing during boot" is not well-defined with a partially ordered and parallel init system like systemd or upstart (or even sysvinit+startpar).

systemd needs to be fixed then.
because rc.local worked well with upstart.

[Martin Pitt (pitti)]

Please see comment #7 -- cloud-final.service declares to run *after* rc-local.service. Thus it is not possible that rc-local.service "runs last" as that would be a contradiction to what cloud-final wants and says. There is nothing to "fix" and this was no different in upstart either -- if something said "start on started rc-local" (the upstart equivalent) it would of course run after /etc/init/rc-local as well.

Please give up on "the last thing" -- this concept hasn't existed in upstart and systemd, and hasn't existed any more in SysV init with startpar either for many years now.

[James Falcon (falcojr)]

I'm closing this as "Won't Fix" for cloud-init now as this behavior has been established and now expected.