Upstart fails run with --session: child-subreaper-failed event

[David Ireland]

I'm running upstart 1.8 with --session --user options. Using a script that looks like this:

#!/bin/sh
if [ -z "${UPSTART_HOME}" ]; then
    UPSTART_HOME=/opt/upstart
    export UPSTART_HOME
fi

PATH=$HOME/test/bin:${UPSTART_HOME}/sbin:$PATH
XDG_CONFIG_HOME=$HOME/test
export XDG_CONFIG_HOME
XDG_CONFIG_DIRS=
export XDG_CONFIG_DIRS
XDG_CACHE_HOME=$HOME/test/log
export XDG_CACHE_HOME
exec ${UPSTART_HOME}/sbin/init --session --user \
     --inherit-env \
     --startup-event=diaas-start

It finds the config files I want it to find, then says:

init: Unable to register as subreaper: Invalid argument
event_new: Pending child-subreaper-failed event

Then lots more stuff about starting my services, then

Handling child-subreaper-failed event
event_finished: Finished child-subreaper-failed event
init:com.ubuntu.Upstart.c:2651: Assertion failed in control_emit_event_emitted: env != NULL
Aborted (core dumped)

Upstart 1.5 doesn't fail this way, but I need --inherit-env, for what I'm trying to do.

I get exactly the same result with a 3.2 kernel, as with the kernel I'm now running which is 3.4.37 from Quantal.

Libc from Quantal doesn't help either.

[David Ireland (e-david)]

The child-subreaper-failed event may be a red herring: If I remove the code that raises it, init still crashes as described above:

init:com.ubuntu.Upstart.c:2651: Assertion failed in control_emit_event_emitted: env != NULL
Aborted (core dumped)

In fact if I remove all the config files, so that I'm asking Upstart to start up and do nothing, the following is the result:

main: Running with UID 511 as PID 26756 (PPID 8298)
Loading configuration from <home>/test/upstart
Loading configuration from <home>/.init
Loading configuration from /etc/xdg/upstart
Loading configuration from /usr/share/upstart/sessions
control_handle_bus_type: Using session bus
event_new: Pending diaas-start event
Handling diaas-start event
event_finished: Finished diaas-start event
init:com.ubuntu.Upstart.c:2651: Assertion failed in control_emit_event_emitted: env != NULL
Aborted (core dumped)

So this is more like 'upstart doesn't start with --user or --session'

[David Ireland (e-david)]

Turns out that specifying --session on the command line makes init crash. There don't seem to be any other factors involved. --user works fine for me, so it's not high priority for me.

[Stéphane Graber (stgraber)]

So you apparently ran into two problems:
 - You're running a kernel < 3.5 so you're missing the subreaper support in the kernel. That's not a big problem as upstart handles that gracefully but you won't be able to track processes for jobs doing double-fork.
 - You are apparently using an old version of libnih. You really should backport the one we currently have in Ubuntu as it includes dbus fixes which solve the crash you're getting. Note that dropping --session won't make you immune from that specific crash, so you really should get a newer libnih on the system and rebuild upstart with it.

[David Ireland (e-david)]

Thanks for the response. I'm using libnih1 version 1.0.3, which is newer than the README says it requires.

The subreaper support seems to be documented as being since 3.4:

http://man7.org/linux/man-pages/man2/prctl.2.html

But I guess I should trust the man page?

[David Ireland (e-david)]

I'd love to install a newer version of libnih to confirm this solves the problem. As you predicted, I still occasionally get this crash without the --session flag. I am running Ubuntu precise, but I've pulled in libnih from quantal. What version should I be using?

[Dimitri John Ledkov (xnox)]

On 2 April 2013 09:36, David Ireland <email address hidden> wrote:
> I'd love to install a newer version of libnih to confirm this solves the
> problem. As you predicted, I still occasionally get this crash without
> the --session flag. I am running Ubuntu precise, but I've pulled in
> libnih from quantal. What version should I be using?
>

Recompile raring's version.

Regards,

Dmitrijs.

[David Ireland (e-david)]

I've built libnih 1.0.3 from source and also made sure that upstart builds with that version of the nih-dbus-tool. I'm still having this problem.

[James Hunt (jamesodhunt)]

Which problem? The crash? If so, you are still using the wrong version of libnih: you should be using the Ubuntu version (specifically 1.0.3-4ubuntu16) from here: https://code.launchpad.net/~ubuntu-branches/ubuntu/raring/libnih/raring

You do not need the --session flag to run a "Session Init" (yes, this is a little confusing but --session was added for testing a long time ago and is still required for that). A "Session Init" only requires "--user".

[pdf (pdffs)]

If you're still trying to make this work, I backport Upstart to Precise to deal with #1169028, and to solve this (at least, the tests now pass - I haven't tested your exact usage), I backported (from Raring):

json-c_0.10-1.2ubuntu2
dbus_1.6.8-1ubuntu6.1
libnih_1.0.3-4ubuntu16

for:
upstart_1.8-0ubuntu1

And everything was functional at that point. There are definitely some missing dependency versions here, though I haven't checked later versions (pretty sure they include systemd, which is a bridge too far for Precise).