[ubuntu-touch] upstart should report applications that hit respawn limit to errors.ubuntu.com

[Antti Kaijanmäki]

We have important system and session services that we rely on being running and functional.

If such a component hits the upstart respawn limit for any reason, we should be able to get a report that something is seriously wrong.

Inspiration for this bug as per the discussion in: https://bugs.launchpad.net/ubuntu/+source/indicator-network/+bug/1371320

[James Hunt (jamesodhunt)]

The most appropriate way to do this would probably be to use apport to check for respawn limit messages in:

- the system log, and
- the users $HOME/.xsession-errors file.

Even at the default upstart log priority of 'message', when a job reaches the respawn limit, upstart will log a message to one of the logs above like this:

    init: foo respawning too fast, stopped

So apport, or some such utility could watch for these messages using inotify and raise an informational error that would be visible on errors.u.c.

[James Hunt (jamesodhunt)]

Note that the kerneloops packages communicates with apport using the /usr/share/apport/kernel_oops "pipe" program so a script could be written to make use of similar pipe specifically for respawn failures. Ideally, that kernel_oops utility would be made generic, accepting args to specify the name of the package to report the problem against, etc, and the kerneloops package updated to handle the new interface.

[James Hunt (jamesodhunt)]

brian has pointed out that changing /usr/share/apport/kernel_oops isn't necessary; apport now has a generic pipe program in the form of /data/bzr/apport/apport/data/recoverable_problem.

We will need to make minor changes to the recoverable_problem utility since it currently requires a pid but in this scenario we don't have one - the process that upstart has been respawning is no longer running (and we don't want apport using the pid of upstart).

[James Hunt (jamesodhunt)]

I've added more info to the cookbook wrt respawn. Here's the simplest method of detecting hitting the respawn limit:

http://upstart.ubuntu.com/cookbook/#detecting-a-job-hitting-its-respawn-limit

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[James Hunt (jamesodhunt)]

Something like the attached can be put into /usr/share/upstart/sessions/ and it will detect when system-level and session-level jobs hit their respawn limit.

We could then use whitelists to identify the desired behaviour (restart job, reboot, etc) for particular services.

[Marcus Tomlinson (marcustomlinson)]

This release of Ubuntu is no longer receiving maintenance updates. If this is still an issue on a maintained version of Ubuntu please let us know.

[Marcus Tomlinson (marcustomlinson)]

This issue has sat incomplete for more than 60 days now. I'm going to close it as invalid. Please feel free re-open if this is still an issue for you. Thank you.