wait-for-state restarts whoopsie every 30sec

[Michael Blennerhassett]

wait-for-state is restarting whoopsie every 30sec:

[14103.317598] init: whoopsie main process ended, respawning
[14133.330315] init: wait-for-state (whoopsienetwork-manager) main process (10221) terminated with status 100
[14133.396185] init: whoopsie main process (10227) terminated with status 1
[14133.396205] init: whoopsie main process ended, respawning
[14163.409765] init: wait-for-state (whoopsienetwork-manager) main process (10231) terminated with status 100
[14163.468346] init: whoopsie main process (10237) terminated with status 1
[14163.468366] init: whoopsie main process ended, respawning
[14193.485204] init: wait-for-state (whoopsienetwork-manager) main process (10241) terminated with status 100
[14193.560511] init: whoopsie main process (10249) terminated with status 1
[14193.560531] init: whoopsie main process ended, respawning

The problem in wait-for-state.conf is this line:
status $WAIT_FOR | grep -q "$TARGET_GOAL/$WAIT_STATE" && exit 0

TARGET_GOAL is either "start" or "stop".
WAIT_STATE by default is "started", but there is no job status like "start/started".

so WAIT_STATE needs to be something like "running" (or "waiting" for goal of "stop").

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: upstart 1.13.2-0ubuntu6
ProcVersionSignature: Ubuntu 3.18.0-9.10-generic 3.18.2
Uname: Linux 3.18.0-9-generic x86_64
ApportVersion: 2.15.1-0ubuntu2
Architecture: amd64
CurrentDesktop: Unity
Date: Tue Jan 20 20:59:02 2015
InstallationDate: Installed on 2014-09-03 (139 days ago)
InstallationMedia: Ubuntu 14.10 "Utopic Unicorn" - Alpha amd64 (20140902)
ProcKernelCmdline: BOOT_IMAGE=/boot/vmlinuz-3.18.0-9-generic root=UUID=93e046a8-7604-4f60-a5f8-b3be22f948aa ro quiet splash vt.handoff=7
SourcePackage: upstart
UpgradeStatus: No upgrade log present (probably fresh install)
UpstartBugCategory: System
UpstartRunningSessionVersion: upstart 1.13.2
UpstartRunningSystemVersion: init (upstart 1.13.2)
modified.conffile..etc.init.wait.for.state.conf: [modified]
mtime.conffile..etc.init.wait.for.state.conf: 2015-01-20T20:02:17.598668

[Michael Blennerhassett (mjblenner)]

This patch changes the defaults for wait-for-state.conf to something sensible for WAIT_STATE.

[Steve Langasek (vorlon)]

The second part of the patch proposed to upstart's wait-for-state job is reasonable, but has nothing to do with the behavior shown here. And the first part of the patch is wrong. It's not appropriate to change the default wait-for-state behavior that other packages are relying on.

/etc/init/whoopsie.conf has:

    status network-manager >/dev/null 2>&1 && start wait-for-state WAITER=whoopsie WAIT_FOR=network-manager GOAL=start || :

- 'GOAL' is incorrect; the variable name is 'TARGET_GOAL', but start is the default so this is a cosmetic issue.
- because network-manager runs a service, WAIT_STATE=running needs to be specified here.
- changing any of this will not cause whoopsie to stop respawning. All of the wait-for-state handling happens in a *pre-start* script which has been written to ignore any failure from wait-for-state. The logs in this bug clearly show that there is a main process which is exiting.

[14103.317598] init: whoopsie main process ended, respawning

This means that whoopsie itself is exiting for some reason and then being respawned. wait-for-state is in no way causing whoopsie to be restarted; the only thing wait-for-state is doing is causing enough of a delay in the pre-start that the respawn limit is never hit, causing this to loop indefinitely. But the real issue is whatever is causing whoopsie to exit in the first place.

[Steve Langasek (vorlon)]

This is a regression introduced in the last whoopsie upload in vivid. Prior to this, the upstart job included deliberate handling in /etc/init/whoopsie.conf to ensure that if whoopsie were disabled in the config, the job would be stopped without triggering upstart's respawn behavior. Assigning this bug to the developer who made this change.

[Michael Blennerhassett (mjblenner)]

Looking at man 8 init, there is no job state "started", so having WAIT_STATE=started is exactly equal to WAIT_STATE=foobar, since they both don't exit.

So you may as well change it to something sensible.

Also, the reason that whoopsie exits as soon as it is respawned is that there is already another whoopsie process running, that either was started by something else (apport?), or upstart tost track of it.

Either way, fixing the wait-for-state (this patch or fixing whoopsie.conf), fixes the 30sec respawn bug.

[Clint Byrum (clint-fewbar)]

Michael, agree that 'started' is the wrong thing to look for and has probably caused a few confusing things through the years since wait-for-state has been in use. I think the patch should be merged.

[Didier Roche (didrocks)]

@Steve: I think I understand what happens.
On package upgrade, if the job is disabled, then, there is an upstart .override file written to disable it and so, that behavior shouldn't be triggered. I guess this case happens currently if you disable through ubuntu-control-center after a new installation as I discovered that there is another library (another source package), using that file: whoopsie-preferences that can read/set values in this file.

However, I see that there is another value about report_metrics that we don't use for job enablement status.
I propose to do the following on ubuntu-control-center: report crashes -> add the override and call systemctl disable if installed, change whoopsie-preferences to be aware of those files and detect usptart/systemd for current values.

This should prevent whoopsie to loop when the job is disabled and only have the current enablement state in one place (through libwhoopsie-preferences).

I can set the other changes that you described as well to fix the upstart job, just following what you see as I'm in no way an upstart expert :)

[Alan (alanjas)]

Tail of dmesg..

[ 4517.664355] init: wait-for-state (whoopsienetwork-manager) main process (7481) terminated with status 100
[ 4517.720627] init: whoopsie main process (7492) terminated with status 1
[ 4517.720645] init: whoopsie main process ended, respawning

[Chad Miller (cmiller)]

@didrocks, I see this bug on my laptop. Ping me if you want testing or info.

[Didier Roche (didrocks)]

Fixed, I added as well Michael's upstart job enhancement in this upload (even if that doesn't fix the real issue, still a nice addition). Didn't merge into upstream trunk as I don't have access right, but Brian would merge them back.

[Launchpad Janitor (janitor)]

This bug was fixed in the package whoopsie-preferences - 0.14

---------------
whoopsie-preferences (0.14) vivid; urgency=medium

  * Now use service status (upstart and systemd) to notify if crash reporting
    is enabled (LP: #1412719)
  * Default file for whoopsie settings (only used for report_metrics currently)
    is now /etc/whoopsie (LP: #1431432)
  * debian/control: add Breaks to ensure we dep on the right whoopsie version
  * Update Standards-Version.
 -- Didier Roche <email address hidden> Thu, 12 Mar 2015 17:39:30 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package whoopsie - 0.2.47

---------------
whoopsie (0.2.47) vivid; urgency=medium

  [ Didier Roche ]
  * Remove /etc/default/whoopsie conffile and avoid upgrade prompt:
    (LP: #1431432)
    - debian/whoopsie.maintscript: get it removed on upgrade
    - debian/whoopsie.preinst: copy the optional report_metrics value if set
      by gnome-control-center capplet to /etc/whoopsie (not a conffile).
      (LP: #1412719)
      Refactor the logic for caching report_crashes as well.
    - debian/whoopsie.postinst: enhance the logic to ensure we don't have the
      service started after upgrade and keep it disable if it was disabled
    - Makefile, data/whoopsie: remove the conffile so that we don't ship it
  * Bump Standards-Version

  [ Michael Blennerhassett ]
  * Under upstart: Set the WAIT_STATE to get wait-for-state to do something
    useful. GOAL is also not needed.
 -- Didier Roche <email address hidden> Thu, 12 Mar 2015 17:21:58 +0100