Upstart jobs can enter “zombie” state that require reboot

[Armin Ronacher]

It appears that a job can enter a special zombie state where it's neither alive nor dead and requires a system reboot to recover.

IRC log with the discussion: https://gist.github.com/3512329
strace of init and the service command: https://gist.github.com/5f9061af79bb8b38d240

System environment: precise64 from the ubuntu cloud package (the amazon eu west AMI).
After an upgrade of a service an upstart job corrupted and left it in a state where it neither starts now stops.

It was later suggested on IRC that this might be by design.

[Armin Ronacher (mitsuhiko)]

The pid wraparound trick works btw. I got it to start applications until it spawned a new dummy with that pid: https://gist.github.com/d55199e9b1ad7fc65504

Interestingly enough the pid shown in the status message was not the pid that it actually tried to kill. I suppose the problem is that on salt update upstart expected it to demonize but the update did not go fully through and the process instead did something else. (Died after the first fork?)

[Cameron Norman (cameronnemo)]

Can you clarify? What I am hearing is that the job is alive, but the process is dead. initctl stop salt-master does not work because the process is not actually running. Perhaps if Upstart tries to stop a job, and the process is not running, that job can be considered stopped/killed.