launchpad process not exiting after make stop on staging

[Herb McNew]

For the last few days the staging restore has failed to restart the staging app server. It appears that the launchpad process isn't exiting after a 'make stop'.

[Diogo Matsubara (matsubara)]

Are there any relevant logs that would help debug this?

[Steve McInerney (spm)]

We see this across a range of services: eg edge*; lpnet*; xmlrpc

The most recent case was edge4.
AFAICT, there are no relevant logs.

Symptoms observed:
* The service in question is no longer responsive
* simple HTTP ping (alive/dead) style checks show the service as being alive
* more detailed checks (looking for XYZ string back) show a fail
* the service will be working fine until the 'make stop' which leaves it in this never-never land

We have an RT against the LOSAs to add a 'die die die' loop to the init.d script; but that's a fairly harsh workaround.

[Stuart Bishop (stub)]

Rather than put the loop in the init.d script, why not just put it in the Makefile so everything benefits?

I doubt we can track down the cause, and even if we did, we have no idea that we tracked down all of the causes or that new issues in Launchpad, Ubuntu, our hundred odd immediate dependencies or the dependencies of the dependencies will remain bug free and hand proof in the future.

[Tom Haddon (mthaddon)]

I think that's a fair point, Stuart. I think it best for the foundations team to determine the best way to handle this if it's going to be done within the Makefile.

Adjusting status and priority as it impacts edge rollouts (among other things).

[Gary Poster (gary)]

Stuart, if I'm right that putting this in the Makefile will be short work (less than a couple of hours) for you, could you tackle this for 3.0? If I'm wrong, please feel free to change the milestone to 3.1, or to bring it up with me.

Thanks

[Tom Haddon (mthaddon)]

This causes intermittent failures in the automatic edge update process if one of the app servers refuses to die. Latest one was Sept 9th.

[Stuart Bishop (stub)]

killservice (and thus make stop) now kills services using a SIGKILL if the SIGTERM did not cause a shutdown after 20 seconds. This timeout can be overridden on the command line.