Service units get stuck in debug mode.

[Adam Gandelman]

Using 0.5+bzr427-1juju2~oneiric1 from PPA.

Depending on how debug-hooks mode is exited, units sometimes get stuck in debug mode. Later attempts to reconnect in debug mode fail with error:

2011-12-09 15:15:12,816 ERROR Service unit 'keystone/0' is already in debug mode.

I've found one simple way to reproduce is to choose "No" when verifying the host fingerprint during the initial ssh connection. Attempting to reconnect often finds the unit wedged in debug-mode. It sometimes takes a couple of reconnections. See log at
http://paste.ubuntu.com/765438/

[Kapil Thangavelu (hazmat)]

debug-hooks mode will exit when two things happen, there is no currently executing hook in the tmux window, and the admin client connection to zookeeper is closed. If the user closes the connection without exiting from the tmux session and a hook is currently executing, then the debug-hooks session is still active and the unit will wait on the hook to finish. Ideally we'd detect the end of the debug-session and forcibly terminate the debug-hook.

[Kapil Thangavelu (hazmat)]

in the context of the pastebin example, its not really an example of the wedged unit, just that the debug-hook flag isn't cleared out eagerly but relies on zk ephemeral node deletion, which should be tied to the end of the session and client closing, so this behavior is a little odd.

[Clint Byrum (clint-fewbar)]

Excerpts from Kapil Thangavelu's message of Fri Jan 27 02:24:34 UTC 2012:
> debug-hooks mode will exit when two things happen, there is no currently
> executing hook in the tmux window, and the admin client connection to
> zookeeper is closed. If the user closes the connection without exiting
> from the tmux session and a hook is currently executing, then the debug-
> hooks session is still active and the unit will wait on the hook to
> finish. Ideally we'd detect the end of the debug-session and forcibly
> terminate the debug-hook.
>
> ** Changed in: juju
> Status: New => Confirmed
>
> ** Changed in: juju
> Importance: Undecided => Low
>

Could we consider raising the importance a bit? debug-hooks is pretty
important, and if it is unreliable, that will be really frustrating
for users.