service with no units stuck in lifecycle dying

[Kapil Thangavelu]

[Impact]
Services will no service units get stuck in 'dying' state preventing their removal from a deployment.

[Test Case]
juju deploy mysql
juju terminate-machine --force <machineid of mysql>
juju destroy-service mysql

[Regression Potential]
Part of the upstream tested 1.16.6 release. Change looks limited to impacted code path only

[Original Report]
[Report from the field, a service with no units (previously destroyed) is stuck in lifecycle dying. Per status snippet

mysql:
 charm: local:precise/mysql-309
 exposed:false
 life: dying
 relations:
     cluster:
     - mysql

[Kapil Thangavelu (hazmat)]

poking at the underlying mongodb shows that the mysql service still has an extant relation and no units, per william on irc <fwereade> hazmat, to me the really critical thing is that one of those units apparently managed to leave scope without updating the relation doc's unitcount

[Kapil Thangavelu (hazmat)]

<fwereade> hazmat, the service was kept alive by the relation, which was kept alive by its unit count, which implied there'd be a unit to do the final leavescope and set off the dominos to take down the relation and the service

[Kapil Thangavelu (hazmat)]

I have an export of the mongodb for this environment, if anyone needs it
for additional analysis.

On Sat, Oct 12, 2013 at 1:21 PM, Curtis Hovey <email address hidden> wrote:

> ** Tags added: destroy-service
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1233457
>
> Title:
> service with no units stuck in lifecycle dying
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1233457/+subscriptions
>

[Curtis Hovey (sinzui)]

This issue relates to bug 1205451. In this case, the machine terminated before the state server could tell the agent that it is dead. In the other bug, the machine terminated for other reasons. In both cases, The state-server does not recognise that the agent and machine are gone, so it only needs to remove the record of the agent.

[Kapil Thangavelu (hazmat)]

On Friday, October 25, 2013, Curtis Hovey wrote:

> This issue relates to bug 1205451. In this case, the machine terminated
> before the state server could tell the agent that it is dead. In the
> other bug, the machine terminated for other reasons. In both cases, The
> state-server does not recognise that the agent and machine are gone, so
> it only needs to remove the record of the agent.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1233457
>
> Title:
> service with no units stuck in lifecycle dying
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1233457/+subscriptions
>

[William Reade (fwereade)]

I don't *think* it's related to lp:1205451 -- according to the transaction log captured by hazmat, mysql/0 never actually tried to leave relation scope for that relation... but *did* otherwise shut down cleanly. This points to a bug in the Uniter itself; still investigating.

[William Reade (fwereade)]

Root cause remains undetermined, but we can still ensure units are not removed without leaving all their relation scopes. Fix in progress.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Rolf Leggewie (r0lf)]

saucy has seen the end of its life and is no longer receiving any updates. Marking the saucy task for this ticket as "Won't Fix".