removal of principal ceph-osd charm stuck in 'executing' state

[Shane Peters]

Attempting to remove ceph-osd with ntp subordinate results in the charm getting stuck in terminated/executing/(stop) state. Of course, without the ntp relation, it removes cleanly.

Oddly enough, this doesn't occur in the 'ceph' charm.

# juju remove-application ceph-osd

Unit Workload Agent Machine Public address Ports Message
ceph-osd/27* terminated executing 55 10.191.162.249 (stop)
  ntp/49 active idle 10.191.162.249 Unit is ready

I've tried manually removing relations as well as the units themselves to no avail.

Reproduce
===================
$ juju deploy ntp
$ juju deploy ceph --config ceph_config.yaml
$ juju deploy ceph-osd --config ceph_config.yaml

$ juju add-relation ceph ntp
$ juju add-relation ceph-osd ntp
$ juju add-relation ceph-osd ceph

$ juju remove-application ceph-osd

[Shane Peters (shaner)]

Series: Xenial
Juju: 2.0.2
Charm: ceph-osd 10.2.3

[James Page (james-page)]

Please can you attach the log files from /var/log/juju/*.

Marking 'Incomplete' for now - I'm not sure why this specifically only effects the ceph-osd charm (not the ceph charm as described).

[Anastasia (anastasia-macmood)]

This looks oddly similar to https://bugs.launchpad.net/bugs/1655132... Maybe a duplicate?

[Billy Olsen (billy-olsen)]

I did not encounter this in my environment when I removed the ceph-osd service/application with the ntp subordinate. I suspect this is more of an issue on the juju unit agent (evidence somewhat agrees with another sighting that was unrelated to ceph-osd charm). I'll mark the task against the charm invalid.

[Billy Olsen (billy-olsen)]

Actually marking as Incomplete pending evidence it is an issue in the charm.

[William Grant (wgrant)]

I've seen this with cs:ubuntu with a single subordinate on a fresh 2.1.1 env. terminated/executing/(stop) persisted even across multiple reboots of the controller and the affected unit. There was nothing in the unit agent log to indicate that something had gone wrong after the hook fired the first time, and the log for the hook firing never appeared again.

[Christian Muirhead (2-xtian)]

I can reproduce this behaviour using wordpress and mysql with ntp related to both, it seems to be caused by the same problem as https://bugs.launchpad.net/juju/+bug/1686696.

PR for the fix: https://github.com/juju/juju/pull/7369