Still working with this.  The last time I removed the model from bare-metal to k8s with CMR, it worked fine.

This time, I have CMR from ceph-mon bare-metal openstack to k8s running inside openstack.

K8s model destroyed fine, however, ceph-mon is failing to remove on the bare-metal model.

from 'juju debug-log --include ceph-mon'

---

unit-ceph-mon-0: 16:34:18 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-0: 16:34:18 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-0: 16:34:19 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-0: 16:34:19 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-0: 16:34:20 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-0: 16:34:20 INFO juju.worker.uniter.operation skipped "client-relation-broken" hook (missing)
unit-ceph-mon-0: 16:34:22 INFO unit.ceph-mon/0.juju-log Unknown hook stop - skipping.
unit-ceph-mon-0: 16:34:25 INFO juju.worker.uniter.operation ran "stop" hook
unit-ceph-mon-0: 16:34:25 DEBUG juju.worker.uniter.remotestate got unit change
unit-ceph-mon-0: 16:34:25 INFO juju.worker.uniter unit "ceph-mon/0" shutting down: agent should be terminated
unit-ceph-mon-0: 16:33:40 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-1: 16:33:40 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-2: 16:33:40 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-2: 16:33:40 INFO juju.worker.uniter.operation skipped "mds-relation-departed" hook (missing)
unit-ceph-mon-1: 16:33:40 INFO juju.worker.uniter.operation skipped "mds-relation-departed" hook (missing)
unit-ceph-mon-0: 16:33:40 INFO juju.worker.uniter.operation skipped "mds-relation-departed" hook (missing)
unit-ceph-mon-0: 16:33:41 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-2: 16:33:41 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-1: 16:33:41 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-1: 16:33:41 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-0: 16:33:41 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-2: 16:33:41 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-1: 16:33:41 INFO juju.worker.uniter.operation skipped "mds-relation-broken" hook (missing)
unit-ceph-mon-0: 16:33:41 INFO juju.worker.uniter.operation skipped "mds-relation-broken" hook (missing)
unit-ceph-mon-2: 16:33:41 INFO juju.worker.uniter.operation skipped "mds-relation-broken" hook (missing)
unit-ceph-mon-1: 16:33:42 INFO juju.worker.uniter.operation skipped "osd-relation-departed" hook (missing)
unit-ceph-mon-2: 16:33:43 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-0: 16:33:43 INFO juju.worker.uniter.operation skipped "osd-relation-departed" hook (missing)
unit-ceph-mon-1: 16:33:43 INFO juju.worker.uniter.operation skipped "radosgw-relation-departed" hook (missing)
unit-ceph-mon-2: 16:33:43 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-2: 16:33:43 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-0: 16:33:43 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-1: 16:33:43 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-2: 16:33:43 INFO juju.worker.uniter.operation skipped "osd-relation-departed" hook (missing)
unit-ceph-mon-1: 16:33:44 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-1: 16:33:44 INFO juju.worker.uniter.operation skipped "client-relation-broken" hook (missing)
unit-ceph-mon-2: 16:33:44 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-0: 16:33:45 WARNING unit.ceph-mon/0.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-2: 16:33:45 INFO juju.worker.uniter.operation skipped "radosgw-relation-departed" hook (missing)
unit-ceph-mon-0: 16:33:45 WARNING unit.ceph-mon/0.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-0: 16:33:45 INFO unit.ceph-mon/0.juju-log mon:0: Making dir /var/lib/charm/ceph-mon ceph:ceph 555
unit-ceph-mon-2: 16:33:46 WARNING unit.ceph-mon/2.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-1: 16:33:46 WARNING unit.ceph-mon/1.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-0: 16:33:46 WARNING unit.ceph-mon/0.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-2: 16:33:46 WARNING unit.ceph-mon/2.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-1: 16:33:46 WARNING unit.ceph-mon/1.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-2: 16:33:47 INFO unit.ceph-mon/2.juju-log mon:0: Making dir /var/lib/charm/ceph-mon ceph:ceph 555
unit-ceph-mon-0: 16:33:47 DEBUG juju.worker.uniter.remotestate got a relation units change: {41 {map[] map[] [cinder-ceph/0]}}
unit-ceph-mon-1: 16:33:47 DEBUG juju.worker.uniter.remotestate got a relation units change: {41 {map[] map[] [cinder-ceph/0]}}
unit-ceph-mon-2: 16:33:47 DEBUG juju.worker.uniter.remotestate got a relation units change: {41 {map[] map[] [cinder-ceph/0]}}
unit-ceph-mon-0: 16:33:47 DEBUG unit.ceph-mon/0.juju-log mon:0: Writing file /var/lib/charm/ceph-mon/ceph.conf root:root 644
unit-ceph-mon-1: 16:33:47 INFO unit.ceph-mon/1.juju-log mon:0: Making dir /var/lib/charm/ceph-mon ceph:ceph 555
unit-ceph-mon-0: 16:33:47 INFO unit.ceph-mon/0.juju-log mon:0: Not enough mons (2), punting.
unit-ceph-mon-0: 16:33:47 WARNING unit.ceph-mon/0.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-2: 16:33:47 WARNING unit.ceph-mon/2.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-1: 16:33:48 WARNING unit.ceph-mon/1.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-2: 16:33:48 DEBUG unit.ceph-mon/2.juju-log mon:0: Writing file /var/lib/charm/ceph-mon/ceph.conf root:root 644
unit-ceph-mon-2: 16:33:48 INFO unit.ceph-mon/2.juju-log mon:0: Not enough mons (2), punting.
unit-ceph-mon-2: 16:33:48 WARNING unit.ceph-mon/2.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-1: 16:33:48 DEBUG unit.ceph-mon/1.juju-log mon:0: Writing file /var/lib/charm/ceph-mon/ceph.conf root:root 644
unit-ceph-mon-1: 16:33:48 INFO unit.ceph-mon/1.juju-log mon:0: Not enough mons (2), punting.
unit-ceph-mon-1: 16:33:48 WARNING unit.ceph-mon/1.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-0: 16:33:51 INFO juju.worker.uniter.operation ran "mon-relation-departed" hook
unit-ceph-mon-0: 16:33:52 WARNING unit.ceph-mon/0.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-2: 16:33:52 INFO juju.worker.uniter.operation ran "mon-relation-departed" hook
unit-ceph-mon-0: 16:33:52 WARNING unit.ceph-mon/0.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-1: 16:33:52 INFO juju.worker.uniter.operation ran "mon-relation-departed" hook
unit-ceph-mon-2: 16:33:52 WARNING unit.ceph-mon/2.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-0: 16:33:53 INFO unit.ceph-mon/0.juju-log mon:0: Making dir /var/lib/charm/ceph-mon ceph:ceph 555
unit-ceph-mon-1: 16:33:53 WARNING unit.ceph-mon/1.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-2: 16:33:53 WARNING unit.ceph-mon/2.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-0: 16:33:53 WARNING unit.ceph-mon/0.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-1: 16:33:53 WARNING unit.ceph-mon/1.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-2: 16:33:53 INFO unit.ceph-mon/2.juju-log mon:0: Making dir /var/lib/charm/ceph-mon ceph:ceph 555
unit-ceph-mon-1: 16:33:54 INFO unit.ceph-mon/1.juju-log mon:0: Making dir /var/lib/charm/ceph-mon ceph:ceph 555
unit-ceph-mon-2: 16:33:54 WARNING unit.ceph-mon/2.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-0: 16:33:54 DEBUG unit.ceph-mon/0.juju-log mon:0: Writing file /var/lib/charm/ceph-mon/ceph.conf root:root 644
unit-ceph-mon-0: 16:33:54 INFO unit.ceph-mon/0.juju-log mon:0: Not enough mons (1), punting.
unit-ceph-mon-0: 16:33:54 WARNING unit.ceph-mon/0.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-1: 16:33:54 WARNING unit.ceph-mon/1.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-2: 16:33:54 DEBUG unit.ceph-mon/2.juju-log mon:0: Writing file /var/lib/charm/ceph-mon/ceph.conf root:root 644
unit-ceph-mon-2: 16:33:55 INFO unit.ceph-mon/2.juju-log mon:0: Not enough mons (1), punting.
unit-ceph-mon-2: 16:33:55 WARNING unit.ceph-mon/2.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-1: 16:33:55 DEBUG unit.ceph-mon/1.juju-log mon:0: Writing file /var/lib/charm/ceph-mon/ceph.conf root:root 644
unit-ceph-mon-0: 16:33:55 DEBUG juju.worker.uniter.remotestate got a relation units change: {42 {map[] map[] [glance/0]}}
unit-ceph-mon-1: 16:33:55 DEBUG juju.worker.uniter.remotestate got a relation units change: {42 {map[] map[] [glance/0]}}
unit-ceph-mon-2: 16:33:55 DEBUG juju.worker.uniter.remotestate got a relation units change: {42 {map[] map[] [glance/0]}}
unit-ceph-mon-1: 16:33:55 INFO unit.ceph-mon/1.juju-log mon:0: Not enough mons (1), punting.
unit-ceph-mon-1: 16:33:55 WARNING unit.ceph-mon/1.juju-log mon:0: Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-0: 16:33:55 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-2: 16:33:55 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-1: 16:33:55 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-0: 16:33:55 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-1: 16:33:55 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-2: 16:33:55 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-0: 16:33:58 INFO juju.worker.uniter.operation ran "mon-relation-departed" hook
unit-ceph-mon-2: 16:33:58 INFO juju.worker.uniter.operation ran "mon-relation-departed" hook
unit-ceph-mon-0: 16:33:58 INFO juju.worker.uniter.operation skipped "client-relation-broken" hook (missing)
unit-ceph-mon-0: 16:33:58 DEBUG juju.worker.uniter.remotestate got unit change
unit-ceph-mon-1: 16:33:58 INFO juju.worker.uniter.operation ran "mon-relation-departed" hook
unit-ceph-mon-2: 16:33:59 INFO juju.worker.uniter.operation skipped "client-relation-broken" hook (missing)
unit-ceph-mon-0: 16:33:59 WARNING juju.worker.uniter.operation we should run a leader-deposed hook here, but we can't yet
unit-ceph-mon-1: 16:33:59 INFO juju.worker.uniter.operation skipped "radosgw-relation-broken" hook (missing)
unit-ceph-mon-1: 16:33:59 DEBUG juju.worker.uniter.remotestate got unit change
unit-ceph-mon-2: 16:34:00 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-2: 16:34:00 DEBUG juju.worker.uniter.remotestate got unit change
unit-ceph-mon-0: 16:34:00 WARNING unit.ceph-mon/0.juju-log Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-2: 16:34:00 INFO juju.worker.uniter.operation skipped "client-relation-broken" hook (missing)
unit-ceph-mon-1: 16:34:00 INFO juju.worker.uniter.operation skipped "osd-relation-departed" hook (missing)
unit-ceph-mon-0: 16:34:00 WARNING unit.ceph-mon/0.juju-log Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-0: 16:34:01 INFO unit.ceph-mon/0.juju-log Making dir /var/lib/charm/ceph-mon ceph:ceph 555
unit-ceph-mon-2: 16:34:01 INFO juju.worker.uniter.operation skipped "osd-relation-departed" hook (missing)
unit-ceph-mon-1: 16:34:01 INFO juju.worker.uniter.operation skipped "mon-relation-broken" hook (missing)
unit-ceph-mon-0: 16:34:01 WARNING unit.ceph-mon/0.juju-log Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-2: 16:34:02 INFO juju.worker.uniter.operation skipped "client-relation-broken" hook (missing)
unit-ceph-mon-2: 16:34:02 DEBUG juju.worker.uniter.remotestate got a relation units change: {0 {map[] map[] [ceph-mon/1]}}
unit-ceph-mon-0: 16:34:02 DEBUG juju.worker.uniter.remotestate got a relation units change: {0 {map[] map[] [ceph-mon/1]}}
unit-ceph-mon-0: 16:34:02 DEBUG unit.ceph-mon/0.juju-log Changing permissions on existing content: 33188 -> 420
unit-ceph-mon-0: 16:34:02 INFO unit.ceph-mon/0.juju-log Not enough mons (1), punting.
unit-ceph-mon-0: 16:34:02 WARNING unit.ceph-mon/0.juju-log Support for use of upstream ``apt_pkg`` module in conjunctionwith charm-helpers is deprecated since 2019-06-25
unit-ceph-mon-1: 16:34:03 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-1: 16:34:03 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-0: 16:34:03 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-2: 16:34:03 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-1: 16:34:03 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-0: 16:34:03 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-2: 16:34:03 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-1: 16:34:03 INFO juju.worker.uniter.operation skipped "osd-relation-departed" hook (missing)
unit-ceph-mon-2: 16:34:04 INFO juju.worker.uniter.operation skipped "radosgw-relation-broken" hook (missing)
unit-ceph-mon-1: 16:34:04 INFO juju.worker.uniter.operation skipped "osd-relation-broken" hook (missing)
unit-ceph-mon-2: 16:34:05 INFO juju.worker.uniter.operation skipped "osd-relation-departed" hook (missing)
unit-ceph-mon-1: 16:34:06 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-2: 16:34:06 INFO juju.worker.uniter.operation skipped "mon-relation-broken" hook (missing)
unit-ceph-mon-1: 16:34:06 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-0: 16:34:07 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook
unit-ceph-mon-1: 16:34:07 DEBUG juju.worker.uniter.remotestate got a relation units change: {0 {map[] map[] [ceph-mon/2]}}
unit-ceph-mon-0: 16:34:07 DEBUG juju.worker.uniter.remotestate got a relation units change: {0 {map[] map[] [ceph-mon/2]}}
unit-ceph-mon-1: 16:34:07 INFO juju.worker.uniter.operation skipped "client-relation-broken" hook (missing)
unit-ceph-mon-2: 16:34:07 INFO juju.worker.uniter.operation skipped "osd-relation-broken" hook (missing)
unit-ceph-mon-0: 16:34:07 INFO juju.worker.uniter.operation skipped "radosgw-relation-departed" hook (missing)
unit-ceph-mon-1: 16:34:07 INFO juju.worker.uniter.operation skipped "client-relation-broken" hook (missing)
unit-ceph-mon-0: 16:34:08 INFO juju.worker.uniter.operation skipped "osd-relation-departed" hook (missing)
unit-ceph-mon-0: 16:34:09 INFO juju.worker.uniter.operation skipped "radosgw-relation-broken" hook (missing)
unit-ceph-mon-1: 16:34:09 INFO unit.ceph-mon/1.juju-log Unknown hook stop - skipping.
unit-ceph-mon-2: 16:34:10 INFO unit.ceph-mon/2.juju-log Unknown hook stop - skipping.
unit-ceph-mon-0: 16:34:10 INFO juju.worker.uniter.operation skipped "mon-relation-broken" hook (missing)
unit-ceph-mon-2: 16:34:11 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-0: 16:34:11 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-1: 16:34:11 DEBUG juju.worker.uniter.remotestate got application change
unit-ceph-mon-0: 16:34:11 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-2: 16:34:11 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-1: 16:34:11 DEBUG juju.worker.uniter.remotestate got a relation units change: {0 {map[] map[] [ceph-mon/0]}}
unit-ceph-mon-1: 16:34:11 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-ceph-mon-1: 16:34:12 INFO juju.worker.uniter.operation ran "stop" hook
unit-ceph-mon-0: 16:34:12 INFO juju.worker.uniter.operation skipped "osd-relation-departed" hook (missing)
unit-ceph-mon-2: 16:34:12 INFO juju.worker.uniter.operation ran "stop" hook
unit-ceph-mon-0: 16:34:12 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-1: 16:34:12 DEBUG juju.worker.uniter.remotestate got unit change
unit-ceph-mon-1: 16:34:12 INFO juju.worker.uniter unit "ceph-mon/1" shutting down: agent should be terminated
unit-ceph-mon-0: 16:34:12 INFO juju.worker.uniter.operation skipped "client-relation-departed" hook (missing)
unit-ceph-mon-2: 16:34:13 DEBUG juju.worker.uniter.remotestate got unit change
unit-ceph-mon-2: 16:34:13 INFO juju.worker.uniter unit "ceph-mon/2" shutting down: agent should be terminated
unit-ceph-mon-0: 16:34:13 INFO juju.worker.uniter.operation skipped "osd-relation-broken" hook (missing)
unit-ceph-mon-0: 16:34:15 INFO juju.worker.uniter.operation skipped "client-relation-broken" hook (missing)
unit-ceph-mon-0: 16:34:16 DEBUG juju.worker.uniter.remotestate got application change

---

Waiting for the destroy-model to timeout and i can send the debug model-logs from the bare-metal controller the model is on.