Cannot detach storage

[Peter Matulis]

Juju says the detachment of the storage was successful but Ceph appears to ignore it.

snap info juju | grep installed
installed: 2.3-alpha1+develop-29aaaa2 (2268) 43MB classic

===

juju storage | grep osd-devices
ceph-osd/0 osd-devices/0 block ebs vol-0501570112618debf 32GiB attached
ceph-osd/0 osd-devices/1 block ebs vol-04345b11abe6bd5e8 32GiB attached
ceph-osd/1 osd-devices/3 block ebs vol-0eaada2acc2dc851b 32GiB attached
ceph-osd/1 osd-devices/4 block ebs vol-08d7fefb838d72217 32GiB attached
ceph-osd/2 osd-devices/6 block ebs vol-0d0a9a3a8cb4c4464 32GiB attached
ceph-osd/2 osd-devices/7 block ebs vol-0e0b54e0b70bb8ebd 32GiB attached

===

juju detach-storage osd-devices/7
detaching osd-devices/7

===

juju storage | grep osd-devices
            osd-devices/7 block ebs vol-0e0b54e0b70bb8ebd 32GiB detached
ceph-osd/0 osd-devices/0 block ebs vol-0501570112618debf 32GiB attached
ceph-osd/0 osd-devices/1 block ebs vol-04345b11abe6bd5e8 32GiB attached
ceph-osd/1 osd-devices/3 block ebs vol-0eaada2acc2dc851b 32GiB attached
ceph-osd/1 osd-devices/4 block ebs vol-08d7fefb838d72217 32GiB attached
ceph-osd/2 osd-devices/6 block ebs vol-0d0a9a3a8cb4c4464 32GiB attached

===

juju status ceph-osd
Model Controller Cloud/Region Version SLA
ceph aws-controller aws/us-east-1 2.3-alpha1.1 unsupported

App Version Status Scale Charm Store Rev OS Notes
ceph-osd 10.2.7 active 3 ceph-osd jujucharms 245 ubuntu

Unit Workload Agent Machine Public address Ports Message
ceph-osd/0 active idle 3 54.197.85.213 Unit is ready (2 OSD)
ceph-osd/1* active idle 4 54.146.68.53 Unit is ready (2 OSD)
ceph-osd/2 active idle 5 54.224.59.60 Unit is ready (2 OSD) <----------- SHOULD BE 1 OSD

Machine State DNS Inst id Series AZ Message
3 started 54.197.85.213 i-0a421dca4221bcf60 xenial us-east-1c running
4 started 54.146.68.53 i-016d906c158e2e2c0 xenial us-east-1e running
5 started 54.224.59.60 i-0d4b087f32320a560 xenial us-east-1a running

Relation Provides Consumes Type
mon ceph-mon ceph-osd regular

===

juju ssh ceph-osd/2 ps ax | grep /usr/bin/ceph-osd
   7198 ? Ssl 0:55 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
   7949 ? Ssl 0:50 /usr/bin/ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph

[Billy Olsen (billy-olsen)]

The charm reports the number of OSDs as the number of running OSD processes, so from this perspective the charm is reporting the correct value since it appears that the process is running. To better determine where the issue might lie, is the storage truly removed, or is the disk actually still attached to the running instance?

[Andrew Wilkins (axwalk)]

Billy, I can reproduce this with the following (on AWS):
 - juju deploy -n 3 ceph-mon
 - juju deploy -n 3 ceph-osd --storage osd-devices=ebs,32G,2 --storage osd-journals=ebs,8G,1
 - juju add-relation ceph-mon ceph-osd

(wait for all to settle)

Run "juju storage" to list the storage connected to ceph-osd/0, and detach one of the osd-devices storage instance. In my case, I have osd-devices/0 attached to ceph-osd/0, so I run:

 - juju detach-storage osd-devices/0

Wait, and it will show as "detached" in Juju. It seems that we're marking it as detached too eagerly. In fact, the EBS volume is still attached to the machine, and the reason for that is that the block device is still mounted and in use by the ceph-osd process.

The ceph-osd charm should be handling the "storage-detaching" hook by stopping the ceph-osd process, and unmounting the block device. Unless that happens, the EBS volume cannot be cleanly detached from the machine.

[James Page (james-page)]

I'm marking this as 'Low' as the typical deployment use case for Ceph is not clouds, where block device detachment is something that can actually happen; most Ceph deployments are hardware bases where the block devices attached don't change via software operations.