relation data bootstrapped-osds remains at 0 even when there are OSDs

[Xav Paice]

Xenial, Pike, charms 18.11

We have ceph-mon and ceph-osd. We related ceph-mon to cinder-ceph, and expected the relation to complete but it did not because:

$ juju run --unit ceph-mon/4 'relation-get -r osd:258 bootstrapped-osds ceph-osd/1'
0

That unit actually has 6 OSDs. All the units were the same. I then updated the relation data:
juju run --unit ceph-osd/1 'relation-set -r mon:258 bootstrapped-osds=6'

Repeated that for all the ceph-osd units. The relation between ceph-mon and cinder-ceph was then able to complete.

[Xav Paice (xavpaice)]

Added ~field-critical.

Because bootstrapped-osds was set to 0, despite there being 48 OSDs in total all up and in, the relation never completed. This meant downtime for Cinder, and we were unable to roll back from the change to use the cinder-ceph subordinate as the relation between cinder and ceph-mon would also not complete.

This also prevents addition of Gnocchi to production environments, as the relation will again not complete.

[Chris MacNaughton (chris.macnaughton)]

Xav,

Can we please get a sanitized bundle that includes ceph-mon and ceph-osd?

[Chris MacNaughton (chris.macnaughton)]

Running this on a freshly deployed setup from stable, I ee:

$ juju run --unit=ceph-mon/0 "relation-get -r osd:1 bootstrapped-osds ceph-osd/1"
6

[James Page (james-page)]

The bootstrapped-osd's value is taken from the persistent record of block devices which each unit of the ceph-osd application tried to bootstrap into the cluster.

However if a deployment pre-dates the work to record that information during OSD bootstrap it will always return 0 - so setting the relation data will workaround the fix temporarily however the next time a config-changed hook runs (or an upgrade-charm) the value will get reset back to 0.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/634915

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/634919

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/634915
Committed: https://git.openstack.org/cgit/openstack/charms.ceph/commit/?id=886ef0b2243ffd0fc243ee01b5c9c58b622788f7
Submitter: Zuul
Branch: master

commit 886ef0b2243ffd0fc243ee01b5c9c58b622788f7
Author: Chris MacNaughton <email address hidden>
Date: Tue Feb 5 12:20:25 2019 +0100

    Ensure we populate osd-devices with existing devices

    If an older version of ceph-osd is deployed and then
    upgraded to a version that keeps track of bootstrapped
    OSDs, then the list of osd-devices never gets updated
    with the pre-existing devices.

    This change allows us to add existing, mounted Ceph
    OSDs to the osd-devices entry in the local KV storage.

    Change-Id: I940b108d914b39b55013a4617c3d17ff7122df60
    Closes-Bug: #1814597

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/634919
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-osd/commit/?id=9bc5abab555a77d746da221a3bbeae8e81b53fcc
Submitter: Zuul
Branch: master

commit 9bc5abab555a77d746da221a3bbeae8e81b53fcc
Author: Chris MacNaughton <email address hidden>
Date: Tue Feb 5 12:34:47 2019 +0100

    Ensure we populate osd-devices with existing devices

    If an older version of ceph-osd is deployed and then
    upgraded to a version that keeps track of bootstrapped
    OSDs, then the list of osd-devices never gets updated
    with the pre-existing devices.

    This change allows us to add existing, mounted Ceph OSDs
    to the osd-devices entry in the local KV storage.

    Change-Id: I17fab658511275f1dde15683ef296d4c72e7980e
    Closes-Bug: #1814597
    Depends-On: I940b108d914b39b55013a4617c3d17ff7122df60

[Xav Paice (xavpaice)]

is there an option to backport this fix to 18.11 or do we have to wait for 19.04 (i.e. fork the charm to get the fix)?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/636099

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/636099
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-osd/commit/?id=326f366cc7f97612d5f25df26f4f0ba9c3a286e4
Submitter: Zuul
Branch: stable/18.11

commit 326f366cc7f97612d5f25df26f4f0ba9c3a286e4
Author: Chris MacNaughton <email address hidden>
Date: Tue Feb 5 12:34:47 2019 +0100

    Ensure we populate osd-devices with existing devices

    If an older version of ceph-osd is deployed and then
    upgraded to a version that keeps track of bootstrapped
    OSDs, then the list of osd-devices never gets updated
    with the pre-existing devices.

    This change allows us to add existing, mounted Ceph OSDs
    to the osd-devices entry in the local KV storage.

    Change-Id: I17fab658511275f1dde15683ef296d4c72e7980e
    Closes-Bug: #1814597
    (cherry picked from commit 9bc5abab555a77d746da221a3bbeae8e81b53fcc)