bluestore-specific configuration support

[Dmitrii Shcherbakov]

With bluestore support to be declared stable and default in Luminous there will be several changes needed in charm-ceph-osd to support new deployment layouts.

http://ceph.com/releases/v12-1-0-luminous-rc-released/

"The new BlueStore backend for ceph-osd is now stable and the new default for newly created OSDs."

A couple of overviews of changes in general:
http://events.linuxfoundation.org/sites/events/files/slides/20170323%20bluestore.pdf
https://www.sebastien-han.fr/blog/2016/03/21/ceph-a-new-store-is-coming/

1. Ceph Journal used to store both data and metadata and with BlueStore this is no longer the case - data will be stored directly on a block device without first being stored on a journal and later on an XFS partition;

2. There will be no double-write penalty for data due to that;

3. It will be possible to store data, RocksDB WAL (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log-File-Format) and RocksDB SSTs (https://github.com/facebook/rocksdb/wiki/A-Tutorial-of-RocksDB-SST-formats) on separate devices via symlinks.

We need to teach the ceph-osd charm to respect both FileStore- and BlueStore-specific options as journal configuration is different for BlueStore:
https://github.com/ceph/ceph/blob/luminous/src/common/options.cc#L3147-L3190

Code-wise, _setup_block_symlink_or_file is called multiple times for different purposes.
https://github.com/ceph/ceph/blob/luminous/src/os/bluestore/BlueStore.cc#L4953

This has a flexibility benefit of using either block special files or regular files by changing where a symlink points to:

https://github.com/ceph/ceph/blob/luminous/src/os/bluestore/BlueStore.cc#L5125-L5141
src/os/bluestore/BlueStore.cc|5125| r = _setup_block_symlink_or_file("block", cct->_conf->bluestore_block_path,
src/os/bluestore/BlueStore.cc|5131| r = _setup_block_symlink_or_file("block.wal", cct->_conf->bluestore_block_wal_path,
src/os/bluestore/BlueStore.cc|5136| r = _setup_block_symlink_or_file("block.db", cct->_conf->bluestore_block_db_path,

In other words, we should supports setups like:

1) block - HDD or bcache;
2) block.wal - a relatively fast SSD;
3) block.db - very fast, low-latency device (e.g. Intel Optane).

And hence support 3 config options in the charm for setting up those symlinks depending on the backing store currently configured.

[James Page (james-page)]

Marking as a feature bug; we have basic bluestore support in for the next release, these further improve the support to allow broader configuration of WAL and DB devices for bluestore configurations.

[Dmitrii Shcherbakov (dmitriis)]

I have reviews here which I tested with config options and juju storage on MAAS. I submitted them on the Pike charms freeze day so they did not make it there.

https://review.openstack.org/#/c/497686/
https://review.openstack.org/#/c/497691/

Given that configuration code for WAL and DB is identical to the FileStore journal configuration code it is quite a simple fix.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/497691
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-osd/commit/?id=189e7620c07d81c5fdaa2c680c51ffd2d29f168b
Submitter: Zuul
Branch: master

commit 189e7620c07d81c5fdaa2c680c51ffd2d29f168b
Author: Dmitrii Shcherbakov <email address hidden>
Date: Mon Aug 28 16:13:52 2017 +0300

    add bluestore-specific config options

    Adds bluestore-specific options related to the metadata-only journal.

    The options allow a user to control:

    1. path to a bluestore wal (block special file or regular file)
    2. path to a bluestore db (block special file or regular file)
    3. size of both

    Their configuration works similarly to the FileStore journal. If paths
    are not specified both WAL and DB will be collocated on the same block
    device as data.

    Other options can be configured via an existing config-flags option if needed.
    http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/

    Closes-Bug: #1710474
    Change-Id: Ia85092230d4dcb0435354deb276012f923547393
    Depends-On: I483ee9dae4ce69c71ae06359d0fb96aaa1c56cbc
    Depends-On: Idbbb69acec92b2f2efca80691ca73a2030bcf633

[Chris MacNaughton (chris.macnaughton)]

Marking the charm-ceph task wontfix as the ceph charm has been removed from support for a while now