support of "rbd cache = true" and "rbd cache writethrough until flush = true"

[Nobuto Murata]

Ceph v0.87 Giant release or later enables "rbd cache" and "rbd cache writethrough until flush" by default[1].
In Ubuntu repositories, vivid+ or cloud-archive:kilo+.

It would be nice if nova-compute charm has a config to enable "rbd cache" and "rbd cache writethrough until flush" as an option for Firefly. Regarding rbd cache settings, please refer [2].

[1] http://ceph.com/docs/master/release-notes/#upgrading-from-v0-80x-firefly
> The client-side caching for librbd is now enabled by default (rbd
> cache = true). A safety option (rbd cache writethrough until flush =
> true) is also enabled so that writeback caching is not used until the
> library observes a ‘flush’ command, indicating that the librbd
> users is passing that operation through from the guest VM. This
> avoids potential data loss when used with older versions of qemu that
> do not support flush.

[2] http://ceph.com/docs/master/rbd/rbd-config-ref/
> The user space implementation of the Ceph block device (i.e., librbd)
> cannot take advantage of the Linux page cache, so it includes its own
> in-memory caching, called “RBD caching.” RBD caching behaves just
> like well-behaved hard disk caching.

[Edward Hope-Morley (hopem)]

Making rbd cache settings controllable via charm makes sense for the reason
stated above but also to allow caching to be disabled for Ceph >= Giant since
some deployments may not want it to be enabled, perhaps for consistency when
performing an upgrade. Question is do we enable or disable by default in the
charms. I propose that we use an option like the following to allow Ceph
default behaviour to be used:

rbd-client-cache:
  type: string
  default:
  description: |
    Enable/disable rbd client cache. Leaving this value unset will result in
    default Ceph rbd client settings being used (rbd cache is enabled by
    default for Ceph >= Giant). Supported values here are "enabled" or
    "disabled".

[Nobuto Murata (nobuto)]

oh, the relation between libvirt/qemu cache and rbd cache is rather complicated then I originally thought.

http://ceph.com/docs/master/rbd/qemu-rbd/#qemu-cache-options
> QEMU’s cache settings override Ceph’s default settings (i.e., settings that are not explicitly set in the Ceph configuration file). If you explicitly set RBD Cache settings in your Ceph configuration file, your Ceph settings override the QEMU cache settings. If you set cache settings on the QEMU command line, the QEMU command line settings override the Ceph configuration file settings.

http://ceph.com/docs/master/rbd/qemu-rbd/#running-qemu-with-rbd
> Important
> If you set rbd_cache=true, you must set cache=writeback or risk data loss. Without cache=writeback, QEMU will not send flush requests to librbd. If QEMU exits uncleanly in this configuration, filesystems on top of rbd can be corrupted.

http://ceph.com/docs/master/rbd/qemu-rbd/#creating-images-with-qemu
> Important
> The raw data format is really the only sensible format option to use with RBD. Technically, you could use other QEMU-supported formats (such as qcow2 or vmdk), but doing so would add additional overhead, and would also render the volume unsafe for virtual machine live migration when caching (see below) is enabled.

It sounds like rbd cache is safe even for live-migration if we configure it correctly. Therefore having a explicit control to turn on/off is a good thing, I believe.

[Tilman Baumann (tilmanbaumann)]

Those commits never made it into master and subsequently never in any later releases. I can only find them in the stable/17.02 branch.

I suppose that wasn't intentional?