ceph charms should use the CFQ IO scheduler so scrubbing can be de-prioritized

[James Troup]

Since firefly Ceph can set the IO priority and class for scrub traffic; this would help immensely on busy Ceph clusters. It does require using the CFQ scheduler for OSDs but this should be fine since OSDs are (generally) the only user of that device.

More information here: http://dachary.org/?p=3268

[Chris MacNaughton (chris.macnaughton)]

This could be done on all machines that house disks by setting the system scheduler to cfq, it looks like changing it per-disk, permanently will be a bit of a hack with a service to write `cfq` to the scheduler file. On the other hand, according to this (http://www.ilsistemista.net/index.php/linux-a-unix/38-linux-i-o-schedulers-benchmarked-anticipatory-vs-cfq-vs-deadline-vs-noop.html) somewhat older benchmark, cfq vs deadline isn't a huge performance killer unless using direct-io in which case deadline performs a bit better.

@elmo how would you feel about the ceph[-osd] units having the default scheduler be cfq instead of deadline?

[Chris MacNaughton (chris.macnaughton)]

After a bit more research (and IRC), I think I can implement this with a udev rule to set the scheduler whenever a disk we own is attached.

[Chris MacNaughton (chris.macnaughton)]

Hoping to get this upstreamed as a reasonable default in Ceph proper (https://github.com/ceph/ceph/pull/12940), depending on the outcome of that, we will either update our packages or make a similar change through the charms

[Edward Hope-Morley (hopem)]

@chris.macnaughton you currently have this bug set as In Progress for the charms but since you are investigating for the ceph packages first I suggest you mark the bug accordingly so that we can track the work properly.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[James Page (james-page)]

Reading: https://www.sebastien-han.fr/blog/2012/10/29/optimized-your-ssd/

Also feels like SSD's should use noop rather than deadline or cfq.

[James Page (james-page)]

I don't think this should be done in the package; if there was a good opinionated choice here I think upstream ceph would have already done it.

[James Page (james-page)]

Marking ceph charm task as won't fix; we have a migration path for ceph charm deployments to ceph-mon/ceph-osd, so ceph won't get any further new features or improvements.

[Frode Nordahl (fnordahl)]

For spinning disks, I/O schedulers other than deadline are subject to issues with read operations taking too long due to write starvation.

Just my $ 0.2