support advanced bcache tuning flags

[Nobuto Murata]

It would be nice if curin supported some bcache tuning flags in addition to the current cache_mode settings.

For example, a config flag for sequential_cutoff would be nice to have to maximize the usage of SSD.

> - Bad performance, or traffic not going to the SSD that you'd expect
>
> By default, bcache doesn't cache everything. It tries to skip sequential IO -
> because you really want to be caching the random IO, and if you copy a 10
> gigabyte file you probably don't want that pushing 10 gigabytes of randomly
> accessed data out of your cache.
>
> But if you want to benchmark reads from cache, and you start out with fio
> writing an 8 gigabyte test file - so you want to disable that::
>
> # echo 0 > /sys/block/bcache0/bcache/sequential_cutoff

[Nobuto Murata (nobuto)]

https://www.kernel.org/doc/Documentation/bcache.txt

[Ryan Harper (raharper)]

I think the priority would be for any config that is required during the initial creation; other settings can be configured via scripts or other tuning mechanism. The cutoff, for example isn't something curtin should do by default ,rather it's a knob for benchmarking. I'd think the benchmark test would set and clear that as needed.

I think the following config options make sense (creation time settings):

--data-offset <sectors> # defaults to 8k, but could get aligned for RAID backing devices
--cset-uuid <UUID> # maas already sents UUID values for filesystems, it could set these too

Most of the rest can be changed at runtime. We also need to investigate to see if the
bcache sysfs changes are persistent across reboot.

Besides the cutoff setting, are there other tunables that you would find useful to have inplace at install time?

[Nobuto Murata (nobuto)]

2018年3月26日(月) 23:20 Ryan Harper <email address hidden>:

> I think the priority would be for any config that is required during the
> initial creation; other settings can be configured via scripts or other
> tuning mechanism. The cutoff, for example isn't something curtin should
> do by default ,rather it's a knob for benchmarking. I'd think the
> benchmark test would set and clear that as needed.
>

I agree that those flags can be changed on the fly and good on the
benchmarking story. But what if we want to set the value permanently in a
new deployment after coming up with a good value with benchmarking/testing?

With juju story, I'm not sure it's appropriate to configure those with
charms since they don't have enough idea about bcache because they didn't
create it. I thought curtin is the only sensible place, but I'm not 100%
sure.

> I think the following config options make sense (creation time
> settings):
>
> --data-offset <sectors> # defaults to 8k, but could get aligned for RAID
> backing devices
> --cset-uuid <UUID> # maas already sents UUID values for filesystems,
> it could set these too
>
> Most of the rest can be changed at runtime. We also need to investigate
> to see if the
> bcache sysfs changes are persistent across reboot.
>
> Besides the cutoff setting, are there other tunables that you would find
> useful to have inplace at install time?
>
> ** Changed in: curtin
> Importance: Undecided => Wishlist
>
> ** Changed in: curtin
> Status: New => Triaged
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1758697
>
> Title:
> support advanced bcache tuning flags
>
> Status in curtin:
> Triaged
>
> Bug description:
> It would be nice if curin supported some bcache tuning flags in
> addition to the current cache_mode settings.
>
> For example, a config flag for sequential_cutoff would be nice to have
> to maximize the usage of SSD.
>
> > - Bad performance, or traffic not going to the SSD that you'd expect
> >
> > By default, bcache doesn't cache everything. It tries to skip
> sequential IO -
> > because you really want to be caching the random IO, and if you
> copy a 10
> > gigabyte file you probably don't want that pushing 10 gigabytes of
> randomly
> > accessed data out of your cache.
> >
> > But if you want to benchmark reads from cache, and you start out
> with fio
> > writing an 8 gigabyte test file - so you want to disable that::
> >
> > # echo 0 > /sys/block/bcache0/bcache/sequential_cutoff
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1758697/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=curtin; status=Triaged; importance=Wishlist;
> assignee=None;
> Launchpad-Bug-Tags: cpe-onsite
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: nobuto raharper
> Launchpad-Bug-Reporter: Nobuto Murata (nobuto)
> Launchpad-Bug-Modifier: Ryan Harper (raharper)
> Launchpad-Message-Rationale: Subscriber
> Launchpad-Message-F...

[Ante Karamatić (ivoks)]

I've subscribed two charms that are hit the most by this. Both Nobuto and Ryan are right. It's a in-flight option and charms should handle this. Charm should be intelligent enough to realize that OSD/ephemeral block device is bcache and, especially in Ceph OSD case, that sequential cutoff should be disabled.

[James Troup (elmo)]

+1 to having the charms do this; we turn off sequential cutoff wherever we use bcache.

[James Page (james-page)]

I've just spent the last hour scratching my head as to the best way to tune bcache devices via the charms; I wanted to take and approach where only devices presented to the charm where tuned; my first thought was to use tmpfiles.d to persist the cache settings however that's not going to be reliable with bcache due to the fact the bcache device naming is not consistent between reboots in systems with multiple bcache devices - which in this case is very likely.

[James Page (james-page)]

See attachments for current thinking - I'd want the tuning of the devices to be independent of the charm code, so having a systemd unit that runs after block device udev processing seems to make sense to me.

With that in mind, this might be nicer done as a subordinate charm; this will support converged and split deployments of nova-compute/swift/ceph-osd better, and we can drop it completely if MAAS/curtin grows support for configuring bcache params as part of installation (which could give per-device granularity to tuning).