Add ratelimiting to more background daemons

[Tim Burke]

Background daemons have two main ways to make sure they don't hog resources:

* sleep between cycles (with something like interval or the deprecated run_pause) and
* ratelimiting within a cycle (with *_per_second settings).

The former prevents busy loops in a mostly empty cluster but causes square waves where the process hammers disks as fast as it can, then leaves them idle a while, then hammers them, leaves them idle, etc.

Ratelimiting is nice because it smooths that usage out -- instead of eating all the available I/O 50% of the time, the process can use ~50% of available I/O all the time, keeping the same cycle time but leaving overhead available for things like client requests.

Currently, however, only the following background daemons support ratelimiting:

* object-auditor
* object-updater
* container-auditor
* container-replicator
* container-sharder
* container-updater
* account-auditor
* account-replicator

It would be nice to have ratelimiting available for:

* object-reconstructor
* object-replicator
* container-reconciler

Maybe it'd be nice to have for object-expirer and account-reaper, too? Not sure.

[Matthew Oliver (matt-0)]

There is also nice and ionice features or rather the
'nice_priority, ionice_priority and ionice_class. For completion of options
of how much they hog resources.. tho I guess in this case it's how well
they share.

On Wed, Aug 1, 2018 at 9:13 AM, Tim Burke <email address hidden> wrote:

> Public bug reported:
>
> Background daemons have two main ways to make sure they don't hog
> resources:
>
> * sleep between cycles (with something like interval or the deprecated
> run_pause) and
> * ratelimiting within a cycle (with *_per_second settings).
>
> The former prevents busy loops in a mostly empty cluster but causes
> square waves where the process hammers disks as fast as it can, then
> leaves them idle a while, then hammers them, leaves them idle, etc.
>
> Ratelimiting is nice because it smooths that usage out -- instead of
> eating all the available I/O 50% of the time, the process can use ~50%
> of available I/O all the time, keeping the same cycle time but leaving
> overhead available for things like client requests.
>
> Currently, however, only the following background daemons support
> ratelimiting:
>
> * object-auditor
> * object-updater
> * container-auditor
> * container-updater
> * account-auditor
>
> It would be nice to have ratelimiting available for:
>
> * object-reconstructor
> * object-replicator
> * container-reconciler
> * container-replicator
> * container-sharder
> * account-replicator
>
> Maybe it'd be nice to have for object-expirer and account-reaper, too?
> Not sure.
>
> ** Affects: swift
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to
> OpenStack Object Storage (swift).
> Matching subscriptions: <email address hidden>
> https://bugs.launchpad.net/bugs/1784753
>
> Title:
> Add ratelimiting to more background daemons
>
> Status in OpenStack Object Storage (swift):
> New
>
> Bug description:
> Background daemons have two main ways to make sure they don't hog
> resources:
>
> * sleep between cycles (with something like interval or the deprecated
> run_pause) and
> * ratelimiting within a cycle (with *_per_second settings).
>
> The former prevents busy loops in a mostly empty cluster but causes
> square waves where the process hammers disks as fast as it can, then
> leaves them idle a while, then hammers them, leaves them idle, etc.
>
> Ratelimiting is nice because it smooths that usage out -- instead of
> eating all the available I/O 50% of the time, the process can use ~50%
> of available I/O all the time, keeping the same cycle time but leaving
> overhead available for things like client requests.
>
> Currently, however, only the following background daemons support
> ratelimiting:
>
> * object-auditor
> * object-updater
> * container-auditor
> * container-updater
> * account-auditor
>
> It would be nice to have ratelimiting available for:
>
> * object-reconstructor
> * object-replicator
> * container-reconciler
> * container-replicator
> * container-sharder
> * account-replicator
>
> Maybe it'd be nice to have for object-expirer and account-reaper, too?
> Not sure.
>
> To manage notifications abo...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/614018
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=06cf5d298fb6b103899aa358e1cb4b828f502dc5
Submitter: Zuul
Branch: master

commit 06cf5d298fb6b103899aa358e1cb4b828f502dc5
Author: Clay Gerrard <email address hidden>
Date: Mon Oct 29 14:49:48 2018 -0500

    Add databases_per_second to db daemons

    Most daemons have a "go as fast as you can then sleep for 30 seconds"
    strategy towards resource utilization; the object-updater and
    object-auditor however have some "X_per_second" options that allow
    operators much better control over how they spend their I/O budget.

    This change extends that pattern into the account-replicator,
    container-replicator, and container-sharder which have been known to peg
    CPUs when they're not IO limited.

    Partial-Bug: #1784753
    Change-Id: Ib7f2497794fa2f384a1a6ab500b657c624426384

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/755894
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=b05ad82959f41c5831ca7765e0b36594d6e2bf23
Submitter: Zuul
Branch: master

commit b05ad82959f41c5831ca7765e0b36594d6e2bf23
Author: Clay Gerrard <email address hidden>
Date: Fri Oct 2 17:16:09 2020 -0500

    Add tasks_per_second option to expirer

    This allows operators to throttle expirers as needed.

    Partial-Bug: #1784753
    Change-Id: If75dabb431bddd4ad6100e41395bb6c31a4ce569