object auditor creates high amount of writes on nearly empty clusters
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Fix Released
|
High
|
Christian Schwede |
Bug Description
Let's assume the following: you have a nearly empty cluster, with a high amount of partitions per disk. Each partition might store a small object; it's fast to audit.
The auditor will write a new status file after each partition, and this will also update the journal on disk. However, each partition will only take a few milliseconds to audit, so you're writing out a few KB every few milliseconds, plus updating the journal every time.
This can create quite some I/O load, at least every 30 seconds.
You can easily simulate this when doing the following test on a SAIO:
- upload 1000 empty objects
- set the auditor interval to 0
- run iostat -dx 2
There are always write requests, and depending on the amount of partitions and small objects this might be quite a few MB/s per disk.
I propose to update the auditor status file only once a minute at max, and will submit a patch for this.
Reviewed: https:/ /review. openstack. org/383707 /git.openstack. org/cgit/ openstack/ swift/commit/ ?id=77f5b201248 c0684d87289540d 3f94873f5c5e38
Committed: https:/
Submitter: Jenkins
Branch: master
commit 77f5b201248c068 4d87289540d3f94 873f5c5e38
Author: Christian Schwede <email address hidden>
Date: Fri Oct 7 12:17:08 2016 +0000
Throttle update_ auditor_ status calls
If there are quite a few nearly empty partitions per disk you might see
some write load even if your cluster is unused. The auditor will update
the status file after every partition, and this might happen multiple
times within a second if there is not much data stored yet.
This patch throttles updates, and will only write out an updated status
if the file was last updated more than a minute ago.
Closes-Bug: 1631352 2d28756f6ca4780 1674a7e6060
Change-Id: Ib61ec9cd945e6b