cluster log slow request spam
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu Cloud Archive |
Fix Released
|
High
|
Unassigned | ||
Train |
Fix Released
|
High
|
gerald.yang | ||
Ussuri |
Fix Released
|
High
|
gerald.yang | ||
ceph (Ubuntu) |
Fix Released
|
High
|
gerald.yang | ||
Focal |
Fix Released
|
High
|
gerald.yang | ||
Groovy |
Fix Released
|
High
|
gerald.yang | ||
Hirsute |
Fix Released
|
High
|
gerald.yang |
Bug Description
[Impact]
A recent change (issue#43975 [0]) was made to slow request logging to include detail on each operation in the cluster logs. With this change, detail for every slow request is always sent to the monitors and added to the cluster logs.
This does not scale. Large, high-throughput clusters can overwhelm their monitors with spurious logs in the event of a performance issue. Disrupting the monitors can then cause further instability in the cluster.
This SRU reverts the cluster logging of every slow request the osd is processing.
The slow request clog change was added in nautilus (14.2.10) and octopus (15.2.0).
[Test Case]
Stress the cluster with a benchmarking tool to generate slow requests and observe the cluster logs.
[Where problems could occur]
The cluster logs contain detailed debug information on slow requests that is useful for smaller, low-throughput clusters. While these logs are not used by ceph, they may be used by the cluster administrators (for monitoring or alerts). Changing this logging behavior may be unexpected.
[Other Info]
The intent is to re-enable this feature behind a configurable setting, but the solution must be discussed upstream.
The same slow request detail can be enabled for each osd by raising the "debug osd" log level to 20.
Related branches
- James Page: Pending requested
-
Diff: 73 lines (+51/-0)3 files modifieddebian/changelog (+7/-0)
debian/patches/disable-log-slow-requests.patch (+43/-0)
debian/patches/series (+1/-0)
tags: | added: seg sts |
Changed in ceph (Ubuntu Hirsute): | |
status: | New → In Progress |
importance: | Undecided → High |
Changed in ceph (Ubuntu Groovy): | |
importance: | Undecided → High |
Changed in ceph (Ubuntu Focal): | |
importance: | Undecided → High |
Changed in cloud-archive: | |
importance: | Undecided → High |
Changed in ceph (Ubuntu Groovy): | |
status: | New → In Progress |
Changed in ceph (Ubuntu Focal): | |
status: | New → In Progress |
Changed in cloud-archive: | |
status: | New → In Progress |
Changed in ceph (Ubuntu Focal): | |
assignee: | nobody → gerald.yang (gerald-yang-tw) |
Changed in ceph (Ubuntu Groovy): | |
assignee: | nobody → gerald.yang (gerald-yang-tw) |
Changed in ceph (Ubuntu Hirsute): | |
assignee: | nobody → gerald.yang (gerald-yang-tw) |
tags: | added: sts-sru-needed |
Changed in cloud-archive: | |
status: | In Progress → Fix Committed |
The attachment "0001-Remove- logging- every-slow- request- details- to-monito. patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.
[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]