Extremely high memory consumption under heavy workload
Bug #1831021 reported by
Dariusz Gadomski
This bug affects 2 people
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
cups (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
Under heavy workload conditions cups can reach irrationally high memory consumption very quickly (tens of GBs).
Test case:
1. Set MaxJobs to 40000 in cupsd.conf.
2. sudo apt install cups-pdf
3. Fill the queue with jobs:
while [ 1 ]; lp -d PDF /usr/share/
4. Cancel all jobs
cancel -a PDF
5. Restart cups.
6. Start filling the queue again (as in step 3).
Expected result:
Jobs are processed and memory consumption is proportional to the number of jobs.
Actual result:
After step 5 or at latest step 6 memory consumption starts to increase exponentially - from ~150-200 MB to 8+GB. Without foreseeing this it's very easy to get cupsd killed by OOM killer.
description: | updated |
tags: | added: rls-ee-incoming |
Changed in cups (Ubuntu): | |
status: | Confirmed → Triaged |
tags: | added: sts |
tags: |
added: rls-ee-notfixing removed: rls-ee-incoming |
To post a comment you must log in.
After some analysis I believe the very high default value of LogDebugHistory (99999) is to blame for this (attaching massif report from the test case).
With a high number of jobs it's very easy to get hunderds-thousands of messages in LogDebugHistory which in turn will lead to enormous memory consumption, e.g.
40000 jobs * 2000 lines * (16 bytes for cupsd_joblog_t structure + 50-60 chars per message on average) - 5.66 GB.
After setting LogDebugHistory to 0 the memory consumption never exceeded 300 MB in my tests under the same heavy load. That's a very significant difference.
The possible workarounds:
1. Decrease LogDebugHistory to a low value (even to 0, upstream it's set to 200 by default).
2. Change LogLevel to debug (it will slow down cupsd operation significantly, but at the price of lower memory consumption as the LogDebugHistory is unused).
3. Make sure jobs are cleaned periodically by setting e.g. PreserveJobHistory 300.
I am wondering what was the rationale behind increasing LogDebugHistory to such a high value.