Enable lowlatency settings in the generic kernel
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Noble |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analysis of the specific low-latency features:
- CONFIG_
- CONFIG_
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://<email address hidden>/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
With all of that in place we can provide a kernel that has the flexibility to be more responsive, more performant and more power efficient (therefore more "generic"), simply by tuning run-time and boot-time options.
Moreover, once these changes are applied we will be able to deprecate the lowlatency kernel, saving engineering time and also reducing power consumption (required to build the kernel and do all the testing).
Optionally, we can also provide optimal "lowlatency" settings as a user-space package that would set the proper options in the kernel boot command line (GRUB, or similar).
[Test case]
There are plenty of benchmarks that can prove the validity of each one of the setting mentioned above, providing huge benefits in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should only be focused on this particular aspect, to evaluate the impact of this change in the worst case scenario.
Test case (CPU-intensive stress test):
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case it can show the impact of the additional kernel jitter introduced by the different CONFIG_HZ)
Results (linux-unstable 6.8.0-2.2, avg of 10 runs of 5min each):
- CONFIG_HZ=250 : 17415.60 bogo ops/s
- CONFIG_HZ=1000 : 14866.05 bogo ops/s
- CONFIG_
Results confirm the theory about the performance drop of CPU-intensive workloads (-~14%), but also confirms the benefit of NO_HZ_FULL (+~6%) compared to the current HZ settings.
Let's also keep in mind that this is the worst case scenario and a very specific one, where only HPC / scientific applications can be affected, and even in this case we can always compensate and actually get a better level performance exploiting the nohz_full capability.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option.
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
Changed in linux (Ubuntu Noble): | |
status: | New → Fix Committed |
I found this bug report by accident, while searching for something else.
I pretty much only use mainline kernels and only 1000 Hertz.
I support this proposed default Ubuntu kernel configuration change.
The tick ISR is incredibly efficient (less than 2 uSec on my test system), and I do not understand your test results as I would have expected a lot less difference. Using mainline kernel 6.8-rc1 and limiting my test system max CPU frequency so that I get similar bogo op/s as you I get:
- CONFIG_HZ=250 : 14853.14 bogo ops/s HZ=1000+ nohz_full : 15100 bogo ops/s (1.6% better)
- CONFIG_HZ=1000 : 14714.01 bogo ops/s (0.94% worse)
- CONFIG_
There is no power or thermal or active cores throttling on my test system.
Note: with my CPU frequency limited for this test, the tick ISR takes about 4 uSec.
The difference between the 250 and 1000 Hz kernel tests is 750 tick interrupts per second or 3 milliseconds or about 0.3%
If I do not limit my max CPU frequency I get:
- CONFIG_HZ=250 : 38518.64 bogo ops/s HZ=1000+ nohz_full : 39391.32 bogo ops/s (2.3% better)
- CONFIG_HZ=1000 : 37765.55 bogo ops/s (2.0% worse)
- CONFIG_
There was no power or thermal or active cores throttling for this test. However I did have to raise my processor max temperature limit from 75 to 80 degrees C for this test. Also the power is very close to the limit at 122 watts, where it will throttle at 125 watts.