| 2024-01-26 08:57:01 |
Andrea Righi |
bug |
|
|
added bug |
| 2024-01-26 08:57:10 |
Andrea Righi |
nominated for series |
|
Ubuntu Noble |
|
| 2024-01-26 08:57:10 |
Andrea Righi |
bug task added |
|
linux (Ubuntu Noble) |
|
| 2024-01-26 08:59:43 |
Andrea Righi |
description |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analisys of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (can provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492)
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
[Test case]
There are plenty of micro-benchmarks to prove the validity of each one of the config mentioned above, in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should be focused on this particular aspect.
We also have the opportunity to test these changes using the lowlatency kernel, therefore a reasonable test plan could be defined as following.
Test case (a CPU-intensive stress test)
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the addditional kernel jitter introduced by the different CONFIG_HZ)
Perform multiple runs of the stress test, both on amd64 and arm64. Repeat the test also with CONFIG_NO_HZ_FULL. If NO_HZ_FULL can provide better results than the CONFIG_HZ=250 case, we can safely switch to CONFIG_HZ=1000, since we have the possibility to provide a boot-time option to improve CPU-intensive workloads and get better performance than the previous kernel.
TODO: results will follow.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analisys of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (can provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492)
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.). Moreover, it is worth noticing that pretty much all the other major Linux distributions are also using CONFIG_HZ0=1000.
[Test case]
There are plenty of micro-benchmarks to prove the validity of each one of the config mentioned above, in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should be focused on this particular aspect.
We also have the opportunity to test these changes using the lowlatency kernel, therefore a reasonable test plan could be defined as following.
Test case (a CPU-intensive stress test)
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the addditional kernel jitter introduced by the different CONFIG_HZ)
Perform multiple runs of the stress test, both on amd64 and arm64. Repeat the test also with CONFIG_NO_HZ_FULL. If NO_HZ_FULL can provide better results than the CONFIG_HZ=250 case, we can safely switch to CONFIG_HZ=1000, since we have the possibility to provide a boot-time option to improve CPU-intensive workloads and get better performance than the previous kernel.
TODO: results will follow.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
|
| 2024-01-26 12:21:44 |
Andrea Righi |
description |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analisys of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (can provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492)
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.). Moreover, it is worth noticing that pretty much all the other major Linux distributions are also using CONFIG_HZ0=1000.
[Test case]
There are plenty of micro-benchmarks to prove the validity of each one of the config mentioned above, in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should be focused on this particular aspect.
We also have the opportunity to test these changes using the lowlatency kernel, therefore a reasonable test plan could be defined as following.
Test case (a CPU-intensive stress test)
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the addditional kernel jitter introduced by the different CONFIG_HZ)
Perform multiple runs of the stress test, both on amd64 and arm64. Repeat the test also with CONFIG_NO_HZ_FULL. If NO_HZ_FULL can provide better results than the CONFIG_HZ=250 case, we can safely switch to CONFIG_HZ=1000, since we have the possibility to provide a boot-time option to improve CPU-intensive workloads and get better performance than the previous kernel.
TODO: results will follow.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analisys of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (can provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492)
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
Other latency-related options:
- CONFIG_LATENCYTOP=y: we should definitely support latencytop in the generic kernel, considering that we are shipping the user-space package
[Test case]
There are plenty of micro-benchmarks to prove the validity of each one of the config mentioned above, in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should be focused on this particular aspect.
We also have the opportunity to test these changes using the lowlatency kernel, therefore a reasonable test plan could be defined as following.
Test case (a CPU-intensive stress test)
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the addditional kernel jitter introduced by the different CONFIG_HZ)
Perform multiple runs of the stress test, both on amd64 and arm64. Repeat the test also with CONFIG_NO_HZ_FULL. If NO_HZ_FULL can provide better results than the CONFIG_HZ=250 case, we can safely switch to CONFIG_HZ=1000, since we have the possibility to provide a boot-time option to improve CPU-intensive workloads and get better performance than the previous kernel.
TODO: results will follow.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
|
| 2024-01-26 14:17:58 |
Andrea Righi |
description |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analisys of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (can provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492)
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
Other latency-related options:
- CONFIG_LATENCYTOP=y: we should definitely support latencytop in the generic kernel, considering that we are shipping the user-space package
[Test case]
There are plenty of micro-benchmarks to prove the validity of each one of the config mentioned above, in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should be focused on this particular aspect.
We also have the opportunity to test these changes using the lowlatency kernel, therefore a reasonable test plan could be defined as following.
Test case (a CPU-intensive stress test)
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the addditional kernel jitter introduced by the different CONFIG_HZ)
Perform multiple runs of the stress test, both on amd64 and arm64. Repeat the test also with CONFIG_NO_HZ_FULL. If NO_HZ_FULL can provide better results than the CONFIG_HZ=250 case, we can safely switch to CONFIG_HZ=1000, since we have the possibility to provide a boot-time option to improve CPU-intensive workloads and get better performance than the previous kernel.
TODO: results will follow.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analisys of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_RCU_LAZY_DEFAULT_OFF=y).
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
[Test case]
There are plenty of micro-benchmarks to prove the validity of each one of the config mentioned above, in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should be focused on this particular aspect.
We also have the opportunity to test these changes using the lowlatency kernel, therefore a reasonable test plan could be defined as following.
Test case (a CPU-intensive stress test)
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the addditional kernel jitter introduced by the different CONFIG_HZ)
Perform multiple runs of the stress test, both on amd64 and arm64. Repeat the test also with CONFIG_NO_HZ_FULL. If NO_HZ_FULL can provide better results than the CONFIG_HZ=250 case, we can safely switch to CONFIG_HZ=1000, since we have the possibility to provide a boot-time option to improve CPU-intensive workloads and get better performance than the previous kernel.
TODO: results will follow.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
|
| 2024-01-26 15:33:29 |
Andrea Righi |
description |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analisys of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_RCU_LAZY_DEFAULT_OFF=y).
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
[Test case]
There are plenty of micro-benchmarks to prove the validity of each one of the config mentioned above, in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should be focused on this particular aspect.
We also have the opportunity to test these changes using the lowlatency kernel, therefore a reasonable test plan could be defined as following.
Test case (a CPU-intensive stress test)
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the addditional kernel jitter introduced by the different CONFIG_HZ)
Perform multiple runs of the stress test, both on amd64 and arm64. Repeat the test also with CONFIG_NO_HZ_FULL. If NO_HZ_FULL can provide better results than the CONFIG_HZ=250 case, we can safely switch to CONFIG_HZ=1000, since we have the possibility to provide a boot-time option to improve CPU-intensive workloads and get better performance than the previous kernel.
TODO: results will follow.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analisys of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_RCU_LAZY_DEFAULT_OFF=y).
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
With all of that in place we can provide a kernel that has the flexibility to be more responsive, more performant and more power efficient, simply by tuning run-time and boot-time options.
Once these changes are applied we will be able to deprecate the lowlatency kernel, saving engineering time and also reducing power consumption (required to build the kernel and do all the testing).
Optionally, we can also provide optimal "lowlatency" settings as a user-space package that would set the proper options in the kernel boot command line (GRUB, or similar).
[Test case]
There are plenty of benchmarks that can prove the validity of each one of the setting mentioned above, providing huge benefits in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should only be focused on this particular aspect, to evaluate the impact of this change in the worst case scenario.
Test case (CPU-intensive stress test):
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the addditional kernel jitter introduced by the different CONFIG_HZ)
Results (linux-unstable 6.8.0-2.2, avg of 10 runs of 5min each):
- CONFIG_HZ=250 : 17415.60 bogo ops/s
- CONFIG_HZ=1000 : 14866.05 bogo ops/s
- CONFIG_HZ=1000+nohz_full : 18505.52 bogo ops/s
Results confirm the theory about the performance drop of CPU-intensive workloads (~14%), but also confirms the benefit of NO_HZ_FULL (~6%) compared to the current HZ settings.
Let's also keep in mind that this is the worst case scenario and a very specific one, where only HPC / scientific applications can be affected, and even in this case we can always compensate and actually get a better level performance exploiting the nohz_full capability.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
|
| 2024-01-26 16:13:17 |
Andrea Righi |
description |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analisys of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_RCU_LAZY_DEFAULT_OFF=y).
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
With all of that in place we can provide a kernel that has the flexibility to be more responsive, more performant and more power efficient, simply by tuning run-time and boot-time options.
Once these changes are applied we will be able to deprecate the lowlatency kernel, saving engineering time and also reducing power consumption (required to build the kernel and do all the testing).
Optionally, we can also provide optimal "lowlatency" settings as a user-space package that would set the proper options in the kernel boot command line (GRUB, or similar).
[Test case]
There are plenty of benchmarks that can prove the validity of each one of the setting mentioned above, providing huge benefits in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should only be focused on this particular aspect, to evaluate the impact of this change in the worst case scenario.
Test case (CPU-intensive stress test):
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the addditional kernel jitter introduced by the different CONFIG_HZ)
Results (linux-unstable 6.8.0-2.2, avg of 10 runs of 5min each):
- CONFIG_HZ=250 : 17415.60 bogo ops/s
- CONFIG_HZ=1000 : 14866.05 bogo ops/s
- CONFIG_HZ=1000+nohz_full : 18505.52 bogo ops/s
Results confirm the theory about the performance drop of CPU-intensive workloads (~14%), but also confirms the benefit of NO_HZ_FULL (~6%) compared to the current HZ settings.
Let's also keep in mind that this is the worst case scenario and a very specific one, where only HPC / scientific applications can be affected, and even in this case we can always compensate and actually get a better level performance exploiting the nohz_full capability.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analisys of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_RCU_LAZY_DEFAULT_OFF=y).
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
With all of that in place we can provide a kernel that has the flexibility to be more responsive, more performant and more power efficient (therefore more "generic"), simply by tuning run-time and boot-time options.
Moreover, once these changes are applied we will be able to deprecate the lowlatency kernel, saving engineering time and also reducing power consumption (required to build the kernel and do all the testing).
Optionally, we can also provide optimal "lowlatency" settings as a user-space package that would set the proper options in the kernel boot command line (GRUB, or similar).
[Test case]
There are plenty of benchmarks that can prove the validity of each one of the setting mentioned above, providing huge benefits in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should only be focused on this particular aspect, to evaluate the impact of this change in the worst case scenario.
Test case (CPU-intensive stress test):
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the additional kernel jitter introduced by the different CONFIG_HZ)
Results (linux-unstable 6.8.0-2.2, avg of 10 runs of 5min each):
- CONFIG_HZ=250 : 17415.60 bogo ops/s
- CONFIG_HZ=1000 : 14866.05 bogo ops/s
- CONFIG_HZ=1000+nohz_full : 18505.52 bogo ops/s
Results confirm the theory about the performance drop of CPU-intensive workloads (~14%), but also confirms the benefit of NO_HZ_FULL (~6%) compared to the current HZ settings.
Let's also keep in mind that this is the worst case scenario and a very specific one, where only HPC / scientific applications can be affected, and even in this case we can always compensate and actually get a better level performance exploiting the nohz_full capability.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
|
| 2024-01-26 16:15:07 |
Andrea Righi |
description |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analisys of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_RCU_LAZY_DEFAULT_OFF=y).
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
With all of that in place we can provide a kernel that has the flexibility to be more responsive, more performant and more power efficient (therefore more "generic"), simply by tuning run-time and boot-time options.
Moreover, once these changes are applied we will be able to deprecate the lowlatency kernel, saving engineering time and also reducing power consumption (required to build the kernel and do all the testing).
Optionally, we can also provide optimal "lowlatency" settings as a user-space package that would set the proper options in the kernel boot command line (GRUB, or similar).
[Test case]
There are plenty of benchmarks that can prove the validity of each one of the setting mentioned above, providing huge benefits in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should only be focused on this particular aspect, to evaluate the impact of this change in the worst case scenario.
Test case (CPU-intensive stress test):
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the additional kernel jitter introduced by the different CONFIG_HZ)
Results (linux-unstable 6.8.0-2.2, avg of 10 runs of 5min each):
- CONFIG_HZ=250 : 17415.60 bogo ops/s
- CONFIG_HZ=1000 : 14866.05 bogo ops/s
- CONFIG_HZ=1000+nohz_full : 18505.52 bogo ops/s
Results confirm the theory about the performance drop of CPU-intensive workloads (~14%), but also confirms the benefit of NO_HZ_FULL (~6%) compared to the current HZ settings.
Let's also keep in mind that this is the worst case scenario and a very specific one, where only HPC / scientific applications can be affected, and even in this case we can always compensate and actually get a better level performance exploiting the nohz_full capability.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analysis of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_RCU_LAZY_DEFAULT_OFF=y).
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
With all of that in place we can provide a kernel that has the flexibility to be more responsive, more performant and more power efficient (therefore more "generic"), simply by tuning run-time and boot-time options.
Moreover, once these changes are applied we will be able to deprecate the lowlatency kernel, saving engineering time and also reducing power consumption (required to build the kernel and do all the testing).
Optionally, we can also provide optimal "lowlatency" settings as a user-space package that would set the proper options in the kernel boot command line (GRUB, or similar).
[Test case]
There are plenty of benchmarks that can prove the validity of each one of the setting mentioned above, providing huge benefits in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should only be focused on this particular aspect, to evaluate the impact of this change in the worst case scenario.
Test case (CPU-intensive stress test):
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the additional kernel jitter introduced by the different CONFIG_HZ)
Results (linux-unstable 6.8.0-2.2, avg of 10 runs of 5min each):
- CONFIG_HZ=250 : 17415.60 bogo ops/s
- CONFIG_HZ=1000 : 14866.05 bogo ops/s
- CONFIG_HZ=1000+nohz_full : 18505.52 bogo ops/s
Results confirm the theory about the performance drop of CPU-intensive workloads (~14%), but also confirms the benefit of NO_HZ_FULL (~6%) compared to the current HZ settings.
Let's also keep in mind that this is the worst case scenario and a very specific one, where only HPC / scientific applications can be affected, and even in this case we can always compensate and actually get a better level performance exploiting the nohz_full capability.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
|
| 2024-01-26 16:16:15 |
Andrea Righi |
description |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analysis of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_RCU_LAZY_DEFAULT_OFF=y).
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
With all of that in place we can provide a kernel that has the flexibility to be more responsive, more performant and more power efficient (therefore more "generic"), simply by tuning run-time and boot-time options.
Moreover, once these changes are applied we will be able to deprecate the lowlatency kernel, saving engineering time and also reducing power consumption (required to build the kernel and do all the testing).
Optionally, we can also provide optimal "lowlatency" settings as a user-space package that would set the proper options in the kernel boot command line (GRUB, or similar).
[Test case]
There are plenty of benchmarks that can prove the validity of each one of the setting mentioned above, providing huge benefits in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should only be focused on this particular aspect, to evaluate the impact of this change in the worst case scenario.
Test case (CPU-intensive stress test):
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the additional kernel jitter introduced by the different CONFIG_HZ)
Results (linux-unstable 6.8.0-2.2, avg of 10 runs of 5min each):
- CONFIG_HZ=250 : 17415.60 bogo ops/s
- CONFIG_HZ=1000 : 14866.05 bogo ops/s
- CONFIG_HZ=1000+nohz_full : 18505.52 bogo ops/s
Results confirm the theory about the performance drop of CPU-intensive workloads (~14%), but also confirms the benefit of NO_HZ_FULL (~6%) compared to the current HZ settings.
Let's also keep in mind that this is the worst case scenario and a very specific one, where only HPC / scientific applications can be affected, and even in this case we can always compensate and actually get a better level performance exploiting the nohz_full capability.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analysis of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_RCU_LAZY_DEFAULT_OFF=y)
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
With all of that in place we can provide a kernel that has the flexibility to be more responsive, more performant and more power efficient (therefore more "generic"), simply by tuning run-time and boot-time options.
Moreover, once these changes are applied we will be able to deprecate the lowlatency kernel, saving engineering time and also reducing power consumption (required to build the kernel and do all the testing).
Optionally, we can also provide optimal "lowlatency" settings as a user-space package that would set the proper options in the kernel boot command line (GRUB, or similar).
[Test case]
There are plenty of benchmarks that can prove the validity of each one of the setting mentioned above, providing huge benefits in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should only be focused on this particular aspect, to evaluate the impact of this change in the worst case scenario.
Test case (CPU-intensive stress test):
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the additional kernel jitter introduced by the different CONFIG_HZ)
Results (linux-unstable 6.8.0-2.2, avg of 10 runs of 5min each):
- CONFIG_HZ=250 : 17415.60 bogo ops/s
- CONFIG_HZ=1000 : 14866.05 bogo ops/s
- CONFIG_HZ=1000+nohz_full : 18505.52 bogo ops/s
Results confirm the theory about the performance drop of CPU-intensive workloads (~14%), but also confirms the benefit of NO_HZ_FULL (~6%) compared to the current HZ settings.
Let's also keep in mind that this is the worst case scenario and a very specific one, where only HPC / scientific applications can be affected, and even in this case we can always compensate and actually get a better level performance exploiting the nohz_full capability.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
|
| 2024-01-26 16:17:20 |
Andrea Righi |
description |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analysis of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_RCU_LAZY_DEFAULT_OFF=y)
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
With all of that in place we can provide a kernel that has the flexibility to be more responsive, more performant and more power efficient (therefore more "generic"), simply by tuning run-time and boot-time options.
Moreover, once these changes are applied we will be able to deprecate the lowlatency kernel, saving engineering time and also reducing power consumption (required to build the kernel and do all the testing).
Optionally, we can also provide optimal "lowlatency" settings as a user-space package that would set the proper options in the kernel boot command line (GRUB, or similar).
[Test case]
There are plenty of benchmarks that can prove the validity of each one of the setting mentioned above, providing huge benefits in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should only be focused on this particular aspect, to evaluate the impact of this change in the worst case scenario.
Test case (CPU-intensive stress test):
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the additional kernel jitter introduced by the different CONFIG_HZ)
Results (linux-unstable 6.8.0-2.2, avg of 10 runs of 5min each):
- CONFIG_HZ=250 : 17415.60 bogo ops/s
- CONFIG_HZ=1000 : 14866.05 bogo ops/s
- CONFIG_HZ=1000+nohz_full : 18505.52 bogo ops/s
Results confirm the theory about the performance drop of CPU-intensive workloads (~14%), but also confirms the benefit of NO_HZ_FULL (~6%) compared to the current HZ settings.
Let's also keep in mind that this is the worst case scenario and a very specific one, where only HPC / scientific applications can be affected, and even in this case we can always compensate and actually get a better level performance exploiting the nohz_full capability.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analysis of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_RCU_LAZY_DEFAULT_OFF=y)
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
With all of that in place we can provide a kernel that has the flexibility to be more responsive, more performant and more power efficient (therefore more "generic"), simply by tuning run-time and boot-time options.
Moreover, once these changes are applied we will be able to deprecate the lowlatency kernel, saving engineering time and also reducing power consumption (required to build the kernel and do all the testing).
Optionally, we can also provide optimal "lowlatency" settings as a user-space package that would set the proper options in the kernel boot command line (GRUB, or similar).
[Test case]
There are plenty of benchmarks that can prove the validity of each one of the setting mentioned above, providing huge benefits in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should only be focused on this particular aspect, to evaluate the impact of this change in the worst case scenario.
Test case (CPU-intensive stress test):
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case it can show the impact of the additional kernel jitter introduced by the different CONFIG_HZ)
Results (linux-unstable 6.8.0-2.2, avg of 10 runs of 5min each):
- CONFIG_HZ=250 : 17415.60 bogo ops/s
- CONFIG_HZ=1000 : 14866.05 bogo ops/s
- CONFIG_HZ=1000+nohz_full : 18505.52 bogo ops/s
Results confirm the theory about the performance drop of CPU-intensive workloads (~14%), but also confirms the benefit of NO_HZ_FULL (~6%) compared to the current HZ settings.
Let's also keep in mind that this is the worst case scenario and a very specific one, where only HPC / scientific applications can be affected, and even in this case we can always compensate and actually get a better level performance exploiting the nohz_full capability.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
|
| 2024-01-26 16:18:26 |
Andrea Righi |
description |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analysis of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_RCU_LAZY_DEFAULT_OFF=y)
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
With all of that in place we can provide a kernel that has the flexibility to be more responsive, more performant and more power efficient (therefore more "generic"), simply by tuning run-time and boot-time options.
Moreover, once these changes are applied we will be able to deprecate the lowlatency kernel, saving engineering time and also reducing power consumption (required to build the kernel and do all the testing).
Optionally, we can also provide optimal "lowlatency" settings as a user-space package that would set the proper options in the kernel boot command line (GRUB, or similar).
[Test case]
There are plenty of benchmarks that can prove the validity of each one of the setting mentioned above, providing huge benefits in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should only be focused on this particular aspect, to evaluate the impact of this change in the worst case scenario.
Test case (CPU-intensive stress test):
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case it can show the impact of the additional kernel jitter introduced by the different CONFIG_HZ)
Results (linux-unstable 6.8.0-2.2, avg of 10 runs of 5min each):
- CONFIG_HZ=250 : 17415.60 bogo ops/s
- CONFIG_HZ=1000 : 14866.05 bogo ops/s
- CONFIG_HZ=1000+nohz_full : 18505.52 bogo ops/s
Results confirm the theory about the performance drop of CPU-intensive workloads (~14%), but also confirms the benefit of NO_HZ_FULL (~6%) compared to the current HZ settings.
Let's also keep in mind that this is the worst case scenario and a very specific one, where only HPC / scientific applications can be affected, and even in this case we can always compensate and actually get a better level performance exploiting the nohz_full capability.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
[Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.
Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.
Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).
Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.
Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).
As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again.
Following a detailed analysis of the specific low-latency features:
- CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time
- CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter)
- CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (c'an provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-joel@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492), and by default it will be disabled (CONFIG_RCU_LAZY_DEFAULT_OFF=y)
- CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.).
With all of that in place we can provide a kernel that has the flexibility to be more responsive, more performant and more power efficient (therefore more "generic"), simply by tuning run-time and boot-time options.
Moreover, once these changes are applied we will be able to deprecate the lowlatency kernel, saving engineering time and also reducing power consumption (required to build the kernel and do all the testing).
Optionally, we can also provide optimal "lowlatency" settings as a user-space package that would set the proper options in the kernel boot command line (GRUB, or similar).
[Test case]
There are plenty of benchmarks that can prove the validity of each one of the setting mentioned above, providing huge benefits in terms of system responsive.
However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should only be focused on this particular aspect, to evaluate the impact of this change in the worst case scenario.
Test case (CPU-intensive stress test):
- stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief
Metrics:
- measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case it can show the impact of the additional kernel jitter introduced by the different CONFIG_HZ)
Results (linux-unstable 6.8.0-2.2, avg of 10 runs of 5min each):
- CONFIG_HZ=250 : 17415.60 bogo ops/s
- CONFIG_HZ=1000 : 14866.05 bogo ops/s
- CONFIG_HZ=1000+nohz_full : 18505.52 bogo ops/s
Results confirm the theory about the performance drop of CPU-intensive workloads (-~14%), but also confirms the benefit of NO_HZ_FULL (+~6%) compared to the current HZ settings.
Let's also keep in mind that this is the worst case scenario and a very specific one, where only HPC / scientific applications can be affected, and even in this case we can always compensate and actually get a better level performance exploiting the nohz_full capability.
[Fix]
Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now).
[Regression potential]
As already covered we may experience performance regressions in CPU-intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. |
|
| 2024-01-28 00:33:21 |
Doug Smythies |
bug |
|
|
added subscriber Doug Smythies |
| 2024-01-29 11:32:49 |
Lastique |
bug |
|
|
added subscriber Lastique |
| 2024-01-30 04:57:35 |
Vladislav Shumkin |
bug |
|
|
added subscriber Vladislav Shumkin |
| 2024-01-31 10:59:37 |
pauldoo |
bug |
|
|
added subscriber pauldoo |
| 2024-02-03 02:26:41 |
Chris Guiver |
bug |
|
|
added subscriber Chris Guiver |
| 2024-02-08 09:51:53 |
Nicolas Dichtel |
bug |
|
|
added subscriber Nicolas Dichtel |
| 2024-03-07 09:55:59 |
Andrea Righi |
linux (Ubuntu Noble): status |
New |
Fix Committed |
|
| 2024-03-28 08:15:32 |
Launchpad Janitor |
linux (Ubuntu Noble): status |
Fix Committed |
Fix Released |
|
| 2024-05-14 19:46:53 |
Ubuntu Kernel Bot |
tags |
|
kernel-spammed-jammy-linux-nvidia-6.8-v2 verification-needed-jammy-linux-nvidia-6.8 |
|
