NO_HZ does not work for RT kernels

Do we really need the (full) PREEMPT_RT patch set? AFAIK it is very disruptive to the kernel design with regards to ISR's and locks.
Would't core isolation (we are only considering multicore processors) and purpose-optimized interrupt handling and user process signalling (from special purpose ISR to an application which does not use system calls indiscriminately) be enough for the relevant use cases? Possibly this would be of more general usefulness than full PREEMPT_RT. And much more robust considering future modifications to the kernel and all drivers.

We can discuss f2f at the upcoming Linaro Connect.

I agree ultimately, but right now under load the RT patch set makes an
enormous difference to the latency, so we would need to get all code to the
point it was in no_hz cores as appropriate.

If you can get the CPU load down, then the low latency desktop
configuration performs almost as well as RT, and is part of the main stream
kernel, Gary has a lot to say about this in his presentation at connect.

On 27 February 2014 14:48, Ola Liljedahl <email address hidden> wrote:

> Do we really need the (full) PREEMPT_RT patch set? AFAIK it is very
> disruptive to the kernel design with regards to ISR's and locks.
> Would't core isolation (we are only considering multicore processors) and
> purpose-optimized interrupt handling and user process signalling (from
> special purpose ISR to an application which does not use system calls
> indiscriminately) be enough for the relevant use cases? Possibly this would
> be of more general usefulness than full PREEMPT_RT. And much more robust
> considering future modifications to the kernel and all drivers.
>
> We can discuss f2f at the upcoming Linaro Connect.
>
> --
> You received this bug notification because you are a member of Linaro
> Networking Group, which is subscribed to linaro-networking.
> Matching subscriptions: LNG all
> https://bugs.launchpad.net/bugs/1285772
>
> Title:
> NO_HZ does not work for RT kernels
>
> Status in Linaro networking Group:
> Confirmed
>
> Bug description:
> https://validation.linaro.org/dashboard/image-charts/LNG-NO_HZ
>
> Although recent patches have fix NO_HZ for linux-lng and the
> preemption build, the RT build fails with NO_HZ.
>
> This may be ok, if you have many cores running no_hz then you would
> have less need for the cores running the kernel to have the best
> possible latency ?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-networking/+bug/1285772/+subscriptions
>

Wireless basestation applications need PREEMPT_RT_FULL, irrespective of NO_HZ_FULL, in order to meet e.g., 1ms LTE physical layer subframe requirement specifications. They can use NO_HZ_FULL iff enough cores are available, which is not a valid assumption given the variety of available solutions. For more details, see e.g., this paper:

https://amplab.cs.berkeley.edu/wp-content/uploads/2012/08/cloudIQ.pdf

I know. But the use case and requirements have not been made explicit in
LNG. E.g. how many interrupts and context switches will occur per 1ms TTI
and how much of cycle budget can be used.
There are also other potential approaches that don't involve the kernel
doing context switches, e.g. with user space threads, if PREEMPT_RT doesn't
cut it.
Den 4 mar 2014 06:45 skrev "Kim Phillips" <email address hidden>:

> Wireless basestation applications need PREEMPT_RT_FULL, irrespective of
> NO_HZ_FULL, in order to meet e.g., 1ms LTE physical layer subframe
> requirement specifications. They can use NO_HZ_FULL iff enough cores are
> available, which is not a valid assumption given the variety of
> available solutions. For more details, see e.g., this paper:
>
> https://amplab.cs.berkeley.edu/wp-content/uploads/2012/08/cloudIQ.pdf
>
> --
> You received this bug notification because you are a member of Linaro
> Networking Group, which is subscribed to linaro-networking.
> Matching subscriptions: LNG all
> https://bugs.launchpad.net/bugs/1285772
>
> Title:
> NO_HZ does not work for RT kernels
>
> Status in Linaro networking Group:
> Confirmed
>
> Bug description:
> https://validation.linaro.org/dashboard/image-charts/LNG-NO_HZ
>
> Although recent patches have fix NO_HZ for linux-lng and the
> preemption build, the RT build fails with NO_HZ.
>
> This may be ok, if you have many cores running no_hz then you would
> have less need for the cores running the kernel to have the best
> possible latency ?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-networking/+bug/1285772/+subscriptions
>

> I know. But the use case and requirements have not been made explicit in LNG. E.g. how many interrupts and context switches will occur per 1ms TTI and how much of cycle budget can be used.

Basestations typically rely on 30-50usec max. latency, which is currently being measured in LNG CI tests (but haven't been achieved yet on arndale). I've communicated this within LNG for quite some time now, including on the linaro-networking ML.

> There are also other potential approaches that don't involve the kernel doing context switches, e.g. with user space threads, if PREEMPT_RT doesn't cut it.

sure, but we're trying to make PREEMPT_RT "cut it" here so things like user space threads won't be necessary.

I would prefer PREEMPT_RT not to be necessary as it is very intrusive to
the kernel design and most likely will always require extra patches and
verification. I.e. a never ending project of getting PREEMPT_RT support
working (building, running with the required characteristics). Get the
standard kernel good enough and solve the remaining problems in user space.
Two smaller problem spaces which can be worked on separately and which can
have standard solutions and not exist as separate patch sets.

On Tue, Mar 4, 2014 at 8:07 AM, Kim Phillips <email address hidden>wrote:

> > I know. But the use case and requirements have not been made explicit
> in LNG. E.g. how many interrupts and context switches will occur per 1ms
> TTI and how much of cycle budget can be used.
>
> Basestations typically rely on 30-50usec max. latency, which is
> currently being measured in LNG CI tests (but haven't been achieved yet
> on arndale). I've communicated this within LNG for quite some time now,
> including on the linaro-networking ML.
>
> > There are also other potential approaches that don't involve the
> kernel doing context switches, e.g. with user space threads, if
> PREEMPT_RT doesn't cut it.
>
> sure, but we're trying to make PREEMPT_RT "cut it" here so things like
> user space threads won't be necessary.
>
> --
> You received this bug notification because you are a member of Linaro
> Networking Group, which is subscribed to linaro-networking.
> Matching subscriptions: LNG all
> https://bugs.launchpad.net/bugs/1285772
>
> Title:
> NO_HZ does not work for RT kernels
>
> Status in Linaro networking Group:
> Confirmed
>
> Bug description:
> https://validation.linaro.org/dashboard/image-charts/LNG-NO_HZ
>
> Although recent patches have fix NO_HZ for linux-lng and the
> preemption build, the RT build fails with NO_HZ.
>
> This may be ok, if you have many cores running no_hz then you would
> have less need for the cores running the kernel to have the best
> possible latency ?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-networking/+bug/1285772/+subscriptions
>

As I said, "getting the standard kernel good enough" is currently == patching it with PREEMPT_RT, at least for basestation users.

No we don't know whether PREEMPT_RT is good enough (50-60 us interrupt
latency under load on A15 according to Gary R. at LCA14) for the actual
(base station) use cases considered by LNG (for NSN and Ericsson). Why?
Because no actual requirements have been specified. Depending on the
application design, multiple interrupt-based context switches per TTI with
60us latency might not be good enough, the interrupt handling and context
switch overhead will also consume cycles that might be needed by the
application. I had previously heard a requirement (or a wish) for interrupt
latency in the 5-10us range (this also limits the overhead).

On Tue, Mar 11, 2014 at 12:17 AM, Kim Phillips <email address hidden>wrote:

> As I said, "getting the standard kernel good enough" is currently ==
> patching it with PREEMPT_RT, at least for basestation users.
>
> --
> You received this bug notification because you are a member of Linaro
> Networking Group, which is subscribed to linaro-networking.
> Matching subscriptions: LNG all
> https://bugs.launchpad.net/bugs/1285772
>
> Title:
> NO_HZ does not work for RT kernels
>
> Status in Linaro networking Group:
> Confirmed
>
> Bug description:
> https://validation.linaro.org/dashboard/image-charts/LNG-NO_HZ
>
> Although recent patches have fix NO_HZ for linux-lng and the
> preemption build, the RT build fails with NO_HZ.
>
> This may be ok, if you have many cores running no_hz then you would
> have less need for the cores running the kernel to have the best
> possible latency ?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-networking/+bug/1285772/+subscriptions
>

I think there are dual efforts in progress here. For ODP purposes, the
'full RT' kernel probably should not be required even if/when we get
NO_HZ_FULL/cpu isolation working there. However for those users who want
to work within the kernel stacks rather than from isolated cores, full RT
control of scheduling latencies may be a requirement. So we will have
efforts to optimize latencies on the RT kernel as well as providing support
for operation on isolated cores with minimal interference from the
kernel... RT or otherwise.

As per interrupt latency, that is mostly a function of the drivers selected
in a given install since the generic parts of the kernel already have
interrupts disabled for the briefest times possible... and interrupt
contexts operate independently of the scheduler itself. Where 'threaded
ISRs' are configured, interrupts not explicitly prevented from being
threaded will of course have most of their operation shifted into
schedulable contexts - but in cases where that induces unacceptable
latencies specific interrupts may be forced to remain unthreaded to
minimize any contention for CPU time.

On Mon, Mar 10, 2014 at 6:37 PM, Ola Liljedahl <email address hidden>wrote:

> No we don't know whether PREEMPT_RT is good enough (50-60 us interrupt
> latency under load on A15 according to Gary R. at LCA14) for the actual
> (base station) use cases considered by LNG (for NSN and Ericsson). Why?
> Because no actual requirements have been specified. Depending on the
> application design, multiple interrupt-based context switches per TTI with
> 60us latency might not be good enough, the interrupt handling and context
> switch overhead will also consume cycles that might be needed by the
> application. I had previously heard a requirement (or a wish) for interrupt
> latency in the 5-10us range (this also limits the overhead).
>
>
> On Tue, Mar 11, 2014 at 12:17 AM, Kim Phillips <<email address hidden>
> >wrote:
>
> > As I said, "getting the standard kernel good enough" is currently ==
> > patching it with PREEMPT_RT, at least for basestation users.
> >
> > --
> > You received this bug notification because you are a member of Linaro
> > Networking Group, which is subscribed to linaro-networking.
> > Matching subscriptions: LNG all
> > https://bugs.launchpad.net/bugs/1285772
> >
> > Title:
> > NO_HZ does not work for RT kernels
> >
> > Status in Linaro networking Group:
> > Confirmed
> >
> > Bug description:
> > https://validation.linaro.org/dashboard/image-charts/LNG-NO_HZ
> >
> > Although recent patches have fix NO_HZ for linux-lng and the
> > preemption build, the RT build fails with NO_HZ.
> >
> > This may be ok, if you have many cores running no_hz then you would
> > have less need for the cores running the kernel to have the best
> > possible latency ?
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/linaro-networking/+bug/1285772/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed to linaro-
> networking.
> Matching subscriptions: LNG all, lng-bugs
> https://bugs.launchpad.net/bugs/1285772
>
> Title:
> NO_HZ does not work for RT kernel...

"However for those users who want to work within the kernel stacks rather
than from isolated cores, full RT control of scheduling latencies may be a
requirement. "
Who are those users and what are their use cases? Are they members of LNG
or potential customers of LNG members? Are we attempting to solve a real or
a hypothetical problem?

I am sure this is an interesting problem. But I am not sure this is the
most important problem that we at the moment. Directives from LNG-SC seems
not to be very clear, at least not in the past. There was a discussion on
goals and priorities in the LNG-SC at LCA14. I don't remember the details
though.

On Tue, Mar 11, 2014 at 3:27 PM, Gary S. Robertson <
<email address hidden>> wrote:

> I think there are dual efforts in progress here. For ODP purposes, the
> 'full RT' kernel probably should not be required even if/when we get
> NO_HZ_FULL/cpu isolation working there. However for those users who want
> to work within the kernel stacks rather than from isolated cores, full RT
> control of scheduling latencies may be a requirement. So we will have
> efforts to optimize latencies on the RT kernel as well as providing support
> for operation on isolated cores with minimal interference from the
> kernel... RT or otherwise.
>
> As per interrupt latency, that is mostly a function of the drivers selected
> in a given install since the generic parts of the kernel already have
> interrupts disabled for the briefest times possible... and interrupt
> contexts operate independently of the scheduler itself. Where 'threaded
> ISRs' are configured, interrupts not explicitly prevented from being
> threaded will of course have most of their operation shifted into
> schedulable contexts - but in cases where that induces unacceptable
> latencies specific interrupts may be forced to remain unthreaded to
> minimize any contention for CPU time.
>
>
>
> On Mon, Mar 10, 2014 at 6:37 PM, Ola Liljedahl <<email address hidden>
> >wrote:
>
> > No we don't know whether PREEMPT_RT is good enough (50-60 us interrupt
> > latency under load on A15 according to Gary R. at LCA14) for the actual
> > (base station) use cases considered by LNG (for NSN and Ericsson). Why?
> > Because no actual requirements have been specified. Depending on the
> > application design, multiple interrupt-based context switches per TTI
> with
> > 60us latency might not be good enough, the interrupt handling and context
> > switch overhead will also consume cycles that might be needed by the
> > application. I had previously heard a requirement (or a wish) for
> interrupt
> > latency in the 5-10us range (this also limits the overhead).
> >
> >
> > On Tue, Mar 11, 2014 at 12:17 AM, Kim Phillips <<email address hidden>
> > >wrote:
> >
> > > As I said, "getting the standard kernel good enough" is currently ==
> > > patching it with PREEMPT_RT, at least for basestation users.
> > >
> > > --
> > > You received this bug notification because you are a member of Linaro
> > > Networking Group, which is subscribed to linaro-networking.
> > > Matching subscriptions: LNG all
> > > https://bugs.launchpad.net/bugs/1285772
> > >
> > > Title:
> > > NO_HZ does not work for...

On Tue, 11 Mar 2014 20:45:57 -0000
Ola Liljedahl <email address hidden> wrote:

> "However for those users who want to work within the kernel stacks rather
> than from isolated cores, full RT control of scheduling latencies may be a
> requirement. "

Basestation systems would still use PREEMPT_RT_FULL whether or not a
dataplane were running simultaneously on other NO_HZ_FULL cores.

> Who are those users and what are their use cases?

At least Freescale, and IIRC, TI, their customers, and their
basestations.

> Are they members of LNG or potential customers of LNG members?

both.

> Are we attempting to solve a real or a hypothetical problem?

this is a real problem.

> I am sure this is an interesting problem. But I am not sure this is the
> most important problem that we at the moment. Directives from LNG-SC seems
> not to be very clear, at least not in the past. There was a discussion on
> goals and priorities in the LNG-SC at LCA14. I don't remember the details
> though.

pray tell. Or rather, have this conversation with the LNG-SC directly,
and not in the context of a public bug report.

Kim

I don't think these goals are mutually exclusive. There are legitimate use cases for both approaches and we need to support and enhance both capabilities.

Without doubt the highest possible performance can be had by running user-space code which interacts directly with the hardware and has minimal kernel interaction or interference. However this mandates that the application writer must work in a much less abstracted environment and maintain a larger body of non-portable code. That said, for those use cases where maximum performance is the overriding consideration, we need to provide an environment in which isolated user-space applications can run without unnecessary interference from the kernel.

For applications which can afford some kernel overhead and for which code portability is of higher importance relative to raw performance considerations, the kernel-assisted approach may be preferable. In such cases we need to provide the best and most stable latency behaviour achievable from the kernel so as to minimize the performance penalties incurred in this more generic environment.

Gary,

I couldn't find what problems exactly are we getting with NO_HZ + RT. Can you please mention all those here? I will see if I have the right amount of knowledge to crack it :)

Viresh,

I haven't run any CPU isolation tests against RT with the latest patch
series - I expect that to be forthcoming shortly. Last time I looked,
though, the RT kernels were just returning immediately with CPU isolation
times of zero seconds. AFAIK there were no problems with hangups, etc...
we just weren't getting any isolation. I haven't had a chance to look into
this any deeper thus far.

On Thu, Apr 10, 2014 at 12:51 AM, viresh kumar <email address hidden>wrote:

> Gary,
>
> I couldn't find what problems exactly are we getting with NO_HZ + RT.
> Can you please mention all those here? I will see if I have the right
> amount of knowledge to crack it :)
>
> --
> You received this bug notification because you are subscribed to linaro-
> networking.
> Matching subscriptions: LNG all, lng-bugs
> https://bugs.launchpad.net/bugs/1285772
>
> Title:
> NO_HZ does not work for RT kernels
>
> Status in Linaro networking Group:
> Confirmed
>
> Bug description:
> https://validation.linaro.org/dashboard/image-charts/LNG-NO_HZ
>
> Although recent patches have fix NO_HZ for linux-lng and the
> preemption build, the RT build fails with NO_HZ.
>
> This may be ok, if you have many cores running no_hz then you would
> have less need for the cores running the kernel to have the best
> possible latency ?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-networking/+bug/1285772/+subscriptions
>

The last discussions agreed that we need to have no_hz stable for non RT before digging deeper. moving to low priority.

This bug is being closed and replaced with a card - LNG-489.

It was mis-classified as a bug since the code involved is still under development and cannot yet be expected to be stable or functional.