Preempt_rt kernel enters idle loop even when there are processes ready

Added config file.

Since this occurs with or without NO_HZ_FULL configured in, it actually sounds more like a CPU isolation failure than a NO_HZ_FULL failure. It rather sounds as though the scheduler isn't fully honoring the 'isolcpus=1' boot command line option, and is causing the idle task to run on the 'isolated' CPU since no other tasks are 'scheduled' there. But this is just conjecture until we can do some actual testing.

Magnus,

I hope the testcase would be similar if this happens on our non-RT kernel as well.. (i.e. linux-lng)..
I don't thing there should be any difference here with or with RT support.. as isolcpus should work
for you busy task..

Can you give it a try on non-RT kernel? That will enable us to separate the offending piece of code..

Magnus,

Can you provide us your test scripts that you run... Also what hardware you used for your tests? Exynos? and maybe any other detail that you think is important for reproducing this issue... Probably all the minute steps to follow ?

Viresh,

Here is the latency benchmark that was used.

Here is the configuration.

I am running on an Arndale board at 1.4 GHz. Boot cmdline options in previous posting. Once up and running, I load core 0 with:

find /usr -type f -exec scp -q \{\} mkarlsso@10.0.0.1:/dev/null \; &

But you can probably skip this as the number will be bad even without it.

Then I launch the latency benchmark with ./latency. A max latency around 15 us is expected from 3.6 and 3.7. ignore the first run though. On 3.10 I get nearly 5 times as much.

I am using linux-linaro-lng-preempt-rt-3.10.6-2013.08.

BTW, this might not have anything to do with preempt_rt as I can get bad numbers with the regular preempt in linux-linaro-lng-3.10.6-2013.08 too.

Let me know if you need more information.

/Magnus

Forgot to add that you have to launch the latency application on core 1 by using e.g., cpuset file system or taskset. Or you could add this into the benchmark itself:

  cpu_set_t cpu_set;

  CPU_ZERO(&cpu_set);
  CPU_SET(1, &cpu_set);
  if (sched_setaffinity(0, sizeof(cpu_set_t), &cpu_set) == -1)
  {
      perror("sched_setaffinity");
  }

Magnus Karlsson <email address hidden> writes:

> Here is the configuration.
>
> I am running on an Arndale board at 1.4 GHz. Boot cmdline options in
> previous posting. Once up and running, I load core 0 with:
>
> find /usr -type f -exec scp -q \{\} mkarlsso@10.0.0.1:/dev/null \; &
>
> But you can probably skip this as the number will be bad even without
> it.
>
> Then I launch the latency benchmark with ./latency. A max latency around
> 15 us is expected from 3.6 and 3.7. ignore the first run though. On 3.10
> I get nearly 5 times as much.
>
> I am using linux-linaro-lng-preempt-rt-3.10.6-2013.08.
>
> BTW, this might not have anything to do with preempt_rt as I can get bad
> numbers with the regular preempt in linux-linaro-lng-3.10.6-2013.08 too.

Can you reproduce against mainline?

Kevin
<>

I can reproduce this in mainline. Tried tip of linux-3.10.y (3.10.13) from linux.org. Problem is still there. I have attached the function trace. Search for the "Max latency" marker that I print out when the latency in the loop is above 500 us in the run. If you go back in time in the trace, you can see that the "latency" app is context switched in from the idle thread on core 1.

Note that I am running with the standard preempt option (not the preempt_full patch or the server version option).

Here is the config.

The latency bechmark is being is added to CI, pending B&B supporting out of tree kernel module builds.

I cannot reproduce this in latest kernel, so I am going to close it. Do not know exactly what fixed it.

Magnus,

I tried the cpuset code you talked about earlier, but with that my task isn't actually attached to cpu 1. I would be attaching my updated code as well..

I traced it using ps -aF with running latency in background.. And on a number of occasions it went to cpu 0.

Then I used `taskset -c 1 ./latency` and it worked without any issues and so task is sticking to cpu1..

The numbers I am getting on 3.10.13, with top most commit:
cff43fc Linux 3.10.13

are:

maxrange=16301 cycles = ~12 us.

And so I am unable to reproduce bug with the kernel where Magnus has reported it.