missing trace on function "cpu_hotplug_done"

[Naresh Kamboju]

while testing cpu hotplug latency test i have found.

FAILED to set "cpu_hotplug_done"
#echo cpu_hotplug_done > /sys/kernel/debug/tracing/set_ftrace_filter

"cpu_hotplug_done" function is missing.

root@android:/data/hotplug # cat /sys/kernel/debug/tracing/available_filter_functions | grep cpu_hotplug
cpu_hotplug_begin
cpu_hotplug_disable_before_freeze
cpu_hotplug_enable_after_thaw
cpu_hotplug_pm_callback
root@android:/data/hotplug #

Linux Kernel:
root@android:/data/hotplug # uname -a
Linux localhost 3.7.0-rc6-00620-g8890900 #1 SMP Wed Dec 5 07:08:22 UTC 2012 armv7l GNU/Linux
root@android:/data/hotplug #

Android Build:
https://android-build.linaro.org/builds/~linaro-android/vexpress-jb-gcc47-armlt-tracking-open/#build=121

[Tixy (Jon Medhurst) (tixy)]

I assume the compiler has inlined the function cpu_hotplug_done, it looks like it from a disassembly of the kernel.

[Chander Mohan Kashyap (chander-kashyap)]

I have compiled the kernel with linaro-gcc-4.5-arm-linux-gnueabi-32bit toolcahin and cpu_hotplug_done is not inline.

[Naresh Kamboju (naresh-kamboju)]

I have recompiled kernel with Chander Mohan Kashyap's patch

cpu_hotplug_begin and cpu_hotplug_done are used in script to
calculate cpu_hotplug latency. So it is necessary these funtion
do not get inlined by compiler.

Signed-off-by: Chander Kashyap <email address hidden>
---
 kernel/cpu.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/kernel/cpu.c b/kernel/cpu.c
index e909b41..bb54ca4 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -113,7 +113,7 @@ EXPORT_SYMBOL_GPL(put_online_cpus);
  * get_online_cpus() not an api which is called all that often.
  *
  */
-static void cpu_hotplug_begin(void)
+static noinline void cpu_hotplug_begin(void)
 {
        cpu_hotplug.active_writer = current;

@@ -127,15 +127,15 @@ static void cpu_hotplug_begin(void)
        }
 }

-static void cpu_hotplug_done(void)
+static noinline void cpu_hotplug_done(void)
 {
        cpu_hotplug.active_writer = NULL;
        mutex_unlock(&cpu_hotplug.lock);
 }

 #else /* #if CONFIG_HOTPLUG_CPU */
-static void cpu_hotplug_begin(void) {}
-static void cpu_hotplug_done(void) {}
+static noinline void cpu_hotplug_begin(void) {}
+static noinline void cpu_hotplug_done(void) {}
 #endif /* #else #if CONFIG_HOTPLUG_CPU */

 /* Need to know about CPUs going up/down? */

following bug is produced
root@android:/data/hotplug # sh cpu_hotplug_latency.sh
**** TEST LOADED CPU*****
**** TEST 20 *****

[ 253.182342] CPU0 packing on CPU-1
[ 253.192348] CPU1 packing on CPU-1
[ 253.202322] CPU2 packing on CPU-1
[ 253.212333] CPU3 packing on CPU-1
[ 253.222273] CPU4 packing on CPU-1
[ 253.232428] CPU0 packing on CPU2
[ 253.242120] CPU2 packing on CPU2
[ 253.251805] CPU3 packing on CPU4
[ 253.261484] CPU4 packing on CPU2
[ 253.273026] CPU1: shutdown
Disable

[Naresh Kamboju (naresh-kamboju)]

Linux Kernel:
Linux version 3.7.0-rc8-00606-g59b6ef4-dirty (naresh@Linaro) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #1 SMP Tue Dec 11 13:31:26 IST 2012

After applying above patch we can see "cpu_hotplug_done" and "cpu_hotplug_begin" functions in below file
root@android:/ #cat /sys/kernel/debug/tracing/available_filter_functions | grep cpu_hot
cpu_hotplug_done
cpu_hotplug_begin
cpu_hotplug_disable_before_freeze
cpu_hotplug_enable_after_thaw
cpu_hotplug_pm_callback
root@android:/ #

but when i executed the script to start cpu_hotplug_latency test . board produce below output and hugs. I did not see any kernel crash dump.
root@android:/data/hotplug # sh cpu_hotplug_latency.sh
**** TEST LOADED CPU*****
**** TEST 20 *****

[ 253.182342] CPU0 packing on CPU-1
[ 253.192348] CPU1 packing on CPU-1
[ 253.202322] CPU2 packing on CPU-1
[ 253.212333] CPU3 packing on CPU-1
[ 253.222273] CPU4 packing on CPU-1
[ 253.232428] CPU0 packing on CPU2
[ 253.242120] CPU2 packing on CPU2
[ 253.251805] CPU3 packing on CPU4
[ 253.261484] CPU4 packing on CPU2
[ 253.273026] CPU1: shutdown
Disable

[Naresh Kamboju (naresh-kamboju)]

[Chander Kashyap] wrote
cpu_hot_plug function is getting inline with new toolchain.
By exporting it we can avoid it.

This is needed to run cpu_hotplug_latency.sh script.

Signed-off-by: Chander Kashyap <email address hidden>
---
 kernel/cpu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/cpu.c b/kernel/cpu.c
index e909b41..9d1f531 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -138,6 +138,7 @@ static void cpu_hotplug_begin(void) {}
 static void cpu_hotplug_done(void) {}
 #endif /* #else #if CONFIG_HOTPLUG_CPU */

+EXPORT_SYMBOL(cpu_hotplug_done);
 /* Need to know about CPUs going up/down? */
 int __ref register_cpu_notifier(struct notifier_block *nb)
 {

[naresh kamboju] wrote:
I have added this patch and following are the observations.
root@android:/ #cat /sys/kernel/debug/tracing/available_filter_functions | grep cpu_hot
cpu_hotplug_done
cpu_hotplug_begin
cpu_hotplug_disable_before_freeze
cpu_hotplug_enable_after_thaw
cpu_hotplug_pm_callback

test produced below output and hungs
root@android:/data/hotplug # sh cpu_hotplug_latency.sh
**** TEST LOADED CPU*****
**** TEST 20 *****

[ 254.715471] CPU0 packing on CPU-1
[ 254.725431] CPU1 packing on CPU-1
[ 254.735375] CPU2 packing on CPU-1
[ 254.745326] CPU3 packing on CPU-1
[ 254.755260] CPU4 packing on CPU-1
[ 254.765414] CPU0 packing on CPU2
[ 254.775100] CPU2 packing on CPU2
[ 254.784789] CPU3 packing on CPU4
[ 254.794471] CPU4 packing on CPU2
[ 254.805993] CPU1: shutdown

[Naresh Kamboju (naresh-kamboju)]

I have tested attached cpu_hotplug_latency.sh file on android successful.
test execution required following kernel configurations changes.
  --->vexpress_bL_mp_defconfig is the base kernel config
  --->Cpu idle has been disabled.
  --->Ftrace has been Enabled
  --->Ftrace Dynamic has been enabled
  --->Ftrace Graph has been enabled

We do not have above config as default so i have modified and executed test. Do you have plan to make these config as default to vexpress_bL_mp_defconfig? Which would be helps me in testing release build rather than local builds.

Test files.
1. cpu_hotplug_latency.sh
2. test_load.txt test output log.
3. test execution log https://pastebin.linaro.org/1450/
4. vexpress_bL_mp_defconfig (modified)

[Naresh Kamboju (naresh-kamboju)]

[Amit Kucheria] wrote
Daniel is currently investigating a bug with hotplug/RCU in mainline
(https://bugs.launchpad.net/linaro-power-kernel/+bug/1102347).

We shouldn't mark this as done until we are able to reliably hotplug
_with_ cpuidle enabled.

[Naresh Kamboju (naresh-kamboju)]

since bug (https://bugs.launchpad.net/linaro-power-kernel/+bug/1102347). talks about same issue
I have update bug 1102347 "exynos : rcu stall with 3.8-rc2 " with cpu_hotplug_latency test done on 13.03 kernel.

If you have fix for this issue please point me that kernel, I will test and confirm.
Thanks
Naresh kamboju

[Naresh Kamboju (naresh-kamboju)]

this bug issue is same as reported in bug 1102347

cpu_hotplug_latency.sh use to crash kernel after applying patch [1].
I have tested stall.sh and cpu_hotplug_latency.sh test files and confirmed that no more kernel crash found on this current vexpress-mp kernel.

[tixy wrote]
The linaro kernel trees now have the files mcpm.c, dcscb.c and tc2_pm.c all excluded from tracing. This wasn't sufficient to fix the issue however after further debugging I found that adding notrace to cpu_init() prevents any lockups or crashes. I tested this by toggling current_tracer between 'nop' and 'function' in an endless loop whilst running the bbench+audio workbench test.

The patch has been posted to the linux-arm-kernel list:
http://lists.infradead.org/pipermail/linux-arm-kernel/2013-April/163983.html

Fixed in linux-linaro tree by: https://git.linaro.org/gitweb?p=kernel/linux-linaro-tracking.git;a=commit;h=3a58649068df52162ef63dd0f1c98d231f667b05

Build
http://snapshots.linaro.org/android/~linaro-android/vexpress-linaro-mp/266/

Linux kernel version:
Linux localhost 3.9.0-rc7-00198-g03d27d8 #1 SMP Fri Apr 19 06:05:40 UTC 2013 armv7l GNU/Linux

I have attached test log to this bug: cpu_hotplug_latency_test_bug_1068595.log

[Naresh Kamboju (naresh-kamboju)]

with the above comment the issues fixed and released.