panic occurred while running powertop with panda JB tracking-blob build

[Amit Khare]

observed with panda android jelly bean tracking-blob build
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=28

Panic occurs when powertop is run. Panic_log file is attached.

Also observed with below builds:

https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-stable-blob/#build=40
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-stable-blob/#build=44
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-stable-blob/#build=49
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-stable-blob/#build=69
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-stable-blob/#build=75
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-stable-blob/#build=82
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=106
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=161
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=168
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=174
https://android-build.linaro.org/builds/~linaro-android-archive/panda-jb-gcc47-tilt-tracking-blob/#build=186
https://android-build.linaro.org/builds/~linaro-android-archive/panda-jb-gcc47-tilt-tracking-blob/#build=196

[Amit Kucheria (amitk)]

That's a spectacular crash. Is this the latest version of powertop Rajagopal?

/Amit

[Rajagopal Venkat (rajagopalvenkat)]

This build referes to old version of powertop(as per manifest.xml at https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=28).

Suggest to use new version of powertop integrated on 27th August 2012.

[Amit Khare (amit-khare)]

Also Observed with panda JB android build
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=40

[ 89.013458] CPU1: stopping
[ 89.016326] Backtrace:
[ 89.018920] [<c0012a04>] (dump_backtrace+0x0/0x114) from [<c04ff950>] (dump_stack+0x20/0x24)
[ 89.027832] r6:00000000 r5:fa240100 r4:00000001 r3:c0771ccc
[ 89.033843] [<c04ff930>] (dump_stack+0x0/0x24) from [<c00141ec>] (handle_IPI+0x10c/0x1e0)
[ 89.042480] [<c00140e0>] (handle_IPI+0x0/0x1e0) from [<c0008678>] (gic_handle_irq+0x68/0x70)
[ 89.051391] r7:e609ff9c r6:c0747900 r5:fa240100 r4:e609ff68
[ 89.057403] [<c0008610>] (gic_handle_irq+0x0/0x70) from [<c000e080>] (__irq_svc+0x40/0x70)
[ 89.066131] Exception stack(0xe609ff68 to 0xe609ffb0)
[ 89.071441] ff60: c07a33a8 aaa0aaaa f0004807 fe404000 e609e000 c050d610
[ 89.080078] ff80: c07a31f0 c07a3204 8000406a 411fc092 00000000 e609ffbc e609ff9c e609ffb0
[ 89.088714] ffa0: c0025384 c000f7a8 a0010113 ffffffff
[ 89.094055] r6:ffffffff r5:a0010113 r4:c000f7a8 r3:c0025384
[ 89.100067] [<c000f774>] (default_idle+0x0/0x3c) from [<c000fb94>] (cpu_idle+0xbc/0x114)
[ 89.108612] [<c000fad8>] (cpu_idle+0x0/0x114) from [<c04fc754>] (secondary_start_kernel+0x130/0x154)
[ 89.118225] r5:00000001 r4:00000001
[ 89.122039] [<c04fc624>] (secondary_start_kernel+0x0/0x154) from [<804fc094>] (0x804fc094)
[ 89.130767] r6:10c0387d r5:00000015 r4:a608c06a r3:c04fc07c
[ 89.136749] panic occurred, switching back to text console

[Rajagopal Venkat (rajagopalvenkat)]

Hi Amit Khare / Botao Sun,

Could you please run attached test on Panda board to isolate the problem?

Steps on target -
$ gcc vm_user.c -o vm_user
$ echo 8 > /proc/sys/kernel/printk
$ ./vm_user 10000

Thanks.

[Jassi Brar (jassisinghbrar)]

Amit Khare : The build-28 and build-40 crash logs show different bugs ?

Rajagopal : Just to isolate the problem, I ran 'vm_user 10000' without crash, twice over plain tilt-3.4 kernel which is supposed to form the basis of the android kernel(?) It seems some "androidization" patch introduced the issue. I'll try now with the build-40 kernel to zero in.

[warmcat (andy-warmcat)]

Jassi the Androidization series is already in tilt-3.4 courtesy of llct. I guess we mean additional Android-only patches added by Android guys.

Also vm_user may not provoke the problem. Original report got the problem by running powertop, not vm_user.

Did we update to latest powertop as suggested in #2 and are we still seeing the same problem?

Has anyone been able to provoke the problem with vm_user test from Rajagopal?

[Amit Khare (amit-khare)]

observed with panda android JB tracking-blob build
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=44

on running powertop, board freezes. Log output is same as build #28 with which this bug was raised.

@rajagopal - I tried the commands on serial console.

root@android:/ # gcc vm_user.c -o vm_user
/system/bin/sh: gcc: not found
root@android:/ # echo 8 > /proc/sys/kernel/printk
root@android:/ # ./vm_user 10000
/system/bin/sh: ./vm_user: not found

[Amit Khare (amit-khare)]

Also Observed with panda JB android build
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=49

[Zach Pfeffer (pfefferz)]

Any update on this Rajagopal? Moving to 12.10.

[Rajagopal Venkat (rajagopalvenkat)]

This crash is reported on Panda and issue seems to do with Perf.

Unfortunately, I don't have Panda board at this moment to debug. Though I have requested for one.

[Amit Khare (amit-khare)]

Also Observed with panda JB android tracking-blob build
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=69

[ 1262.578002] Process powertop (pid: 2093, stack limit = 0xe34b62f8)
[ 1262.584503] Stack: (0xe34b7d80 to 0xe34b8000)
[ 1262.589111] 7d80: e34b7e0c 00000000 c07a0a04 e3715880 0000065f fe7f1000 c00d0
[ 1262.597747] 7da0: 00000800 00000000 00000000 00000000 e4bbb200 00000000 e34b8
[ 1262.606384] 7dc0: c00e6f78 c00e59c4 e34b7e44 e34b7dd8 c00d7a88 c00e6f68 e60d8
[ 1262.615020] 7de0: fe7f0000 00001000 e34b7e04 c0747ea8 00000001 00000001 00000
[ 1262.623657] 7e00: 00000001 00000000 e3b1c400 e4bbb200 00000000 00000001 00004
[ 1262.632293] 7e20: c07dc490 c0747efc 00000000 e3b1c400 c07dc5f0 c0774510 e34b8
[ 1262.640930] 7e40: c00d7ecc c00d75d4 e34b7e7c e34b7e58 c00ab680 c00d7ebc 00000
[ 1262.649536] 7e60: e3b1c400 00000000 e3b1c400 e34b7f28 e34b7e9c e34b7e80 c00b8
[ 1262.658172] 7e80: c00b0eec c0776924 e3b1c400 c0776814 e34b7ec4 e34b7ea0 c00b8
[ 1262.666809] 7ea0: 00000000 c0776924 00000001 00000000 00000000 00000000 e34b8
[ 1262.675445] 7ec0: c00b7a44 c00b7728 00000000 00000000 00000000 ffffffff 00000
[ 1262.684082] 7ee0: 00000000 00000000 e34b7fa4 e34b7ef8 c00b8398 c00b77f0 00000
[ 1262.692718] 7f00: 00000000 c0507f64 e6297480 e39dcb40 e39dcb00 00000013 00000
[ 1262.701354] 7f20: 00019b3c 00000000 00000002 00000040 00000037 00000000 00000
[ 1262.709991] 7f40: 00000484 00000000 00000007 00000000 00000301 00000000 00000
[ 1262.718627] 7f60: 00000000 00000000 00000000 00000000 00000000 00000000 00000
[ 1262.727264] 7f80: 00000001 b6f58c91 0000016c c000e6a8 e34b6000 00000000 00008
[ 1262.735870] 7fa0: c000e440 c00b7f64 00000000 00000001 beb18ad8 ffffffff 0000f
[ 1262.744506] 7fc0: 00000000 00000001 b6f58c91 0000016c b97c4940 b6f94d48 00000
[ 1262.753143] 7fe0: beb18ab0 beb18aa0 b6f58f79 b6e66f10 20070110 beb18ad8 00000
[ 1262.877441] Kernel panic - not syncing: Fatal exception
[ 1262.882965] CPU1: stopping
[ 1263.006195] panic occurred, switching back to text console

[Amit Khare (amit-khare)]

Also observed with panda android JB build
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=75

[Amit Khare (amit-khare)]

Also observed with panda android JB build
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=82

[Amit Kucheria (amitk)]

Amit, May I suggest that you and Rajagopal reserve some time next week in Copenhagen to debug this and add it to https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/Copenhagen_2012 under 'Hacking sessions' ?

[Rajagopal Venkat (rajagopalvenkat)]

Thanks Amitk for your suggestion.

This issue is open for quite long due to unavailability of Panda board. As I would receive one Panda board at Connect, we can plan to fix this in hacking sessions.

[Amit Khare (amit-khare)]

Hey Raj, I am in and Since you are getting the panda board at connect i dont need to bring one :-)

[Amit Khare (amit-khare)]

@Zach - For some strange reason, Powertop issue for Panda was not reproducible with any of the builds. I will update this bug if the problem persists and provide more information to Raj.

[Soumya Basak (soumya-basak)]

observed with the panda 4430 jb image
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=100
when enter powertop from adb shell on host the device is go into Hang state & serial console is also stops.

log:

[ 94.826324] Unable to handle kernel paging request at virtual address e66107c0
[ 94.833953] pgd = e1b08000
[ 94.836822] [e66107c0] *pgd=00000000
[ 94.840606] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[ 94.846191] Modules linked in: wl12xx wlcore mac80211 cfg80211 lib80211 btwilink rfcomm bnep bluetooth gator
[ 94.856628] CPU: 0 Not tainted (3.4.0-gc9c93b6 #1)
[ 94.862091] PC is at vmap_page_range_noflush+0xc8/0x19c
[ 94.867584] LR is at map_kernel_range_noflush+0x1c/0x20
[ 94.873107] pc : [<c00e5a38>] lr : [<c00e6f30>] psr: 80070113
[ 94.873107] sp : e1adfd80 ip : e66107c0 fp : e1adfdc4
[ 94.885223] r10: e66107c0 r9 : c074b048 r8 : c07e1040
[ 94.890747] r7 : c0007f98 r6 : fe7f1000 r5 : 00000000 r4 : fe7f0000
[ 94.897644] r3 : e1a82880 r2 : e66107c0 r1 : fe7f0fff r0 : fe7f0000
[ 94.897644] Flags: Nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 94.897644] Control: 10c5387d Table: a1b0804a DAC: 00000015

[Soumya Basak (soumya-basak)]

observed with panda 4460 jb builds
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=100

[Rajagopal Venkat (rajagopalvenkat)]

Powertop crash is re-producible on Android and Ubuntu builds of Panda. Crash is observed due to perf issues and nothing specific to powertop as such.

Any test program which invokes 'sys_perf_event_open' syscall will cause the crash.

Further analysis/debugging is required in perf framework.

[Soumya Basak (soumya-basak)]

observed with panda android jb image builds
https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=115
the device will hang when running powertop.
logcat:

[ 1737.027832] Unable to handle kernel paging request at virtual address e66107c0
[ 1737.035491] pgd = e2ae0000
[ 1737.038360] [e66107c0] *pgd=00000000
[ 1737.042175] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[ 1737.047790] Modules linked in: wl12xx wlcore mac80211 cfg80211 lib80211 btwilink rfcomm bnep bluetooth gator
[ 1737.050872] CPU: 1 Not tainted (3.4.0-gc9c93b6 #1)

[Rajagopal Venkat (rajagopalvenkat)]

Problem is not re-producible in latest kernel - http://git.linaro.org/gitweb?p=landing-teams/working/ti/kernel.git;a=shortlog;h=refs/heads/tilt-3.4

[Soumya Basak (soumya-basak)]

observed with panda android jellybean 4.2 builds

https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-tracking-blob/#build=127

system crashes when running powertop.
log file attach for reference.