kernel oops on running apt-get install on panda 4430

[Milosz Wasilewski]

Environment:
 panda 4430 running 13.12 Ubuntu RC:
Linux linaro-developer 3.13.0-1-linaro-omap #1ubuntu1~ci+131215015239-Ubuntu SMP Sun Dec 15 02:27:47 UTC 201 armv7l armv7l armv7l GNU/Linux

When running apt-get update the kernel oops happened. Log below:

Unpacking python3-urllib3 (from .../python3-urllib3_1.6-2_all.deb) ...
[ 318.827331] Unable to handle kernel NULL pointer dereference at virtual address 000000a3
[ 318.836120] pgd = edd1c000
[ 318.839019] [000000a3] *pgd=ac8e1831, *pte=00000000, *ppte=00000000
[ 318.845581] Internal error: Oops: 1 [#1] SMP ARM
[ 318.850677] Modules linked in: snd_soc_omap_abe_twl6040 snd_soc_omap_mcpdm snd_soc_omap snd_pcm_dmaengine snd_soc_twl6040 snd_soc_core snd_compress regmap_spi snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_timer gator snd soundcore
[ 318.873107] CPU: 0 PID: 828 Comm: mmcqd/0 Not tainted 3.13.0-1-linaro-omap #1ubuntu1~ci+131215015239-Ubuntu
[ 318.883422] task: edaa81c0 ti: ede42000 task.ti: ede42000
[ 318.883422] PC is at mmc_blk_issue_rw_rq+0xec/0xc6c
[ 318.891540] LR is at arm_dma_unmap_sg+0x70/0x94
[ 318.891540] pc : [<c05c0f24>] lr : [<c001c61c>] psr: a00f0113
[ 318.891540] sp : ede43d78 ip : ede43cc0 fp : ede43ea4
[ 318.901733] r10: eddc8000 r9 : edde6800 r8 : ede4c824
[ 318.916839] r7 : ede4c868 r6 : ede43c50 r5 : ede43d74 r4 : 00000073
[ 318.921691] r3 : 00000000 r2 : 00000001 r1 : ec9af000 r0 : ede43c50
[ 318.930664] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
[ 318.932617] Control: 10c53c7d Table: add1c04a DAC: 00000017
[ 318.944488]
[ 318.944488] PC: 0xc05c0ea4:
[ 318.944763] 0ea4 e587312c e50b30fc e3a0c000 e50bc100 e50bc104 e51bc0f0 e51b40f0 e35c0000
[ 318.957763] 0ec4 0a000048 e59630ec e3130007 0a000002 e599333c e3530a01 0a0002d2 e51bc0fc
[ 318.965484] 0ee4 e59802ac e35c0001 e1a01009 9a000071 e1a02008 ebfff9a9 e59812ac e2811f49
[ 318.975250] 0f04 e5990000 e24b20ec ebffbf2c e2505000 0a00023c e5154124 e2456f49 e1a00006
[ 318.981994] 0f24 e1c423d0 e3a03000 e2022001 e192e003 13a07002 03a07001 eb00090d e51b20ec
[ 318.992736] 0f44 e3520007 979ff102 ea000184 c05c1188 c05c1188 c05c1144 c05c0f70 c05c0f84
[ 318.992736] 0f64 c05c0ff4 c05c1020 c05c10cc e51bc100 e28c3001 e35c0004 e50b3100 da000005
[ 319.010223] 0f84 e1a02007 e51b00f4 e5991000 ebfff85b e3500000 1a00004b e5957008 e3570000
[ 319.012969]
[ 319.012969] LR: 0xc001c59c:
[ 319.021697] c59c e89daff0 e59f7000 eaffffce c0aad064 e1a0c00d e92ddff0 e24cb004 e24dd00c
[ 319.032226] c5bc e52de004 e8bd4000 e2507000 e1a04001 e1a06002 e1a08003 e59b9004 0a000015
[ 319.039642] c5dc e597a1dc e59f3054 e35a0000 01a0a003 e3560000 da00000d e3a05000 e594100c
[ 319.049713] c5fc e5942008 e1a00007 e58d9000 e1a03008 e59ac014 e12fff3c e1a00004 eb101294
[ 319.058441] c61c e2855001 e1550006 e1a04000 1afffff2 e24bd028 e89daff0 e59fa000 eaffffeb
[ 319.061676] c63c c0aad064 e1a0c00d e92ddbf0 e24cb004 e52de004 e8bd4000 e2508000 e1a04001
[ 319.071777] c65c e1a07002 e1a09003 0a000013 e59861dc e59f3054 e3560000 01a06003 e3570000
[ 319.071777] c67c d89dabf0 e3a05000 e594100c e5942008 e596c020 e1a00008 e1a03009 e12fff3c
[ 319.093414]
[ 319.093414] SP: 0xede43cf8:
[ 319.093414] 3cf8 c07210a8 c008a9ac 00000000 ec9af000 00000001 c05c0f24 a00f0113 ffffffff
[ 319.106689] 3d18 ede43d64 ede4c824 ede43ea4 ede43d30 c0721024 c00083c4 ede43c50 ec9af000
[ 319.115417] 3d38 00000001 00000000 00000073 ede43d74 ede43c50 ede4c868 ede4c824 edde6800
[ 319.121704] 3d58 eddc8000 ede43ea4 ede43cc0 ede43d78 c001c61c c05c0f24 a00f0113 ffffffff
[ 319.132904] 3d78 00000000 c0aa009c ede43de4 ede43d90 c0049a54 c00495ec ed809c54 ede42000
[ 319.141632] 3d98 00000073 c07204f0 00000000 00000000 00000000 c0087f78 ede4c800 ec82b5b8
[ 319.150390] 3db8 00000000 c102f5e4 000001fb 00000000 c0bde978 edaa81c0 ede43e74 ede43de0
[ 319.159118] 3dd8 c008b00c c0089ffc c0049fb4 c00a1460 000001a8 ede42018 c07210a8 edaa81c0
[ 319.167877]
[ 319.167877] IP: 0xede43c40:
[ 319.172393] 3c40 c0049a54 c00495ec ed809c54 ede42038 00000073 00000001 c001f1e4 c0aad0f4
[ 319.181121] 3c60 000000a3 ede43d30 edde6800 eddc8000 ede43d2c ede43c80 c00083fc c001f1f0
[ 319.189880] 3c80 c0a9b298 ede42000 00000073 00000000 ede43cb4 ede43ca0 c0049fb4 c00a1460
[ 319.198608] 3ca0 000001a8 ede42010 c07210a8 edaa81c0 00000001 ede43db8 eddc8000 ede4c9a0
[ 319.207366] 3cc0 ede43cec ede43cd0 c008a8c8 c008a6a8 c05b0e20 a00f0113 ffffffff ede43d34
[ 319.216094] 3ce0 ede43cfc ede43cf0 c008a9b4 c008a7d4 ede43d74 ede43d00 c07210a8 c008a9ac
[ 319.224853] 3d00 00000000 ec9af000 00000001 c05c0f24 a00f0113 ffffffff ede43d64 ede4c824
[ 319.233581] 3d20 ede43ea4 ede43d30 c0721024 c00083c4 ede43c50 ec9af000 00000001 00000000
[ 319.242309]
[ 319.242309] FP: 0xede43e24:
[ 319.246856] 3e24 edfa3ce0 ede43e54 edfa3ce0 ec82b5b8 ede4c000 ede4d738 00000001 ede4d738
[ 319.255584] 3e44 ede4c824 ede43e74 ede43e58 c040d6fc c03ef790 ede4c000 edfa3ce0 ec82b5b8
[ 319.264343] 3e64 00000119 ede43eac ede43e78 c040da84 bd94352e 00000002 edde6800 ec82b5b8
[ 319.273071] 3e84 00000000 00000000 ede4c824 ede4c800 eddc8000 ede43eec ede43ea8 c05c1c80
[ 319.281829] 3ea4 c05c0e44 ede4c800 ede4d738 ede42000 00000000 ede43ed4 ede43ec8 c008a9b4
[ 319.290557] 3ec4 00000000 ede4d738 ede42000 122c8411 00000001 ec82b5b8 ede4c824 ede43f24
[ 319.299316] 3ee4 ede43ef0 c05c275c c05c1ab0 ede42030 ede4c82c c05c26a4 edd83940 00000000
[ 319.308044] 3f04 ede4c824 c05c26a4 00000000 00000000 00000000 ede43fac ede43f28 c0067064
[ 319.316772]
[ 319.316772] R0: 0xede43bd0:
[ 319.321319] 3bd0 c07205bc c008a9ac 00000001 edacbe00 ede43c14 ede43bf0 c045a850 c0720594
[ 319.330047] 3bf0 ede43bf0 ede43bf0 ede43c14 edaf2f4c edaf2f50 00000000 ede43c44 ede43c18
[ 319.338775] 3c10 00000100 00000100 ede43c44 ede43c28 c0049640 c008d750 edaa81c0 ede42000
[ 319.347534] 3c30 00000000 c0aa009c ede43c9c ede43c48 c0049a54 c00495ec ed809c54 ede42038
[ 319.356262] 3c50 00000073 00000001 c001f1e4 c0aad0f4 000000a3 ede43d30 edde6800 eddc8000
[ 319.365020] 3c70 ede43d2c ede43c80 c00083fc c001f1f0 c0a9b298 ede42000 00000073 00000000
[ 319.373748] 3c90 ede43cb4 ede43ca0 c0049fb4 c00a1460 000001a8 ede42010 c07210a8 edaa81c0
[ 319.382507] 3cb0 00000001 ede43db8 eddc8000 ede4c9a0 ede43cec ede43cd0 c008a8c8 c008a6a8
[ 319.391235]
[ 319.391235] R1: 0xec9aef80:
[ 319.395751] ef80 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 319.404510] efa0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 319.413238] efc0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 319.421997] efe0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 319.430725] f000 98393bc0 01000000 7a000000 1d0b0000 00000000 00000000 00000000 00000000
[ 319.439453] f020 00000000 83270000 02000000 ab0b0000 02000000 87270000 0a000000 00000000
[ 319.448211] f040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 319.456939] f060 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 319.465667]
[ 319.465667] R5: 0xede43cf4:
[ 319.470214] 3cf4 ede43d00 c07210a8 c008a9ac 00000000 ec9af000 00000001 c05c0f24 a00f0113
[ 319.478942] 3d14 ffffffff ede43d64 ede4c824 ede43ea4 ede43d30 c0721024 c00083c4 ede43c50
[ 319.487701] 3d34 ec9af000 00000001 00000000 00000073 ede43d74 ede43c50 ede4c868 ede4c824
[ 319.496429] 3d54 edde6800 eddc8000 ede43ea4 ede43cc0 ede43d78 c001c61c c05c0f24 a00f0113
[ 319.505157] 3d74 ffffffff 00000000 c0aa009c ede43de4 ede43d90 c0049a54 c00495ec ed809c54
[ 319.513916] 3d94 ede42000 00000073 c07204f0 00000000 00000000 00000000 c0087f78 ede4c800
[ 319.522644] 3db4 ec82b5b8 00000000 c102f5e4 000001fb 00000000 c0bde978 edaa81c0 ede43e74
[ 319.531402] 3dd4 ede43de0 c008b00c c0089ffc c0049fb4 c00a1460 000001a8 ede42018 c07210a8
[ 319.540130]
[ 319.540130] R6: 0xede43bd0:
[ 319.544677] 3bd0 c07205bc c008a9ac 00000001 edacbe00 ede43c14 ede43bf0 c045a850 c0720594
[ 319.553405] 3bf0 ede43bf0 ede43bf0 ede43c14 edaf2f4c edaf2f50 00000000 ede43c44 ede43c18
[ 319.562164] 3c10 00000100 00000100 ede43c44 ede43c28 c0049640 c008d750 edaa81c0 ede42000
[ 319.570892] 3c30 00000000 c0aa009c ede43c9c ede43c48 c0049a54 c00495ec ed809c54 ede42038
[ 319.579650] 3c50 00000073 00000001 c001f1e4 c0aad0f4 000000a3 ede43d30 edde6800 eddc8000
[ 319.588378] 3c70 ede43d2c ede43c80 c00083fc c001f1f0 c0a9b298 ede42000 00000073 00000000
[ 319.597137] 3c90 ede43cb4 ede43ca0 c0049fb4 c00a1460 000001a8 ede42010 c07210a8 edaa81c0
[ 319.605865] 3cb0 00000001 ede43db8 eddc8000 ede4c9a0 ede43cec ede43cd0 c008a8c8 c008a6a8
[ 319.614624]
[ 319.614624] R7: 0xede4c7e8:
[ 319.619140] c7e8 00000000 00000000 00000000 00000000 00000000 00000000 6dea6dea dead4ead
[ 319.627868] c808 ffffffff ffffffff c109f488 c0c00318 00000000 c0947bdc ede4c400 edde6800
[ 319.636627] c828 edaa81c0 25452545 dead4ead ffffffff ffffffff c109f4a0 c0c00538 00000000
[ 319.645355] c848 c08dce8c 00000000 ede4c850 ede4c850 00000000 c05c1aa4 ede4c800 ede4d738
[ 319.654113] c868 ec82b5b8 00000000 ede4c8e4 ede4c94c ede4c918 00000000 00000000 00000000
[ 319.662841] c888 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 319.671569] c8a8 c05afd78 eddc8000 00000000 00000000 00000000 00000000 00000000 00000000
[ 319.680328] c8c8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000019
[ 319.689056]
[ 319.689056] R8: 0xede4c7a4:
[ 319.693572] c7a4 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 319.702331] c7c4 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 319.711059] c7e4 00000000 00000000 00000000 00000000 00000000 00000000 00000000 6dea6dea
[ 319.719818] c804 dead4ead ffffffff ffffffff c109f488 c0c00318 00000000 c0947bdc ede4c400
[ 319.728546] c824 edde6800 edaa81c0 25452545 dead4ead ffffffff ffffffff c109f4a0 c0c00538
[ 319.737304] c844 00000000 c08dce8c 00000000 ede4c850 ede4c850 00000000 c05c1aa4 ede4c800
[ 319.746032] c864 ede4d738 ec82b5b8 00000000 ede4c8e4 ede4c94c ede4c918 00000000 00000000
[ 319.754760] c884 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 319.763519]
[ 319.763519] R9: 0xedde6780:
[ 319.768035] 6780 aaaaaa88 aaaaaaaa aa2aaa2a aaaaaaaa a2a8aaaa a0aa8aaa a8aaaaaa aaaaa8aa
[ 319.776763] 67a0 aa8a2aaa aaaaaaa2 aaaaaaaa aa2aaaa2 aaaaaaaa 82a28aaa aaa2aaaa aaa2aaa0
[ 319.785522] 67c0 aaa2aaa2 aaa8aaaa a8aa2aaa aaaaaaaa 8a2aaa2a aa8aaaaa aaaaaaaa 2aa2aaa8
[ 319.794250] 67e0 aaaaaaaa aa8a2aaa aaaaaaaa aaaaaaaa aa8aaaaa 2aaaa2aa aaa8a8aa aaaaaaaa
[ 319.803009] 6800 eddc8000 00000000 eddc8008 ede34f00 edddfb00 ede38814 eddc8014 eddc8010
[ 319.811737] 6820 ed922f80 c0b17228 ede569d0 00000005 00000007 00000000 c0b2c9ec 00000001
[ 319.820465] 6840 00020002 dead4ead ffffffff ffffffff c0b42958 00000000 00000000 c08da748
[ 319.829223] 6860 edde6860 edde6860 00000000 00000000 edde683c c0f4e880 c0bea368 00000000
[ 319.837951]
[ 319.837951] R10: 0xeddc7f80:
[ 319.842590] 7f80 00000000 00000059 00000000 00000100 00633269 00000000 00000000 00000000
[ 319.851318] 7fa0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 319.860076] 7fc0 00000000 00000000 00000000 00000000 eddbba80 00000000 00000000 00000000
[ 319.868804] 7fe0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 319.877532] 8000 eda03010 00000000 eda03010 ede12a80 ede37880 edde6814 ede36c0c edd83740
[ 319.886291] 8020 ed922f80 c0b17228 ede56d40 00000005 00000007 00000000 00000000 00000001
[ 319.895019] 8040 00000000 dead4ead ffffffff ffffffff c0b42958 00000000 00000000 c08da748
[ 319.903747] 8060 eddc8060 eddc8060 00000000 00000000 eddc803c c0f4e880 00000000 00000000
[ 319.912506] Process mmcqd/0 (pid: 828, stack limit = 0xede42248)
[ 319.918884] Stack: (0xede43d78 to 0xede44000)
[ 319.923492] 3d60: 00000000 c0aa009c
[ 319.932159] 3d80: ede43de4 ede43d90 c0049a54 c00495ec ed809c54 ede42000 00000073 c07204f0
[ 319.940826] 3da0: 00000000 00000000 00000000 c0087f78 ede4c800 ec82b5b8 00000000 c102f5e4
[ 319.949493] 3dc0: 000001fb 00000000 c0bde978 edaa81c0 ede43e74 ede43de0 c008b00c c0089ffc
[ 319.958160] 3de0: c0049fb4 c00a1460 000001a8 ede42018 c07210a8 edaa81c0 c0b0f8c4 ede4c824
[ 319.966827] 3e00: ede4c800 00000000 ec82b5b8 ec82b618 ede43e34 ede43e20 c03f060c c040d6fc
[ 319.975494] 3e20: ec82b5b8 edfa3ce0 ede43e54 edfa3ce0 ec82b5b8 ede4c000 ede4d738 00000001
[ 319.984161] 3e40: ede4d738 ede4c824 ede43e74 ede43e58 c040d6fc c03ef790 ede4c000 edfa3ce0
[ 319.992828] 3e60: ec82b5b8 00000119 ede43eac ede43e78 c040da84 bd94352e 00000002 edde6800
[ 320.001495] 3e80: ec82b5b8 00000000 00000000 ede4c824 ede4c800 eddc8000 ede43eec ede43ea8
[ 320.010162] 3ea0: c05c1c80 c05c0e44 ede4c800 ede4d738 ede42000 00000000 ede43ed4 ede43ec8
[ 320.018829] 3ec0: c008a9b4 00000000 ede4d738 ede42000 122c8411 00000001 ec82b5b8 ede4c824
[ 320.027496] 3ee0: ede43f24 ede43ef0 c05c275c c05c1ab0 ede42030 ede4c82c c05c26a4 edd83940
[ 320.036163] 3f00: 00000000 ede4c824 c05c26a4 00000000 00000000 00000000 ede43fac ede43f28
[ 320.044830] 3f20: c0067064 c05c26b0 00000000 c008a7d4 ede4c824 00000000 00000000 dead4ead
[ 320.053497] 3f40: ffffffff ffffffff c0b42424 00000000 00000000 c08d69f8 ede43f58 ede43f58
[ 320.062164] 3f60: 00000000 00000000 dead4ead ffffffff ffffffff c0b42424 00000000 00000000
[ 320.070831] 3f80: c08d69f8 ede43f84 ede43f84 bd94352e edd83940 c0066f7c 00000000 00000000
[ 320.079498] 3fa0: 00000000 ede43fb0 c000eac8 c0066f88 00000000 00000000 00000000 00000000
[ 320.088165] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 320.096832] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 aa3aaaab aaaaa8aa
[ 320.105499] [<c05c0f24>] (mmc_blk_issue_rw_rq+0xec/0xc6c) from [<c05c1c80>] (mmc_blk_issue_rq+0x1dc/0x5a4)
[ 320.115753] [<c05c1c80>] (mmc_blk_issue_rq+0x1dc/0x5a4) from [<c05c275c>] (mmc_queue_thread+0xb8/0x158)
[ 320.125701] [<c05c275c>] (mmc_queue_thread+0xb8/0x158) from [<c0067064>] (kthread+0xe8/0x108)
[ 320.134765] [<c0067064>] (kthread+0xe8/0x108) from [<c000eac8>] (ret_from_fork+0x14/0x20)
[ 320.143432] Code: 0a00023c e5154124 e2456f49 e1a00006 (e1c423d0)
[ 320.150024] ---[ end trace 4a919a30f5cec4c2 ]---

I don't have different SD card to check if the problem is related to faulty hardware. The bug was reproduced twice.

[Botao (botao-sun)]

I'm trying to re-produce this issue on my TI Panda 4430 board with SanDisk SDHC Class 4 4GB card. It doesn't happen every time. I constantly rebooted it 3 times, ran apt-get update 2 times, then rebooted it again. No panic occurred during this period. But that panic did happen once, which was when I power on it earlier today, please refer to attachment to get the log output.

However, even that Kernel panic happened, it won't stop the board to finally boot into system.

[Milosz Wasilewski (mwasilew)]

Just noticed the bug in LAVA:
https://validation.linaro.org/scheduler/job/92371/log_file#L_27_836

[Milosz Wasilewski (mwasilew)]

Changing status to confirmed as it happens not only on one device.

[Fathi Boudra (fboudra)]

Can we still reproduce this bug?

[Botao (botao-sun)]

From the experience of 14.03 release test on Linux Linaro ubuntu image:

http://snapshots.linaro.org/ubuntu/pre-built/panda/616/panda-saucy_developer_20140323-616.img.gz

This issue was not observed.

[Andrey Konovalov (andrey-konovalov)]

I've tried panda-saucy_developer_20140325-618.img.gz on my 4430 and 4460 Panda's, and unfortunately this oops:
-----8<-----
[ 118.044738] Unable to handle kernel NULL pointer dereference at virtual addre
ss 000000a3
[ 118.044830] pgd = ec240000
[ 118.053436] [000000a3] *pgd=ac026831, *pte=00000000, *ppte=00000000
[ 118.063079] Internal error: Oops: 1 [#1] SMP ARM
...
[ 118.111328] PC is at mmc_blk_issue_rw_rq+0xec/0xc8c
[ 118.113342] LR is at arm_dma_unmap_sg+0x70/0x94
-----8<-----

- was fairly easy to reproduce by running:
# time for i in {1..10}; do echo $i; df /; apt-get install -y git; git --version; git clone git://busybox.net/busybox.git; df /; rm -rf busybox; apt-get remove -y git; apt-get remove -y git-man liberror-perl rsync; done

[Andrey Konovalov (andrey-konovalov)]

... and got the same oops on Panda 4430 with panda-saucy_developer_20140323-616.img.gz. At 2nd iteration of the above cycle. Samsung class 10 16GB card.