Kernel Panic shows on Linux Linaro ubuntu TI Panda image.

[Botao]

On TI Panda 4430 board with Linux Linaro ubuntu image:

http://snapshots.linaro.org/ubuntu/pre-built/panda/596/panda-saucy_developer_20140126-596.img.gz

Kernel panic frequently shows in variety of scenarios (version 3.13.0-1-linaro-omap), like:

1. Install some packages via "apt-get" then reboot (power off / on) the board;

2. Compile third-party software, for example:
# apt-get install git
# git clone git://busybox.net/busybox.git
# cd busybox
# make menuconfig
# make

For the log of Kernel panic, please refer to attachment. It always starts from:

Unable to handle kernel NULL pointer dereference at virtual address 00000037

Also, same issue happened on iMX6 (version: 3.13-rc1) as well:

https://lkml.org/lkml/2014/1/22/294

###########################################
This issue is observed on following images:
http://snapshots.linaro.org/ubuntu/pre-built/panda/596/panda-saucy_developer_20140126-596.img.gz
http://snapshots.linaro.org/ubuntu/pre-built/panda/602/panda-saucy_developer_20140203-602.img.gz
http://snapshots.linaro.org/ubuntu/pre-built/panda/611/panda-saucy_developer_20140222-611.img.gz
http://snapshots.linaro.org/kernel-hwpack/linux-linaro-tracking-llct-panda/357/hwpack_linaro-panda_20140402-1449_b357_armhf_supported.tar.gz
http://snapshots.linaro.org/ubuntu/images/nano-lava/650/linaro-saucy-nano-lava-20140325-650.tar.gz
http://snapshots.linaro.org/ubuntu/pre-built/panda/627/panda-saucy_developer_20140414-627.img.gz
http://snapshots.linaro.org/ubuntu/pre-built/panda/630/panda-saucy_developer_20140417-630.img.gz

[Botao (botao-sun)]

Same error observed on TI Panda 4460 with Linux Linaro ubuntu image:

http://snapshots.linaro.org/ubuntu/pre-built/panda/602/panda-saucy_developer_20140203-602.img.gz

Please refer to attachment to get detailed log output.

[Andrey Konovalov (andrey-konovalov)]

These kernel oopses are very sensitive to the gcc version used to build the kernel.
* I've seen no such oopses with gcc 4.6 included into Ubuntu 12.04 LTS.
* gcc 4.8 from the 14.01 Linaro release also looks good ("gcc version 4.8.3 20140106 (prerelease) (crosstool-NG linaro-1.13.1-4.8-2014.01 - Linaro GCC 2013.11"). (used in linux-linaro kernel CI)
* gcc 4.8 from the 13.08 Linaro release produces the kernel very subject to these oopses ("gcc version 4.8.2 20130805 (prerelease) (crosstool-NG linaro-1.13.1-4.8-2013.08 - Linaro GCC 2013.08")
* The pre-built images use gcc-4.8 from Saucy ("gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu9)"), and it also produces the kernel which oopses on Panda.

As we can't just switch the pre-built images from the Saucy's toolchain, I'll have to do some more investigations to find out the cause of the oopses when the 4.8.1 gcc from Saucy is used to build the kernel.

[Botao (botao-sun)]

Just a update, this issue still exists on the Linux Linaro ubuntu Panda 14.02 release candidate image:

http://snapshots.linaro.org/ubuntu/pre-built/panda/611/panda-saucy_developer_20140222-611.img.gz

Detailed console output can be found in attachment.

[Botao (botao-sun)]

For the TI Panda 4430 with Linux Linaro ubuntu image:

http://snapshots.linaro.org/ubuntu/pre-built/panda/616/panda-saucy_developer_20140323-616.img.gz

This issue couldn't be reproduced during the test.

However, for TI Panda 4460 with the same image, an "Internal error" occurred and hung the board, but looks like it's related to the wlan driver, not the problem described in this bug. Anyway, console log output attached, if necessary, I will append the log to Bug 1164329 as well.

[Botao (botao-sun)]

One thing, when trying to install mplayer:

# apt-get install mplayer

following error shows and hangs the board:

BUG: scheduling while atomic: swapper/1/0/0xffff0000

Above error message flushing to the serial console.

[Botao (botao-sun)]

This issue is also observed with following hardware pack and rootfs:
http://snapshots.linaro.org/kernel-hwpack/linux-linaro-tracking-llct-panda/357/hwpack_linaro-panda_20140402-1449_b357_armhf_supported.tar.gz
http://snapshots.linaro.org/ubuntu/images/nano-lava/650/linaro-saucy-nano-lava-20140325-650.tar.gz

"kernel NULL pointer dereference" shows up again:
https://validation.linaro.org/scheduler/job/119870/log_file#L_31_174

For full log output, please refer to attachment.

[Andrey Konovalov (andrey-konovalov)]

Hmm... The discussion in that email thread (https://lkml.org/lkml/2014/1/22/294 - Thank you Botao!) points to:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854
And this seems to be in perfect agreement with my comment #3. I am not 100% sure yet (need to take a better look at the generated code), but it does look like that gcc bug.

[Botao (botao-sun)]

On TI Panda 4460 board with Linux Linaro ubuntu image:

http://snapshots.linaro.org/ubuntu/pre-built/panda/630/panda-saucy_developer_20140417-630.img.gz

The Kernel panic looks a little bit different - it's related to the WiFi driver:

[ 175.915100] Unable to handle kernel NULL pointer dereference at virtual address 00000037
[ 175.924774] pgd = ec114000
[ 175.927795] [00000037] *pgd=ac04b831, *pte=00000000, *ppte=00000000
[ 175.927795] Internal error: Oops: 17 [#1] SMP ARM
[ 175.939422] Modules linked in: arc4 wl12xx wlcore mac80211 cfg80211 snd_soc_omap_abe_twl6040 snd_soc_omap_mcpdm snd_soc_twl6040 snd_soc_ome
[ 175.966705] CPU: 0 PID: 1631 Comm: http Not tainted 3.14.0-1-linaro-omap #1ubuntu1~ci+140417071235-Ubuntu

During the test, it happened twice. The first one is just after the system booted up and ran "apt-get update"; and the second one is after other several test then ran "apt-get install git".

For full log output, please refer to attachment.

Note: Above happened when the WiFi not manually configured.

[Botao (botao-sun)]

On TI Panda 4430 & 4460 with Linux Linaro ubuntu image:
http://snapshots.linaro.org/ubuntu/pre-built/panda/651/panda-trusty_developer_20140522-651.img.gz

This issue has gone, but at least one more build monitoring is required for bug status changing.

[Botao (botao-sun)]

On TI Panda 4430 & 4460 with Linux Linaro ubuntu image:

http://snapshots.linaro.org/ubuntu/pre-built/panda/654/panda-trusty_developer_20140525-654.img.gz

This issue has gone, though for the system hangs issue, please refer to Bug 1324369.