Kernel Crash [Debian or Kernel.org kernels OK]

[mike morrison]

Binary package hint: linux-image-2.6.15-23-386

There seems to be bug with the ubuntu 2.6.15 kernel on my machine. (I run debian with a custom 2.6.15.1 kernel with no problems).

The bug seems to happen at random times when running the 6.06 live/install cd (and also happened to me when running a prerelease dapper based on an older 2.6.15 build of the kernel).

After this happens the system becomes very unstable and programs start to segfault. It makes it impossible to install 6.06.

Here is the output from /var/log/messages:

Jun 1 21:08:43 ubuntu kernel: [4294880.333000] ------------[ cut here ]------------
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] PREEMPT
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] Modules linked in: xfs exportfs reiserfs jfs ext3 jbd ntfs ext2 nls_utf8 vfat fat rfcomm l2cap bluetooth ppdev lp powernow_k8 cpufreq_userspace cpufreq_stats freq_table cpufreq_powersave cpufreq_ondemand cpufreq_conservative video tc1100_wmi sony_acpi pcc_acpi hotkey dev_acpi container button acpi_sbs battery i2c_acpi_ec ac dm_mod md_mod ipv6 snd_seq_dummy snd_seq_oss snd_seq_midi snd_seq_midi_event snd_seq snd_via82xx tsdev usblp snd_ac97_codec snd_ac97_bus snd_pcm_oss i2c_viapro analog gameport psmouse parport_pc parport serio_raw floppy quickcam videodev shpchp pcspkr rtc af_packet snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore i2c_core pci_hotplug amd64_agp agpgart sg evdev 8139too 8139cp mii sd_mod sr_mod usb_storage scsi_mod squashfs unionfs loop nls_cp437 isofs ide_generic ehci_hcd uhci_hcd usbcore ide_cd cdrom ide_disk via82cxxx generic thermal processor fan capability commoncap vga16fb vgastate fbco
Jun 1 21:08:43 ubuntu kernel: tileblit font bitblit softcursor
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] CPU: 0
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] EIP: 0060:[radix_tree_tag_set+136/160] Not tainted VLI
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] EFLAGS: 00210046 (2.6.15-23-386)
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] EIP is at radix_tree_tag_set+0x88/0xa0
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] eax: 00000000 ebx: 00000000 ecx: e1a59160 edx: 00000002
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] esi: 00000006 edi: 00000004 ebp: 00000002 esp: e1689d5c
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] ds: 007b es: 007b ss: 0068
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] Process pdflush (pid: 10333, threadinfo=e1688000 task=f2a84030)
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] Stack: 00000008 c16efd60 f311f844 00000000 00200286 c0146bd1 f311f848 0000009e
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] 00000001 c16efd60 f36dd858 00000000 f36dd858 c0162cf9 c16efd60 00000000
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] 02f7b0ff 00000000 7b100000 0000002f e1689f38 c16efd60 c0164485 f311f7a4
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] Call Trace:
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [test_set_page_writeback+177/256] test_set_page_writeback+0xb1/0x100
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [__block_write_full_page+409/848] __block_write_full_page+0x199/0x350
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [block_write_full_page+149/288] block_write_full_page+0x95/0x120
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [blkdev_get_block+0/112] blkdev_get_block+0x0/0x70
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [pagevec_lookup_tag+34/48] pagevec_lookup_tag+0x22/0x30
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [blkdev_writepage+20/32] blkdev_writepage+0x14/0x20
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [blkdev_get_block+0/112] blkdev_get_block+0x0/0x70
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [mpage_writepages+514/1040] mpage_writepages+0x202/0x410
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [blkdev_writepage+0/32] blkdev_writepage+0x0/0x20
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [generic_writepages+17/32] generic_writepages+0x11/0x20
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [__sync_single_inode+105/592] __sync_single_inode+0x69/0x250
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [__writeback_single_inode+65/368] __writeback_single_inode+0x41/0x170
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [del_timer+21/128] del_timer+0x15/0x80
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [schedule_timeout+73/176] schedule_timeout+0x49/0xb0
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [sync_sb_inodes+435/704] sync_sb_inodes+0x1b3/0x2c0
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [writeback_inodes+102/272] writeback_inodes+0x66/0x110
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [background_writeout+128/192] background_writeout+0x80/0xc0
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [__pdflush+225/432] __pdflush+0xe1/0x1b0
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [pdflush+0/48] pdflush+0x0/0x30
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [pdflush+30/48] pdflush+0x1e/0x30
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [background_writeout+0/192] background_writeout+0x0/0xc0
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [kthread+147/160] kthread+0x93/0xa0
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [kthread+0/160] kthread+0x0/0xa0
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] Code: 03 0f ab 11 8b 5c 93 04 85 db 74 1f 83 ee 06 45 39 ef 75 cf eb b3 8d b6 00 00 00 00 8d bc 27 00 00 00 00 31 c0 5a 5b 5e 5f 5d c3 <0f> 0b 74 01 b8 8e 30 c0 eb d7 8d b4 26 00 00 00 00 8d bc 27 00
Jun 1 21:08:43 ubuntu kernel: [4294880.333000] <6>note: pdflush[10333] exited with preempt_count 1

[Gareth Fitzworthington (mapping-gp-deactivatedaccount)]

This bug has had no activity for a considerable period. This is a check to see if there is still interest in investigating this bug report.

[mike morrison (mike-morrison)]

this is still an issue with the latest gutsy kernel on my desktop machine. every time i do an update i need to download the kernel source from kernel.org and build a custom kernel (which can be difficult with the instability). there is something different about the ubuntu kernel that is causing this.

with each new release i eagerly try the new default kernel only to realize that it is still an issue. it has persisted for so long that i've lost hope. will it still be a problem in hardy? i think so...

[Gareth Fitzworthington (mapping-gp-deactivatedaccount)]

Ok.
Updating effected packages to include Gutsy.
If you are able to access 8.04 pre-release this may be a useful test (it would get noticed).
A Gutsy (& if possible 8.04beta) dmesg of error situation would also be useful (and required for proper analysis by kernel team).

[mike morrison (mike-morrison)]

i just tried the 8.04 beta and the kernel still exhibits the problem. the attached dmesg shows some of the strange behavior just prior to the machine locking up.

[Leann Ogasawara (leannogasawara)]

Hi Mike,

You mention that you are able to run the upstream kernels and do not experience this issue. Do you use the ubuntu kernel config when you build the upstream kernel? Also, would you be able to attach the same stack trace output as in your bug description but with the 2.6.24 Hardy kernel? Thanks.

[Leann Ogasawara (leannogasawara)]

Also, if you do use the Ubuntu kernel config it would be nice to know which specific kernel config you are using, for ex config-2.6.24-16-generic.

[mike morrison (mike-morrison)]

Leann:

What I usually do is download the latest kernel from kernel.org, then make oldconfig, then run make-kpkg to make the packages. So at one point the oldconfig was probably based on the ubuntu kernel config but I may have changed a few settings here and there over the years.

How can I get the stack trace output with the latest kernel? It seems that things are crashing before any kernel panic (as can be seen in the dmesg output). If I'm in the console I can see some of the output when the kernel locks up but at that point the system is frozen and the LEDs on the keyboard are blinking.

I tried to ouput dmesg to a file in a loop :

while test 1; do for i in {1..10}; do dmesg > /mnt/dmesg_$i.txt; sleep 1; done; done

but it must have crashed or stopped too because the trace didn't make it into one of the files.

Mike

[Leann Ogasawara (leannogasawara)]

Hi Mike,

Care to take a peek at /var/log/kern.log.0 and see if anything is logged there regarding the panic.

I'd also be interested to know if you'd be willing to test the upstream kernel from kernel.org with a more recent Ubuntu Hardy Heron kernel config. For example 'cp /boot/config-2.6.24-16-generic linux/.config'. Or maybe try building the Ubuntu kernel source with the kernel config you're using for building the upstream kernel. I'm guessing it's likely that some kernel config option is enabled/disabled in the Ubuntu kernel that maybe triggering the issue you are seeing. Thanks for your help and testing.

[mike morrison (mike-morrison)]

Leann:

Looks like you were right. I tried an upstream kernel (2.6.25 from kernel.org) with the hardy kernel config and it also crashes. I took two pictures of the crashes that occur with the 2.6.24-16-generic kernel (i couldn't get anything from kern.log.0) and I will attach them after this message. I will also attach the working config I have from a 2.6.23.12 kernel (I didn't have time to build a working 2.6.25 yet) and the config for the crashing 2.6.25 kernel that i just built.

Mike

[Leann Ogasawara (leannogasawara)]

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

[Launchpad Janitor (janitor)]

Per a decision made by the Ubuntu Kernel Team, bugs will longer be assigned to the ubuntu-kernel-team in Launchpad as part of the bug triage process. The ubuntu-kernel-team is being unassigned from this bug report. Refer to https://wiki.ubuntu.com/KernelTeamBugPolicies for more information. Thanks.

[Andres Mujica (andres.mujica)]

Hi Mike, Are you able to test with latest intrepid and Jaunty Alpha?

Also would it be possible for you to attach a diff between the Ubuntu kernel config file and the one you're using?

thanks

[Sergio Zanchetta (primes2h)]

The 18 month support period for Gutsy Gibbon 7.10 has reached its end of life -
http://www.ubuntu.com/news/ubuntu-7.10-eol . As a result, we are closing the
linux-source-2.6.22 kernel task. It would be helpful if you could test the
new Jaunty Jackalope 9.04 release and confirm if this issue remains -
http://www.ubuntu.com/getubuntu/releasenotes/904overview. If the issue still exists with the Jaunty
release, please update this report by changing the Status of the "linux (Ubuntu)"
task from "Incomplete" to "New". Also please be sure to run the command below
which will automatically gather and attach updated debug information to this
report. Thanks in advance.

apport-collect -p linux-image-2.6.28-11-generic 48264

[Rolf Leggewie (r0lf)]

I'm afraid there isn't much we can do without further information. Thank you for reporting this issue. I will close it now, but please feel free to reopen with more information.

[mike morrison (mike-morrison)]

this bug still happens on my machine in karmic. i've all but given up hope for it to be fixed and usually just build a kernal with my custom kernal config after each upgrade.