Lucid (10.04) ext4 kernel panic under extreme load

[Ameet Paranjape]

This appears to affect the lucid kernel (2.6.32-33-server) regardless of the backing store, either physical or virtual. Stacking block devices
are also not a variable e.g. LVM or multipath.

To reproduce:

1) Create the following FIO test file (call it say bigfio.fio)

 [global]
 rw=randread
 size=16g
 directory=/var/lib/TESTDATA
 ioengine=libaio
 iodepth=4
 invalidate=1
 direct=1

 [bgwriter]
 rw=randwrite
 iodepth=32

 [queryA]
 iodepth=1
 ioengine=mmap
 direct=0
 thinktime=3

 [queryB]
 iodepth=1
 ioengine=mmap
 direct=0
 thinktime=5

 [bgupdater]
 rw=randrw
 iodepth=16
 thinktime=40
 size=128m

2) # apt-get install -y fio
3) # mkdir /var/lib/TESTDATA
4) # fio bigfio.fio

The time to failure can vary, after an hour or two the kernel panics here:

      KERNEL: syms/usr/lib/debug/boot/vmlinux-2.6.32-33-server
    DUMPFILE: dump/VmCore
        CPUS: 4
        DATE: Thu Oct 13 10:22:05 2011
      UPTIME: 01:08:14
LOAD AVERAGE: 2.54, 2.21, 2.19
       TASKS: 151
    NODENAME: toyon
     RELEASE: 2.6.32-33-server
     VERSION: #72-Ubuntu SMP Fri Jul 29 21:21:55 UTC 2011
     MACHINE: x86_64 (2333 Mhz)
      MEMORY: 4 GB
       PANIC: "[ 4092.217111] kernel BUG at /build/buildd/linux-2.6.32/mm/slub.c:2969!"
         PID: 314
     COMMAND: "ext4-dio-unwrit"
        TASK: ffff880127fc0000 [THREAD_INFO: ffff880127452000]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 314 TASK: ffff880127fc0000 CPU: 0 COMMAND: "ext4-dio-unwrit"
 #0 [ffff880127453bb0] die at ffffffff8101712b
 #1 [ffff880127453be0] do_trap at ffffffff8155acc4
 #2 [ffff880127453c40] do_invalid_op at ffffffff81014ee5
 #3 [ffff880127453ce0] invalid_op at ffffffff81013f7b
    [exception RIP: kfree+300]
    RIP: ffffffff81137c9c RSP: ffff880127453d90 RFLAGS: 00010246
    RAX: 0200000000000000 RBX: ffffffff811dd7a7 RCX: 0000000000000021
    RDX: 0000000000952318 RSI: ffffea000413f5a8 RDI: ffff88012a463300
    RBP: ffff880127453db0 R8: 0000000000000000 R9: feb6f83d31fb6402
    R10: 0000000000000000 R11: 0000000000000000 R12: ffff88012a463300
    R13: ffff88011fc0d608 R14: ffffe8ffffc04340 R15: ffff880127453fd8
    ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #4 [ffff880127453db8] ext4_free_io_end at ffffffff811dd7a7
 #5 [ffff880127453dd8] ext4_end_aio_dio_work at ffffffff811dd823
 #6 [ffff880127453e08] run_workqueue at ffffffff810814f7
 #7 [ffff880127453e68] worker_thread at ffffffff81081673
 #8 [ffff880127453ee8] kthread at ffffffff81085d26
 #9 [ffff880127453f48] kernel_thread at ffffffff810141ea

[Ameet Paranjape (ameetp)]

Logs to follow...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Peter Petrakis (peter-petrakis)]

Marked invalid for oem-priority project based on krafty's comments on 17/10.