trusty: kernel oops during runing xfstests(generic/036)

[Ming Lei]

generic/036 11s ...[ 680.213164] Kernel panic - not syncing: BUG!
[ 680.217414] CPU: 4 PID: 16161 Comm: aio-dio-fcntl-r Not tainted 3.13.0-37-generic #64-Ubuntu
[ 680.225807] Call trace:
[ 680.228246] [<ffffffc000088490>] dump_backtrace+0x0/0x16c
[ 680.233616] [<ffffffc00008860c>] show_stack+0x10/0x1c
[ 680.238642] [<ffffffc0006022f8>] dump_stack+0x74/0x94
[ 680.243666] [<ffffffc0005fc52c>] panic+0xe0/0x20c
[ 680.248346] [<ffffffc000222520>] ext4_direct_IO+0x400/0x438
[ 680.253891] [<ffffffc00014fdd8>] generic_file_direct_write+0xc0/0x1a4
[ 680.260297] [<ffffffc000150114>] __generic_file_aio_write+0x258/0x2e8
[ 680.266703] [<ffffffc0001501f4>] generic_file_aio_write+0x50/0xc8
[ 680.272764] [<ffffffc00021e9a0>] ext4_file_write+0x9c/0x390
[ 680.278307] [<ffffffc0001f22a8>] do_io_submit+0x554/0x7b8
[ 680.283677] [<ffffffc0001f2518>] SyS_io_submit+0xc/0x18
[ 680.288874] CPU5: stopping
[ 680.291567] CPU: 5 PID: 16162 Comm: aio-dio-fcntl-r Not tainted 3.13.0-37-generic #64-Ubuntu
[ 680.299960] Call trace:
[ 680.302394] [<ffffffc000088490>] dump_backtrace+0x0/0x16c
[ 680.307764] [<ffffffc00008860c>] show_stack+0x10/0x1c
[ 680.312788] [<ffffffc0006022f8>] dump_stack+0x74/0x94
[ 680.317812] [<ffffffc00008ed08>] handle_IPI+0x120/0x138
[ 680.323010] [<ffffffc0000812cc>] gic_handle_irq+0x74/0x7c
[ 680.328380] Exception stack(0xffffffc38019feb0 to 0xffffffc38019ffd0)
[ 680.334788] fea0: f1819580 0000007f 00000003 00000000
[ 680.342922] fec0: ffffffff ffffffff 00400ad0 00000000 00000000 00000000 00000004 00000000
[ 680.351057] fee0: 00030002 00000000 ffffff80 00000000 0009cca5 00000000 00000000 00000000
[ 680.359191] ff00: a6d7d000 0000007f 00000000 00000000 00000019 00000000 0001fbe6 00000000
[ 680.367325] ff20: 54d234e6 00000000 0009cca5 00000000 00000018 00000000 e8000000 00000003
[ 680.375459] ff40: 00000000 00000000 68d39b42 00264768 00412068 00000000 a6d25a14 0000007f
[ 680.383593] ff60: f1819230 0000007f f1819580 0000007f 00000003 00000000 00020002 00000000
[ 680.391727] ff80: 00030002 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 680.399862] ffa0: 00000000 00000000 00000000 00000000 00000000 00000000 f18194d0 0000007f
[ 680.407997] ffc0: 00400ad0 00000000 f18194d0 0000007f
[ 680.413021] CPU1: stopping
[ 680.415717] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.13.0-37-generic #64-Ubuntu
[ 680.423247] Call trace:
[ 680.425682] [<ffffffc000088490>] dump_backtrace+0x0/0x16c
[ 680.431052] [<ffffffc00008860c>] show_stack+0x10/0x1c
[ 680.436077] [<ffffffc0006022f8>] dump_stack+0x74/0x94
[ 680.441101] [<ffffffc00008ed08>] handle_IPI+0x120/0x138
[ 680.446298] [<ffffffc0000812cc>] gic_handle_irq+0x74/0x7c
[ 680.451668] Exception stack(0xffffffc3ec17be20 to 0xffffffc3ec17bf40)
[ 680.458074] be20: ec178000 ffffffc3 ec178000 ffffffc3 ec17bf60 ffffffc3 00085600 ffffffc0
[ 680.466209] be40: 000886fe 00000000 00000000 00000000 fff85834 ffffffc3 00009483 00000001
[ 680.474343] be60: 00000020 00000000 00000010 00000000 5e7bbb80 0000009e d3957e28 ffffffc3
[ 680.482477] be80: ec117310 ffffffc3 ec17bd60 ffffffc3 00009481 00000001 00000000 00000000
[ 680.490611] bea0: 000827ce 00000000 00000000 00000000 00000000 00000000 ffffffff ffffffff
[ 680.498745] bec0: 001ad9f8 ffffffc0 004e2488 00000000 f719f060 0000007f ec178000 ffffffc3
[ 680.506879] bee0: ec178000 ffffffc3 00961000 ffffffc0 009793b8 ffffffc0 0096e918 ffffffc0
[ 680.515013] bf00: ec178000 ffffffc3 009608cb ffffffc0 0007d000 00000040 007c54a0 ffffffc0
[ 680.523147] bf20: 00612cd8 ffffffc0 ec17bf60 ffffffc3 000855fc ffffffc0 ec17bf60 ffffffc3
[ 680.531283] [<ffffffc0000845a8>] el1_irq+0x68/0xc0
[ 680.536049] [<ffffffc0000ec7d4>] cpu_startup_entry+0x12c/0x1e4
[ 680.541850] [<ffffffc00008e8b8>] secondary_start_kernel+0xf8/0x108
[ 680.547998] CPU7: stopping
[ 680.550693] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.13.0-37-generic #64-Ubuntu
[ 680.558222] Call trace:
[ 680.560658] [<ffffffc000088490>] dump_backtrace+0x0/0x16c
[ 680.566028] [<ffffffc00008860c>] show_stack+0x10/0x1c
[ 680.571053] [<ffffffc0006022f8>] dump_stack+0x74/0x94
[ 680.576077] [<ffffffc00008ed08>] handle_IPI+0x120/0x138
[ 680.581273] [<ffffffc0000812cc>] gic_handle_irq+0x74/0x7c
[ 680.586642] Exception stack(0xffffffc3ec19be20 to 0xffffffc3ec19bf40)
[ 680.593049] be20: ec198000 ffffffc3 ec198000 ffffffc3 ec19bf60 ffffffc3 00085600 ffffffc0
[ 680.601184] be40: 0006e052 00000000 00000000 00000000 fffcd834 ffffffc3 00009487 00000001
[ 680.609318] be60: 00000000 00000000 ff6f6000 00000003 008f6180 ffffffc0 00000400 00000000
[ 680.617452] be80: ec183c10 ffffffc3 ec19bd60 ffffffc3 00009488 00000001 00000000 00000000
[ 680.625586] bea0: 00000018 00000000 e8000000 00000003 00000000 00000000 9b3801d6 0025f916
[ 680.633720] bec0: 001b93c0 ffffffc0 aedb8280 0000007f eabe33b0 0000007f ec198000 ffffffc3
[ 680.641855] bee0: ec198000 ffffffc3 00961000 ffffffc0 009793b8 ffffffc0 0096e918 ffffffc0
[ 680.649989] bf00: ec198000 ffffffc3 009608cb ffffffc0 0007d000 00000040 007c54a0 ffffffc0
[ 680.658123] bf20: 00612cd8 ffffffc0 ec19bf60 ffffffc3 000855fc ffffffc0 ec19bf60 ffffffc3
[ 680.666257] [<ffffffc0000845a8>] el1_irq+0x68/0xc0
[ 680.671022] [<ffffffc0000ec7d4>] cpu_startup_entry+0x12c/0x1e4
[ 680.676824] [<ffffffc00008e8b8>] secondary_start_kernel+0xf8/0x108
[ 680.682971] CPU6: stopping
[ 680.685665] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 3.13.0-37-generic #64-Ubuntu
[ 680.693194] Call trace:
[ 680.695629] [<ffffffc000088490>] dump_backtrace+0x0/0x16c
[ 680.700999] [<ffffffc00008860c>] show_stack+0x10/0x1c
[ 680.706023] [<ffffffc0006022f8>] dump_stack+0x74/0x94
[ 680.711047] [<ffffffc00008ed08>] handle_IPI+0x120/0x138
[ 680.716243] [<ffffffc0000812cc>] gic_handle_irq+0x74/0x7c
[ 680.721612] Exception stack(0xffffffc3ec197e20 to 0xffffffc3ec197f40)
[ 680.728020] 7e20: ec194000 ffffffc3 ec194000 ffffffc3 ec197f60 ffffffc3 00085600 ffffffc0
[ 680.736154] 7e40: 000788b2 00000000 00000000 00000000 fffc1834 ffffffc3 00009487 00000001
[ 680.744288] 7e60: 00000008 00000000 00000010 00000000 63406f80 0000009e d3d13868 ffffffc3
[ 680.752423] 7e80: ec183110 ffffffc3 ec197d60 ffffffc3 00009477 00000001 00000000 00000000
[ 680.760559] 7ea0: 00000018 00000000 e8000000 00000003 00000000 00000000 e73c0cdf 0025f915
[ 680.768693] 7ec0: 001b93c0 ffffffc0 b199c280 0000007f fc274530 0000007f ec194000 ffffffc3
[ 680.776827] 7ee0: ec194000 ffffffc3 00961000 ffffffc0 009793b8 ffffffc0 0096e918 ffffffc0
[ 680.784961] 7f00: ec194000 ffffffc3 009608cb ffffffc0 0007d000 00000040 007c54a0 ffffffc0
[ 680.793096] 7f20: 00612cd8 ffffffc0 ec197f60 ffffffc3 000855fc ffffffc0 ec197f60 ffffffc3
[ 680.801230] [<ffffffc0000845a8>] el1_irq+0x68/0xc0
[ 680.805994] [<ffffffc0000ec7d4>] cpu_startup_entry+0x12c/0x1e4
[ 680.811796] [<ffffffc00008e8b8>] secondary_start_kernel+0xf8/0x108
[ 680.817944] CPU2: stopping
[ 680.820639] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.13.0-37-generic #64-Ubuntu
[ 680.828168] Call trace:
[ 680.830603] [<ffffffc000088490>] dump_backtrace+0x0/0x16c
[ 680.835974] [<ffffffc00008860c>] show_stack+0x10/0x1c
[ 680.840999] [<ffffffc0006022f8>] dump_stack+0x74/0x94
[ 680.846023] [<ffffffc00008ed08>] handle_IPI+0x120/0x138
[ 680.851219] [<ffffffc0000812cc>] gic_handle_irq+0x74/0x7c
[ 680.856588] Exception stack(0xffffffc3ec17fe20 to 0xffffffc3ec17ff40)
[ 680.862995] fe20: ec17c000 ffffffc3 ec17c000 ffffffc3 ec17ff60 ffffffc3 00085600 ffffffc0
[ 680.871130] fe40: 00077b36 00000000 00000000 00000000 fff91834 ffffffc3 00009481 00000001
[ 680.879264] fe60: 00000008 00000000 00000010 00000000 17dc2180 0000069b d3d9f998 ffffffc3
[ 680.887399] fe80: ec180510 ffffffc3 ec17fd60 ffffffc3 0000947e 00000001 00000000 00000000
[ 680.895534] fea0: 00000008 00000000 00800000 00000000 00448488 00000000 95d71590 0000007f
[ 680.903668] fec0: 001b9bc4 ffffffc0 95ce91ec 0000007f e5b434f0 0000007f ec17c000 ffffffc3
[ 680.911802] fee0: ec17c000 ffffffc3 00961000 ffffffc0 009793b8 ffffffc0 0096e918 ffffffc0
[ 680.919935] ff00: ec17c000 ffffffc3 009608cb ffffffc0 0007d000 00000040 007c54a0 ffffffc0
[ 680.928069] ff20: 00612cd8 ffffffc0 ec17ff60 ffffffc3 000855fc ffffffc0 ec17ff60 ffffffc3
[ 680.936205] [<ffffffc0000845a8>] el1_irq+0x68/0xc0
[ 680.940970] [<ffffffc0000ec7d4>] cpu_startup_entry+0x12c/0x1e4
[ 680.946771] [<ffffffc00008e8b8>] secondary_start_kernel+0xf8/0x108
[ 680.952918] CPU0: stopping
[ 680.955611] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.13.0-37-generic #64-Ubuntu
[ 680.963140] Call trace:
[ 680.965574] [<ffffffc000088490>] dump_backtrace+0x0/0x16c
[ 680.970945] [<ffffffc00008860c>] show_stack+0x10/0x1c
[ 680.975969] [<ffffffc0006022f8>] dump_stack+0x74/0x94
[ 680.980993] [<ffffffc00008ed08>] handle_IPI+0x120/0x138
[ 680.986189] [<ffffffc0000812cc>] gic_handle_irq+0x74/0x7c
[ 680.991558] Exception stack(0xffffffc0008dfde0 to 0xffffffc0008dff00)
[ 680.997965] fde0: 008dc000 ffffffc0 008dc000 ffffffc0 008dff20 ffffffc0 00085600 ffffffc0
[ 681.006100] fe00: 000cda96 00000000 00000000 00000000 fff79834 ffffffc3 00009483 00000001
[ 681.014234] fe20: 00000020 00000000 00000010 00000000 5c196180 0000009e d3ecbe28 ffffffc3
[ 681.022369] fe40: 008e8e80 ffffffc0 008dfd20 ffffffc0 00009478 00000001 00000000 00000000
[ 681.030503] fe60: 0008da83 00000000 00000004 00000000 09090909 09090909 b3319590 0000007f
[ 681.038638] fe80: 001a6bc0 ffffffc0 00436310 00000000 e816d3c0 0000007f 008dc000 ffffffc0
[ 681.046772] fea0: 008dc000 ffffffc0 00961000 ffffffc0 009793b8 ffffffc0 0096e918 ffffffc0
[ 681.054907] fec0: 008dc000 ffffffc0 009608cb ffffffc0 0007d000 00000040 007c54a0 ffffffc0
[ 681.063041] fee0: 00612cd8 ffffffc0 008dff20 ffffffc0 000855fc ffffffc0 008dff20 ffffffc0
[ 681.071176] [<ffffffc0000845a8>] el1_irq+0x68/0xc0
[ 681.075940] [<ffffffc0000ec7d4>] cpu_startup_entry+0x12c/0x1e4
[ 681.081742] [<ffffffc0005f9f20>] rest_init+0x6c/0x78
[ 681.086682] [<ffffffc00088f7dc>] start_kernel+0x348/0x360
[ 681.092051] CPU3: stopping
[ 681.094744] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.13.0-37-generic #64-Ubuntu
[ 681.102273] Call trace:
[ 681.104707] [<ffffffc000088490>] dump_backtrace+0x0/0x16c
[ 681.110078] [<ffffffc00008860c>] show_stack+0x10/0x1c
[ 681.115102] [<ffffffc0006022f8>] dump_stack+0x74/0x94
[ 681.120126] [<ffffffc00008ed08>] handle_IPI+0x120/0x138
[ 681.125322] [<ffffffc0000812cc>] gic_handle_irq+0x74/0x7c
[ 681.130691] Exception stack(0xffffffc3ec18be20 to 0xffffffc3ec18bf40)
[ 681.137098] be20: ec188000 ffffffc3 ec188000 ffffffc3 ec18bf60 ffffffc3 00085600 ffffffc0
[ 681.145233] be40: 000743c4 00000000 00000000 00000000 fff9d834 ffffffc3 00009480 00000001
[ 681.153367] be60: 00000008 00000000 00000010 00000000 0c3c9980 000000a2 d3dbf998 ffffffc3
[ 681.161501] be80: ec181010 ffffffc3 ec18bd60 ffffffc3 00009481 00000001 00000000 00000000
[ 681.169635] bea0: 00000028 00000000 55a24290 00000000 bd4c233c 00000014 84000000 0012d7eb
[ 681.177769] bec0: 001f26cc ffffffc0 ad6436c0 0000007f 0000000e 00000000 ec188000 ffffffc3
[ 681.185903] bee0: ec188000 ffffffc3 00961000 ffffffc0 009793b8 ffffffc0 0096e918 ffffffc0
[ 681.194037] bf00: ec188000 ffffffc3 009608cb ffffffc0 0007d000 00000040 007c54a0 ffffffc0
[ 681.202172] bf20: 00612cd8 ffffffc0 ec18bf60 ffffffc3 000855fc ffffffc0 ec18bf60 ffffffc3
[ 681.210307] [<ffffffc0000845a8>] el1_irq+0x68/0xc0
[ 681.215072] [<ffffffc0000ec7d4>] cpu_startup_entry+0x12c/0x1e4
[ 681.220874] [<ffffffc00008e8b8>] secondary_start_kernel+0xf8/0x108

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1418284

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Ming Lei (tom-leiming)]

xfstests config(local.config):

    export TEST_DEV=/dev/loop0
    export TEST_DIR=/mnt/test
    export SCRATCH_DEV=/dev/loop1
    export SCRATCH_MNT=/mnt/scratch

ming@am2:~/git/xfstests$ sudo losetup -a
/dev/loop0: [0802]:25306845 (/home/ming/git/xfstests/loop0.img)
/dev/loop1: [0802]:25306843 (/home/ming/git/xfstests/loop1.img)

ming@am2:~/git/xfstests$ ls -lh *.img
-rw-r--r-- 1 ming ming 8.0G Feb 4 20:53 loop0.img
-rw-r--r-- 1 ming ming 8.0G Feb 4 20:53 loop1.img

test commands:

cd ~/git/xfstests; sudo ./check generic/036

[Ming Lei (tom-leiming)]

The test box is in private environment, and I can't run apport-collect to collect log automatically.

If you need any specific log, please let me know and I can provide that.

BTW, the bug can be duplicated every time.

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.19 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19-rc7-vivid/

[Ming Lei (tom-leiming)]

xfstests generic/036 works fine with 3.18-rc6 kernel.

[Ming Lei (tom-leiming)]

The patch in the following link does fix the problem:

http://git.kernel.org/cgit/linux/kernel/git/stable/stable-queue.git/tree/queue-3.14/ext4-prevent-bugon-on-race-between-write-fcntl.patch

[Ming Lei (tom-leiming)]

Fixed in 3.13.0-47-generic #78