I have seen this both on EC2 and physical hardware. I installed linux-crashdump on these machines to see if I can get more information but I will have to wait for another crash.
divide error: 0000 [#1] SMP [1449293.452514] last sysfs file: /sys/devices/xen/vbd-16756/block/sdx4/stat [1449293.452518] CPU 0 [1449293.452521] Modules linked in: raid0 ipt_REJECT ipt_LOG xt_limit xt_tcpudp ipt_addrtype xt_state ip6table_filter ip6_tables ipv6 md_mod nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables [1449293.452547] Pid: 10926, comm: beam.smp Not tainted 2.6.32-305-ec2 #9-Ubuntu [1449293.452551] RIP: e030:[<ffffffff8102ceb4>] [<ffffffff8102ceb4>] update_sd_lb_stats+0x3a4/0x4e0 [1449293.452563] RSP: e02b:ffff880443e137e8 EFLAGS: 00010046 [1449293.452566] RAX: 0000000000000000 RBX: ffff880443e139d4 RCX: 0000000000000001 [1449293.452570] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [1449293.452574] RBP: ffff880443e138c8 R08: ffff88000184dbc8 R09: 0000000000000040 [1449293.452578] R10: ffff880444358240 R11: 0000000000000000 R12: 00000000ffffffff [1449293.452582] R13: 000000000000a380 R14: ffffffffffffffff R15: 0000000000000000 [1449293.452591] FS: 00007f0f8fac3710(0000) GS:ffff880001846000(0000) knlGS:0000000000000000 [1449293.452595] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [1449293.452599] CR2: 00007f0f80825008 CR3: 000000043ed16000 CR4: 0000000000002620 [1449293.452603] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1449293.452607] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000 [1449293.452611] Process beam.smp (pid: 10926, threadinfo ffff880443e12000, task ffff88043ee94200) [1449293.452616] Stack: [1449293.452618] 0000000000000246 ffff880443e13868 000000002179d200 0000000000000000 [1449293.452623] <0> ffff88000184daa0 0000000000000000 0000000000000000 0000000000000008 [1449293.452630] <0> 000000000000a380 000000000000a380 ffff88000184dbb0 000000000000a380 [1449293.452637] Call Trace: [1449293.452644] [<ffffffff8103877d>] find_busiest_group+0x4d/0x460 [1449293.452652] [<ffffffff81120650>] ? do_mpage_readpage+0x330/0x630 [1449293.452656] [<ffffffff81039294>] load_balance_newidle+0xa4/0x320 [1449293.452662] [<ffffffff814a1623>] thread_return+0x3cc/0x429 [1449293.452668] [<ffffffff81291d10>] ? xfs_get_blocks+0x0/0x20 [1449293.452673] [<ffffffff8131c5bf>] ? blk_unplug+0x2f/0x70 [1449293.452679] [<ffffffffa0128422>] ? raid0_unplug+0x52/0x70 [raid0] [1449293.452685] [<ffffffff810adc40>] ? sync_page_killable+0x0/0x40 [1449293.452689] [<ffffffff814a16c2>] io_schedule+0x42/0x60 [1449293.452693] [<ffffffff810adc2d>] sync_page+0x3d/0x50 [1449293.452697] [<ffffffff810adc49>] sync_page_killable+0x9/0x40 [1449293.452701] [<ffffffff814a1b32>] __wait_on_bit_lock+0x52/0xb0 [1449293.452707] [<ffffffff81339e1d>] ? radix_tree_prev_hole+0x4d/0x60 [1449293.452711] [<ffffffff810adb62>] __lock_page_killable+0x62/0x70 [1449293.452717] [<ffffffff81059d90>] ? wake_bit_function+0x0/0x40 [1449293.452721] [<ffffffff810ada09>] ? find_get_page+0x19/0xa0 [1449293.452725] [<ffffffff810af8f7>] T.769+0x1b7/0x410 [1449293.452729] [<ffffffff810afc06>] generic_file_aio_read+0xb6/0x1d0 [1449293.452734] [<ffffffff814a3403>] ? __down_read+0xf3/0x110 [1449293.452740] [<ffffffff8129af7a>] xfs_read+0x11a/0x2a0 [1449293.452747] [<ffffffff81066c09>] ? unqueue_me+0x79/0xd0 [1449293.452751] [<ffffffff810679a7>] ? futex_wait+0x257/0x290 [1449293.452756] [<ffffffff8129652b>] xfs_file_aio_read+0x5b/0x70 [1449293.452761] [<ffffffff810ed322>] do_sync_read+0xf2/0x130 [1449293.452765] [<ffffffff81059d50>] ? autoremove_wake_function+0x0/0x40 [1449293.452770] [<ffffffff81066ee2>] ? futex_wake+0x112/0x130 [1449293.452776] [<ffffffff812dab21>] ? security_file_permission+0x11/0x20 [1449293.452780] [<ffffffff810edd05>] vfs_read+0xb5/0x1a0 [1449293.452784] [<ffffffff810edf6a>] sys_pread64+0x7a/0x90 [1449293.452790] [<ffffffff81009ba8>] system_call_fastpath+0x16/0x1b [1449293.452794] [<ffffffff81009b40>] ? system_call+0x0/0x52 [1449293.452797] Code: 06 89 85 50 ff ff ff c7 85 54 ff ff ff 01 00 00 00 e9 cf fd ff ff 90 48 8b 95 70 ff ff ff 48 8b 45 a8 8b 72 08 48 c1 e0 0a 31 d2 <48> f7 f6 48 8b 75 b0 48 89 45 a0 31 c0 48 85 f6 74 0c 48 8b 45 [1449293.452837] RIP [<ffffffff8102ceb4>] update_sd_lb_stats+0x3a4/0x4e0 [1449293.452842] RSP <ffff880443e137e8> [1449293.452848] ---[ end trace 9b3628a023db21fb ]---
ProblemType: Bug DistroRelease: Ubuntu 10.04 Package: linux-image-2.6.32-305-ec2 2.6.32-305.9 ProcVersionSignature: Ubuntu 2.6.32-305.9-ec2 2.6.32.11+drm33.2 Uname: Linux 2.6.32-305-ec2 x86_64 Architecture: amd64 Date: Sat Aug 7 21:55:31 2010 Ec2AMI: ami-fd4aa494 Ec2AMIManifest: ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20100427.1.manifest.xml Ec2AvailabilityZone: us-east-1b Ec2InstanceType: m2.xlarge Ec2Kernel: aki-0b4aa462 Ec2Ramdisk: unavailable ProcEnviron: PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: linux-ec2
I have seen this both on EC2 and physical hardware. I installed linux-crashdump on these machines to see if I can get more information but I will have to wait for another crash.
divide error: 0000 [#1] SMP xen/vbd- 16756/block/ sdx4/stat ffffffff8102ceb 4>] [<ffffffff8102c eb4>] update_ sd_lb_stats+ 0x3a4/0x4e0 e137e8 EFLAGS: 00010046 0(0000) GS:ffff88000184 6000(0000) knlGS:000000000 0000000 77d>] find_busiest_ group+0x4d/ 0x460 650>] ? do_mpage_ readpage+ 0x330/0x630 294>] load_balance_ newidle+ 0xa4/0x320 623>] thread_ return+ 0x3cc/0x429 d10>] ? xfs_get_ blocks+ 0x0/0x20 5bf>] ? blk_unplug+ 0x2f/0x70 422>] ? raid0_unplug+ 0x52/0x70 [raid0] c40>] ? sync_page_ killable+ 0x0/0x40 6c2>] io_schedule+ 0x42/0x60 c2d>] sync_page+0x3d/0x50 c49>] sync_page_ killable+ 0x9/0x40 b32>] __wait_ on_bit_ lock+0x52/ 0xb0 e1d>] ? radix_tree_ prev_hole+ 0x4d/0x60 b62>] __lock_ page_killable+ 0x62/0x70 d90>] ? wake_bit_ function+ 0x0/0x40 a09>] ? find_get_ page+0x19/ 0xa0 8f7>] T.769+0x1b7/0x410 c06>] generic_ file_aio_ read+0xb6/ 0x1d0 403>] ? __down_ read+0xf3/ 0x110 f7a>] xfs_read+ 0x11a/0x2a0 c09>] ? unqueue_ me+0x79/ 0xd0 9a7>] ? futex_wait+ 0x257/0x290 52b>] xfs_file_ aio_read+ 0x5b/0x70 322>] do_sync_ read+0xf2/ 0x130 d50>] ? autoremove_ wake_function+ 0x0/0x40 ee2>] ? futex_wake+ 0x112/0x130 b21>] ? security_ file_permission +0x11/0x20 d05>] vfs_read+0xb5/0x1a0 f6a>] sys_pread64+ 0x7a/0x90 ba8>] system_ call_fastpath+ 0x16/0x1b b40>] ? system_ call+0x0/ 0x52 eb4>] update_ sd_lb_stats+ 0x3a4/0x4e0
[1449293.452514] last sysfs file: /sys/devices/
[1449293.452518] CPU 0
[1449293.452521] Modules linked in: raid0 ipt_REJECT ipt_LOG xt_limit xt_tcpudp ipt_addrtype xt_state ip6table_filter ip6_tables ipv6 md_mod nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables
[1449293.452547] Pid: 10926, comm: beam.smp Not tainted 2.6.32-305-ec2 #9-Ubuntu
[1449293.452551] RIP: e030:[<
[1449293.452563] RSP: e02b:ffff880443
[1449293.452566] RAX: 0000000000000000 RBX: ffff880443e139d4 RCX: 0000000000000001
[1449293.452570] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[1449293.452574] RBP: ffff880443e138c8 R08: ffff88000184dbc8 R09: 0000000000000040
[1449293.452578] R10: ffff880444358240 R11: 0000000000000000 R12: 00000000ffffffff
[1449293.452582] R13: 000000000000a380 R14: ffffffffffffffff R15: 0000000000000000
[1449293.452591] FS: 00007f0f8fac371
[1449293.452595] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[1449293.452599] CR2: 00007f0f80825008 CR3: 000000043ed16000 CR4: 0000000000002620
[1449293.452603] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1449293.452607] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
[1449293.452611] Process beam.smp (pid: 10926, threadinfo ffff880443e12000, task ffff88043ee94200)
[1449293.452616] Stack:
[1449293.452618] 0000000000000246 ffff880443e13868 000000002179d200 0000000000000000
[1449293.452623] <0> ffff88000184daa0 0000000000000000 0000000000000000 0000000000000008
[1449293.452630] <0> 000000000000a380 000000000000a380 ffff88000184dbb0 000000000000a380
[1449293.452637] Call Trace:
[1449293.452644] [<ffffffff81038
[1449293.452652] [<ffffffff81120
[1449293.452656] [<ffffffff81039
[1449293.452662] [<ffffffff814a1
[1449293.452668] [<ffffffff81291
[1449293.452673] [<ffffffff8131c
[1449293.452679] [<ffffffffa0128
[1449293.452685] [<ffffffff810ad
[1449293.452689] [<ffffffff814a1
[1449293.452693] [<ffffffff810ad
[1449293.452697] [<ffffffff810ad
[1449293.452701] [<ffffffff814a1
[1449293.452707] [<ffffffff81339
[1449293.452711] [<ffffffff810ad
[1449293.452717] [<ffffffff81059
[1449293.452721] [<ffffffff810ad
[1449293.452725] [<ffffffff810af
[1449293.452729] [<ffffffff810af
[1449293.452734] [<ffffffff814a3
[1449293.452740] [<ffffffff8129a
[1449293.452747] [<ffffffff81066
[1449293.452751] [<ffffffff81067
[1449293.452756] [<ffffffff81296
[1449293.452761] [<ffffffff810ed
[1449293.452765] [<ffffffff81059
[1449293.452770] [<ffffffff81066
[1449293.452776] [<ffffffff812da
[1449293.452780] [<ffffffff810ed
[1449293.452784] [<ffffffff810ed
[1449293.452790] [<ffffffff81009
[1449293.452794] [<ffffffff81009
[1449293.452797] Code: 06 89 85 50 ff ff ff c7 85 54 ff ff ff 01 00 00 00 e9 cf fd ff ff 90 48 8b 95 70 ff ff ff 48 8b 45 a8 8b 72 08 48 c1 e0 0a 31 d2 <48> f7 f6 48 8b 75 b0 48 89 45 a0 31 c0 48 85 f6 74 0c 48 8b 45
[1449293.452837] RIP [<ffffffff8102c
[1449293.452842] RSP <ffff880443e137e8>
[1449293.452848] ---[ end trace 9b3628a023db21fb ]---
ProblemType: Bug 2.6.32- 305-ec2 2.6.32-305.9 ature: Ubuntu 2.6.32-305.9-ec2 2.6.32.11+drm33.2 images- us/ubuntu- lucid-10. 04-amd64- server- 20100427. 1.manifest. xml Zone: us-east-1b
DistroRelease: Ubuntu 10.04
Package: linux-image-
ProcVersionSign
Uname: Linux 2.6.32-305-ec2 x86_64
Architecture: amd64
Date: Sat Aug 7 21:55:31 2010
Ec2AMI: ami-fd4aa494
Ec2AMIManifest: ubuntu-
Ec2Availability
Ec2InstanceType: m2.xlarge
Ec2Kernel: aki-0b4aa462
Ec2Ramdisk: unavailable
ProcEnviron:
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: linux-ec2